Category : apache-spark

My spark continers (master and workers) is not pushing the logs(stdout/stderr) to cloudwatch. Below is the design. Spark image build using spark-3.1.2-bin-hadoop2.7. below is my docker-compose file: These containers runs on Ec2 instance. Just a single node with standalone cluster (master/workers) . I used Airflow scheduler to submit spark jobs. When submitted the driver runs ..

Read more

I’m trying to run spark in a docker container from a python app which is located in another container: version: ‘3’ services: spark-master: image: docker.io/bitnami/spark:2 environment: – SPARK_MODE=master – SPARK_RPC_AUTHENTICATION_ENABLED=no – SPARK_RPC_ENCRYPTION_ENABLED=no – SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no – SPARK_SSL_ENABLED=no volumes: – type: bind source: ./conf/log4j.properties target: /opt/bitnami/spark/conf/log4j.properties ports: – ‘8080:8080’ – ‘7077:7077’ networks: – spark container_name: spark spark-worker-1: ..

Read more

I am saving csv files as stream with pyspark. When I saving files, I am using output mode is ‘overwrite’ and there is not any problem. But when I want to containerize my spark app is giving an error. I add code and the error below: df.write.format("csv").mode("overwrite") java.io.IOException: Unable to clear output directory file:/app/files prior ..

Read more

I have a docker-swarm setup. I am running two containers over the host network on different docker-hosts/VMs, which are as follow: spark-master: On VM/Host-1 spark-worker: On VM/Host-2 The VMs can ping each other, so can the docker-containers ping each other using the other’s host-ips. However in the spark-UI, the spark-master container is referenced as it’s ..

Read more

Im trying to connect to a mysql container thats running on local maching using a scala code. For some reason Im getting either operation time out(dockerIp) or connection refused error(localhost/127.0.0.1). docker run –name=mysql-docker -p3306:3306 -e MYSQL_ROOT_PASSWORD=’root’ -d mysql:latest docker exec -it mysql-docker bash mysql -u root -p update mysql.user set host = ‘127.0.0.1’ where user=’root’; ..

Read more