I am trying to run a containerized application of Airflow and Spark using the following repository, https://github.com/cordon-thiago/airflow-spark As given in the steps here, I need to edit the spark_default connection for my DAGs to be submitted to Spark, however, I cannot seem to do that. This is what I see when I try doing it, ..
I’m running airflow using docker and I want to use the cli to perform pause and unpause a dag since I’m unable to install a browser on a remote Digital Ocean droplet server in order to access airflow UI. At the moment I’m trying to access the airflow cli with the command sudo docker exec ..
I run Airflow 2.1.2 in AWS ECS using docker. My scheduler shows Error log such as: airflow.exceptions.SerializedDagNotFound Checking in the dag table in the database, the dag that this error shows has active=False but is_paused=False. I found 4 dags that had these same behavior. All of them weren’t in the UI. The ones that are ..
I am experiencing scheduler down suddenly and without a pattern. I run Airflow 2.1.2 in AWS ECS using docker. I know it’s a reported issue in Airflow docs We setup AIRFLOW_SCHEDULER_RUN_DURATION = 3600 to force scheduler restart every hour and avoid this way scheduler is down suddenly. But despite that, we see last hearbeat was ..
I’m running Airflow 2.0 using docker, when i run sudo docker-compose up –build I’m getting an error ERROR: for apache_airflow_scheduler_1 Cannot start service scheduler: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "./scripts/airflow-entrypoint.sh": permission denied: unknown ERROR: for webserver Cannot start service webserver: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: ..
I am running Airflow 2.1.2 in localhost using: docker-compose version: ‘3’ services: webserver: build: context: . dockerfile: Dockerfile-python3 image: analytics-airflow env_file: – ./environtment_config/airflow_config/airflow.env container_name: analytics-container-3 volumes: – ./database_utils:/database_utils – ./maintenance:/maintenance – ./utils:/utils – ./dags:/opt/airflow/dags – ./logs:/opt/airflow/logs – ./datawarehouse:/datawarehouse ports: – "8080:8080" command: webserver scheduler: container_name: analytics-scheduler build: context: . dockerfile: Dockerfile-python3 env_file: – ./environtment_config/airflow_config/airflow.env volumes: ..
I am attempting to run airflow in a docker container, tied to postgresql. I am compiling the scheduler as one container and the airflow server as another. At the moment everything compiles but when I try to access the airflow server I get warnings. When I try to access the webserver I can’t get past ..
I am trying to run airflow on an anaconda3 docker image. Everything is working fine, except I am not able to run airflow webserver and scheduler in the background using: airflow webserver -D airflow scheduler -D Both of them start in the foreground and I have to open a new terminal for them to run. ..
I am fairly new to Airflow and Docker environments. Yet, I need to setup the Airflow 2.0 as a Docker with Local Executor. I have found many resources such as the following: http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/start/docker.html But all refer to the Celery Executor setup. In need for the Local Executor setup and the creation of the docker-compose.yaml files. ..
I am running airflow locally using a docker image. The docker image was build using docker build –rm -f Dockerfile-airflow -t airflow_image . after the docker image is build, I am running it locally by running the webserver using docker run -it -p 8080:8080 –env-file <local.env location> -v <absolute ocation for dags>:/opt/airflow/dags airflow_image:latest webserver The ..