Category : airflow-scheduler

I am experiencing scheduler down suddenly and without a pattern. I run Airflow 2.1.2 in AWS ECS using docker. I know it’s a reported issue in Airflow docs We setup AIRFLOW_SCHEDULER_RUN_DURATION = 3600 to force scheduler restart every hour and avoid this way scheduler is down suddenly. But despite that, we see last hearbeat was ..

Read more

I’m running Airflow 2.0 using docker, when i run sudo docker-compose up –build I’m getting an error ERROR: for apache_airflow_scheduler_1 Cannot start service scheduler: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "./scripts/airflow-entrypoint.sh": permission denied: unknown ERROR: for webserver Cannot start service webserver: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: ..

Read more

I am running Airflow 2.1.2 in localhost using: docker-compose version: ‘3’ services: webserver: build: context: . dockerfile: Dockerfile-python3 image: analytics-airflow env_file: – ./environtment_config/airflow_config/airflow.env container_name: analytics-container-3 volumes: – ./database_utils:/database_utils – ./maintenance:/maintenance – ./utils:/utils – ./dags:/opt/airflow/dags – ./logs:/opt/airflow/logs – ./datawarehouse:/datawarehouse ports: – "8080:8080" command: webserver scheduler: container_name: analytics-scheduler build: context: . dockerfile: Dockerfile-python3 env_file: – ./environtment_config/airflow_config/airflow.env volumes: ..

Read more

I am fairly new to Airflow and Docker environments. Yet, I need to setup the Airflow 2.0 as a Docker with Local Executor. I have found many resources such as the following: http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/start/docker.html But all refer to the Celery Executor setup. In need for the Local Executor setup and the creation of the docker-compose.yaml files. ..

Read more

I am running airflow locally using a docker image. The docker image was build using docker build –rm -f Dockerfile-airflow -t airflow_image . after the docker image is build, I am running it locally by running the webserver using docker run -it -p 8080:8080 –env-file <local.env location> -v <absolute ocation for dags>:/opt/airflow/dags airflow_image:latest webserver The ..

Read more