Category : airflow-scheduler

I am fairly new to Airflow and Docker environments. Yet, I need to setup the Airflow 2.0 as a Docker with Local Executor. I have found many resources such as the following: http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/start/docker.html But all refer to the Celery Executor setup. In need for the Local Executor setup and the creation of the docker-compose.yaml files. ..

Read more

I am running airflow locally using a docker image. The docker image was build using docker build –rm -f Dockerfile-airflow -t airflow_image . after the docker image is build, I am running it locally by running the webserver using docker run -it -p 8080:8080 –env-file <local.env location> -v <absolute ocation for dags>:/opt/airflow/dags airflow_image:latest webserver The ..

Read more

BACKGROUND With the introduction of the Airflow Kubernetes Executor, my company is considering adopting Airflow as our primary ETL framework. I’ve read countless articles online all with slight variations of deploying Airflow on a Kubernetes cluster in the cloud. I’m hoping that someone with experience deploying Airflow in Kubernetes can clear up some of my ..

Read more

I am trying to setup apache airflow via docker and am facing an issue. I am using the following page – https://airflow.apache.org/docs/apache-airflow/2.0.1/start/docker.html and when I run the following command – docker-compose up airflow-init I am getting the following error airflow-docker_redis_1 is up-to-date Starting airflow-docker_airflow-init_1 … done Attaching to airflow-docker_airflow-init_1 airflow-init_1 | BACKEND=postgresql+psycopg2 airflow-init_1 | DB_HOST=postgres ..

Read more