I’m somewhat inexperienced with both Docker and Airflow, so this might be a silly question. I have a Dockerfile that uses the
apache/airflow image together with some of my own DAGs. I would like to launch the airflow web server together with the scheduler and I’m having trouble with this. I can get it working, but I feel that I’m approaching this incorrectly.
Here is what my Dockerfile looks like:
FROM apache/airflow COPY airflow/dags/ /opt/airflow/dags/ RUN airflow initdb
Then I run
docker build -t learning/airflow .. Here is the tough part: I then run
docker run --rm -tp 8080:8080 learning/airflow:latest webserver and in a separate terminal I run
docker exec `docker ps -q` airflow scheduler. The trouble is, that in practice this generally happens on a VM somewhere, so opening up a second terminal is just not an option and multiple machines will probably not have access to the same docker container. Running
webserver && scheduler does not seem to work, the server appears to be blocking and I’m still seeing the message “The scheduler does not appear to be running” in Airflow UI.
Any ideas on what the right way to run server and scheduler should be?