Airflow on Windows with Docker – unable to install or load my own dags/broken dag error

  airflow, docker, windows

I’ve spent an incredible amount of time on this and I believe I’ve now read every single thread or post on this but nothing seems to be helping.

I am trying to run Airflow on my (work) Windows machine using Docker. I’ve tried the following two approaches but without success:

  1. Follow the guide by Airflow, get the yaml file and run compose-up (https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html) – I get the following error:

`Creating airflow_airflow-init_1 … error

ERROR: for airflow_airflow-init_1 Cannot create container for service airflow-init: TCGA-FS01UsersMyNameDocumentsAirflowlogs%!(EXTRA string=is not a valid Windows path`

Nothing that I’ve tried seems to fix this error and I just cannot initiate the actual airflow image. I’m confused as to what the error is referring to because I do not have a "logs%!" folder/file in my Airflow folder – the Airflow folder only has the yaml file and a two dags that I was hoping to load.

  1. Run the puckel version of Airflow. I am able to start this successfully without any issues. If I run the "airflow scheduler" command in the CLI or using bash, I am able to load all of the example dags that come with the puckel airflow container (none will show in the webserver until I ran this).
    Following this guide: http://www.marknagelberg.com/getting-started-with-airflow-using-docker/
    I’ve tried mounting my own local folder to the usr/local/airflow/dags and it seems to mount it successfully again but my own dags never show up.

EDIT: I seem to have made some progress – I run the following:

docker cp sample_dag.py containerName:/usr/local/airflow/dags

And it seems that my dag gets loaded into the right folder because when I go into the webserver, I get the following error:

Broken DAG: [/usr/local/airflow/dags] No module named ‘airflow.operators.bash’

Looking at the different posts, it seems like I need to add a requirements.txt file but I am not sure how to do this exactly – I am guessing that based on the following: https://github.com/puckel/docker-airflow/issues/592

So in summary, Option 1 – I get the windows string error and I am unable to proceed with the start up. Option 2 – I cannot get my load my own dags or I get a Broken DAG/module error.

I am absolutely lost and if anyone could help me, I would be eternally grateful. Thank you folks.

Source: Docker Questions

LEAVE A COMMENT