Category : airflow

We have Airflow running (using Docker compose) with several DAG’s active. Last week we updated our Airflow to version 2.1.3. This resulted in an error for a DAG where we use DockerOperator: airflow.exceptions.AirflowException: Invalid arguments were passed to DockerOperator (task_id: t_docker). Invalid arguments were: **kwargs: {‘volumes’: I found this release note telling me that The ..

Read more

I am running Airflow 2.1.2 in localhost using: docker-compose version: ‘3’ services: webserver: build: context: . dockerfile: Dockerfile-python3 image: analytics-airflow env_file: – ./environtment_config/airflow_config/airflow.env container_name: analytics-container-3 volumes: – ./database_utils:/database_utils – ./maintenance:/maintenance – ./utils:/utils – ./dags:/opt/airflow/dags – ./logs:/opt/airflow/logs – ./datawarehouse:/datawarehouse ports: – "8080:8080" command: webserver scheduler: container_name: analytics-scheduler build: context: . dockerfile: Dockerfile-python3 env_file: – ./environtment_config/airflow_config/airflow.env volumes: ..

Read more

I’m trying to run multiple containers from the same docker-compose, the three of the matter are airflow, superset and postgres. I’m trying to get both airflow and superset to access and write to the same database container postgres, however I keep getting this error for the superset app: ERROR:flask_appbuilder.security.sqla.manager:DB Creation and initialization failed: (psycopg2.OperationalError) could ..

Read more

I have received an unknown error when running airflow. We do install Keras. dockerfile RUN pip install "Keras==2.3.1" RUN pip install "apache-airflow[celery,devel,postgres,redis,s3,ssh]==1.10.9" But we are getting the missing keras.api module. I can’t find keras.api was used anywhere. ml.scheduler_1 | File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed ml.scheduler_1 | File "/robo/airflow/dags/general/video_migration.py", line 5, in <module> ml.scheduler_1 ..

Read more

I have received an unknown error when running airflow. We do install Keras. dockerfile RUN pip install "Keras==2.3.1" RUN pip install "apache-airflow[celery,devel,postgres,redis,s3,ssh]==1.10.9" RUN pip install "keras-models==0.0.7" But we are getting the missing keras.api module. I can’t find keras.api was used anywhere. ml.scheduler_1 | File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed ml.scheduler_1 | File "/robo/airflow/dags/general/video_migration.py", line ..

Read more

I am trying to get an Apache Airflow image to install h2o, usually I would just need to run pip install h2o and it works. Based on this answer I would need to extend the image and I did so. airflow/Dockerfile: FROM apache/airflow:2.1.2 USER root RUN apt-get update && apt-get install -y –no-install-recommends build-essential h2o ..

Read more