Author : Leticia Cruz

I’m trying to use the docker operator to automate the execution of some scripts using airflow. Airflow version: apache-airflow==1.10.12 What I want to do is to "copy" all my project’s files (with folders and files) to the container using this code. The following file ml-intermediate.py is in this directory ~/airflow/dags/ml-intermediate.py: """ Template to convert a ..

Read more

I’m trying to use the docker operator on an airflow pipeline. This is the code I’m using: from airflow import DAG from airflow.operators.bash_operator import BashOperator from datetime import datetime, timedelta from airflow.operators.docker_operator import DockerOperator default_args = { ‘owner’ : ‘airflow’, ‘description’ : ‘Use of the DockerOperator’, ‘depend_on_past’ : False, ‘start_date’ : datetime(2018, 1, 3), ’email_on_failure’ ..

Read more