Category : airflow

According to airflow connection management page, we can use environment variable to create connections: export AIRFLOW_CONN_MY_PROD_DATABASE=’my-conn-type://login:[email protected]:port/schema?param1=val1&param2=val2′ So, I’ve downloaded the official docker-compose.yml: $ curl -LfO ‘https://airflow.apache.org/docs/apache-airflow/2.2.0/docker-compose.yaml’ And added environment variable of a connection like below: … 47 image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.2.0} 48 # build: . 49 environment: 50 &airflow-common-env 51 AIRFLOW_CONN_MY_PROD_DB: my-conn-type://login:[email protected]:port/schema?param1=val1&param2=val2 52 AIRFLOW__CORE__EXECUTOR: CeleryExecutor … Then, ..

Read more

I’m currently using a docker-compose.yml to run a airflow cluster. And I’m doing a little customization on this to configure some environment in airflow cluster. Belows are my working directory(called my_application) structure: ➜ tree -L 2 . ├── Dockerfile ├── docker-compose.yml ├── configs │ ├── configure_all.sh │ ├── create_connections.sh │ ├── requirements │ └── variables.json ..

Read more

My spark continers (master and workers) is not pushing the logs(stdout/stderr) to cloudwatch. Below is the design. Spark image build using spark-3.1.2-bin-hadoop2.7. below is my docker-compose file: These containers runs on Ec2 instance. Just a single node with standalone cluster (master/workers) . I used Airflow scheduler to submit spark jobs. When submitted the driver runs ..

Read more