I would like to install apache airflow and apache beam together using either Docker or Docker-Compose.
What I’m trying to accomplish is: Currently, I’ve apache airflow DAGS with different tasks and in one of the task I want to incorporate/ integrate apache beam data pipeline. And In order to run this workflow successfully I need to install the apache beam dependency. For example: pip install apache-beam and pip install apache-beam[gcp]
I want to go with the first link as its an official image. If anyone has solution with puckel docker image then please help me to understand how can I get the latest changes everytime from the repository.
I’m new to docker, apache-airflow and apache-beam. I’m learning and growing slowly. So, your help is much appreciated. Thank you so much!!! 🙂