I’m trying to set up a task in Airflow that creates a pod in Google Kubernetes from a Docker image in Registry. The task fails and returns: Pod Launching failed: Pod took too long to start Which does not say much. In a similar question, someone suggested adding a tag to the image version when ..
Current State of airflow dag: ml_processors = [a, b, c, d, e] abc_task >> ml_processors (all ml models from a to e run in parallel after abc task is successfully completed) ml_processors >> xyz_task (once a to e all are successful xyz task runs) Problem statement: There are instances when one of the machine learning ..
I’m having a hard time Googling for this. What I’m trying to do is have an Airflow task running on Google Cloud Composer that will download data from BigQuery, massage the data, then issue a docker build command to build a Docker image, then send it to Google Container Registry and do other stuff thereafter. ..