Problem statement I have the following Dockerfile: ARG BASE_IMAGE=python:3.9-slim FROM $BASE_IMAGE as base RUN apt update && apt install –yes –no-install-recommends build-essential libsasl2-dev && apt clean RUN python -m venv /opt/venv ENV PATH="/opt/venv/bin:$PATH" RUN pip install –upgrade –no-cache-dir pip && pip install –no-cache-dir dbt-spark[PyHive]==0.20.1 FROM $BASE_IMAGE as runtime RUN apt update && apt install –yes ..
I’m trying to modify an Airflow docker-compose set-up with an extended image from a Dockerfile to have dbt installed on the container but the docker-compose file seems to be ignoring the Dockerfile: the different airflow containers are launched and run correctly but not a single one has dbt (fully) installed. I get the following error ..
I am new to DBT and currently trying to build a Docker container where I can directly run DBT commands within. I have a file where I export env variables (envs.sh) that looks like: export DB_HOST="secret" export DB_PWD="evenabiggersecret" My packages.yml looks like: packages: – package: fishtown-analytics/dbt_utils version: 0.6.2 I structured my docker file like: FROM ..
Building my question on How to run DBT in airflow without copying our repo, I am currently running airflow and syncing the dags via git. I am considering different option to include DBT within my workflow. One suggestion by louis_guitton is to Dockerize the DBT project, and run it in Airflow via the Docker Operator. ..
Background I am currently deploying Apache Airflow using Helm. I am using a git-sync sidecar to mount the SQL & Python files which Airflow will need to have access to to be able to execute scripts/files. What seems not to work Once I am done with deploying my container, it seems that my Airflow user ..
We’re running DBT in Airflow on a GCP Compute Engine using Docker and docker-compose. In the docker-compose.yml file for our Airflow deployment, the DBT repo is attached as a volume: webserver: … volumes: – ../our-dbt-repo:/usr/local/airflow/dbt Running dbt-run usually generates a /logs directory with DBT logs. However, running dbt-run from the docker container on the GCP ..
I’m trying to deploy dbt on a Google cloud run service with a docker container. following david vasquez and dbt Docker images However when trying to deploy the builded image to cloud run. I’m getting an error. ERROR: (gcloud.run.deploy) Cloud Run error: Container failed to start. Failed to start and then listen on the port ..