docker containers not seeing each other despite created from docker-compose

  airflow, airflow-scheduler, docker, docker-compose

I am running Airflow 2.1.2 in localhost using:

docker-compose

version: '3'
services:
  webserver:
    build:
      context: .
      dockerfile: Dockerfile-python3
    image: analytics-airflow
    env_file:
      - ./environtment_config/airflow_config/airflow.env
    container_name: analytics-container-3
    volumes:
      - ./database_utils:/database_utils
      - ./maintenance:/maintenance
      - ./utils:/utils
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
      - ./datawarehouse:/datawarehouse
    ports:
      - "8080:8080"
    command: webserver
  scheduler:
    container_name: analytics-scheduler
    build:
      context: .
      dockerfile: Dockerfile-python3
    env_file:
      - ./environtment_config/airflow_config/airflow.env
    volumes:
      - ./database_utils:/database_utils
      - ./maintenance:/maintenance
      - ./utils:/utils
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
      - ./datawarehouse:/datawarehouse
    ports:
      - "8793:8793"
    command: scheduler

Dockerfile

FROM apache/airflow:2.1.2-python3.8

ENV PYTHONPATH "${PYTHONPATH}:/"

RUN pip install -r environtment_config/docker_src/requirements.pip

Airflow.env

#Execution behaviour variables
_AIRFLOW_WWW_USER_CREATE=True
_AIRFLOW_WWW_USER_USERNAME=admin
_AIRFLOW_WWW_USER_PASSWORD=admin
_AIRFLOW_DB_UPGRADE=true
#Building variables
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'true'
AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
AIRFLOW__CORE__SQL_ALCHEMY_CONN= sqlite:////opt/airflow/airflow.db
AIRFLOW__CORE__EXECUTOR=SequentialExecutor
AIRFLOW__WEBSERVER__SECRET_KEY=localhost
AIRFLOW__CORE__FERNET_KEY=
AIRFLOW__LOGGING__REMOTE_LOGGING= False
AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER=

The UI shows that scheduler is not running:
enter image description here

If I run docker ps

CONTAINER ID   IMAGE                COMMAND                  CREATED             STATUS                       PORTS                                       NAMES
1d509c71c804   airflow2   "/usr/bin/dumb-init …"   42 minutes ago      Up 42 minutes (healthy)      8080/tcp                                    scheduler
f1a945f1df01   airflow2   "/usr/bin/dumb-init …"   About an hour ago   Up About an hour (healthy)   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp   container-3

I achieved to fix it connecting to webserver container with:

docker exec -it container-3 bash
airflow scheduler

So the fix is to run in the same container both services but I don’t understand why it doesn’t work with two different container as both are created with docker-compose they should see each other.

Also it’s important to achieve one container per service as I will be able to raise multiple scheduler containers and use HA Scheduler, and scale up the server.

Also, if both services runs in a single container logs will be mix, so it will be harded to debug if something happens in one of the services.

Any idea how to fix it? or why this is happening?

Source: Docker Questions

LEAVE A COMMENT