Cannot install h2o in a docker container

  airflow, docker, h2o, python

I am trying to get an Apache Airflow image to install h2o, usually I would just need to run pip install h2o and it works.

Based on this answer I would need to extend the image and I did so.

airflow/Dockerfile:

FROM apache/airflow:2.1.2
USER root
RUN apt-get update 
  && apt-get install -y --no-install-recommends 
         build-essential h2o 
  && apt-get autoremove -yqq --purge 
  && apt-get clean 
  && rm -rf /var/lib/apt/lists/*
USER airflow
RUN pip install --no-cache-dir --user h2o

docker-compose.yaml:

---
version: "3"
x-airflow-common: &airflow-common
  build: ./airflow
  environment: &ref_0
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:[email protected]/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:[email protected]/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: ""
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: "true"
    AIRFLOW__CORE__LOAD_EXAMPLES: "true"
    AIRFLOW__API__AUTH_BACKEND: airflow.api.auth.backend.basic_auth
    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
    AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
    AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
    AWS_SESSION_TOKEN: ${AWS_SESSION_TOKEN}
    AWS_ROLE_ARN: ${AWS_ROLE_ARN}
    REGION_NAME: ${REGION_NAME}
    PYTHONPATH: ${PYTHONPATH} # Set PYTHONPATH
    # rest is not included

When I check the packages installed using docker exec -it <CONTAINER_ID> pip list, h2o cannot be found and I already have the package included in the requirements.txt.

Directory structure:

├── airflow
│   └── Dockerfile
├── dags
│   ├── 01_lasic_retraining_overview.py
│   ├── 02_lasic_retraining_sagemaker_autopilot.py
│   ├── 03_lasic_retraining_h20_automl.py
│   ├── __init__.py
│   └── common
│       ├── __init__.py
│       └── helper.py
├── docker-compose.yaml
├── requirements.txt

Error confirmation inside of airflow:

enter image description here

Source: Docker Questions

LEAVE A COMMENT