Deploying multiple Dask workers on Azure

  azure, dask, dask-distributed, docker, docker-compose

I have a Dask application that works fine in my laptop, and I need to deploy it on Azure.

The image is pushed to the Azure registry, and using the Azure context I’m trying to run docker compose:

docker compose up --scale worker=2

Problem is that the command waits 900 seconds and then fails:

C:daskdiogo>docker compose up --scale worker=2
[+] Running 1/4
 - Group diogo  Created    7.3s
 - scheduler    Creating   900.8s
 - worker       Creating   900.8s
 - web          Creating   900.8s
Future#WaitForCompletion: context has been cancelled: StatusCode=200 -- Original Error: context deadline exceeded

What is this error and how to fix it? How to deploy multiple Dask workers on Azure?

This is the docker compose file:

version: "3.0"
services:

  web:
    image: sofacr.azurecr.io/pablo:job2_v1
    volumes:
      - daskvol:/code/defaults_prediction      
    ports:
      - "5000:5000"
    environment:
      - DATASET_PATH=/data/lc_loan_balances4.csv
      - MODEL_PATH=/model/rfc_model.pkl
      - PREPROCESSING_PATH=/model/data_columns.pkl
      - SCHEDULER_ADDRESS=scheduler
      - SCHEDULER_PORT=8786
    working_dir: /code
    entrypoint:
      - /opt/conda/bin/waitress-serve
    command:
      - --port=5000
      - defaults_prediction:app

  scheduler:
    image: sofacr.azurecr.io/pablo:job2_v1
    ports:
      - "8787:8787"
    entrypoint:
      - /opt/conda/bin/dask-scheduler

  worker:
    image: sofacr.azurecr.io/pablo:job2_v1
    depends_on:
      - scheduler
    environment:
      - PYTHONPATH=/code
      - DATASET_PATH=/data/lc_loan_balances4.csv
      - MODEL_PATH=/model/rfc_model.pkl
      - PREPROCESSING_PATH=/model/data_columns.pkl
      - SCHEDULER_ADDRESS=scheduler
      - SCHEDULER_PORT=8786
    volumes:
      - daskvol:/code/defaults_prediction
      - daskdatavol:/data
      - daskmodelvol:/model
    entrypoint:
      - /opt/conda/bin/dask-worker
    command:
      - scheduler:8786
      
volumes:
  daskvol:
    driver: azure_file
    driver_opts:
     share_name: daskvol-0003
     storage_account_name: sofstoraccount
  daskdatavol:
    driver: azure_file
    driver_opts:
     share_name: daskdatavol-0003
     storage_account_name: sofstoraccount
  daskmodelvol:
    driver: azure_file
    driver_opts:
     share_name: daskmodelvol-0003
     storage_account_name: sofstoraccount

Source: Docker Questions

LEAVE A COMMENT