Naming Dask Worker with Docker Swarm Templating

  dask, dask-distributed, docker, docker-compose, docker-swarm

I’m currently using Docker Swarm to deploy/manage multiple Dask Workers across a cluster. For easier debugging I’d like to be able to name the workers based on what node in the Swarm it is running on.

The dask-worker command has a --name parameter, however, Docker’s templating doesn’t seem to work in the entrypoint or cmd options. e.g.

...
  worker:
    image: myapp:latest
    restart: always
    entrypoint: ["dask-worker", "tcp://scheduler:8786", "--name", "{{.Node.Hostname}}"]
    deploy:
      mode: global
...

Unfortunately, the {{.Node.Hostname}} templating only appears to work in the environment section of a docker-compose.yml file. So my next option was to try and set it via an environment variable like this:

...
  worker:
    image: myapp:latest
    restart: always
    entrypoint: ["dask-worker", "tcp://scheduler:8786"]
    environment:
      DASK_DISTRIBUTED__WORKER__NAME: "{{.Node.Hostname}}"
    deploy:
      mode: global
...

I’ve also been unsuccessful with this, as I assume the workers name cannot be set via an environment variable – though I’ve not been able to find exhaustive documentation of all the supported environment variable config names for dask, so could be a typo or incorrect guess.

Finally, I tried to take the environment variable and bring it back into the entrypoint command via bash templating. This also did not work as it appeared that the environment variable was not set at the time this command was evaluated:

...
  worker:
    image: myapp:latest
    restart: always
    entrypoint: ["sh", "-c", "dask-worker tcp://scheduler:8786 --name $FOO"]
    environment:
      FOO: "{{.Node.Hostname}}"
    deploy:
      mode: global
...

I’m out of ideas at this point, and wondering if anyone could figure out a way.

Thanks.

Source: Docker Questions

LEAVE A COMMENT