Dask Worker not reachable from scheduler on EC2 Instance

I have a dask setup on AWS as follows:

  • Dask Scheduler on an EC2 Instance.
  • Dask worker inside a docker container within another EC2 Instance.

Both of the EC2 instances are on the same VPC. The problem is just spawning the dask-worker on the container with just the ip of the scheduler doesn’t work as, although the workers can complete the task, the scheduler can’t reach the worker to fetch the result. Thus I keep getting:

distributed.client - WARNING - Couldn't gather 1 keys, rescheduling

I have now changed the network_mode of the worker container to awsvpc and added listen-address and contact-address to set to the IP address allotted to the container in the VPC.

dask-worker tcp://scheduler-host:scheduler:port --listen-address tcp://:1234 --contact-address tcp://$(hostname -i):1234

But I still keep getting the same error when trying to fetch the result of a future. What am I doing wrong?

NOTE: The IP address allotted to the container is reachable (ping) from the scheduler instance.

Source: Docker Questions

LEAVE A COMMENT