I tried to run a scheduler and workers docker containers on Amazon’s ECS.
I’m using this example:
The scheduler works perfectly, I successfully connected to it from my local machine:
distributed.scheduler - INFO - Remove client Client-0ae5b0fa distributed.scheduler - INFO - Close client connection: Client-0ae5b0fa distributed.scheduler - INFO - Remove client Client-0ae5b0fa distributed.core - INFO - Starting established connection distributed.scheduler - INFO - Receive client connection: Client-0ae5b0fa
I try to run the worker the same way, with this command:
The worker is writing these logs and exits:
+ exec 'dask-worker tcp://SCHEDULER_PUBLIC_IP:8786' /usr/bin/prepare.sh: line 30: /dask-worker tcp://SCHEDULER_PUBLIC_IP:8786: No such file or directory + '[' '' ']' no environment.yml + '[' -e /opt/app/environment.yml ']' + echo 'no environment.yml' + '[' '' ']' + '[' '' ']'
I expected the worker to connect to the scheduler, because the same commands worked when I tried them on an EC2 instance. Also, I tried doing this with all ports open to tcp connections and still nothing.
- Dask docker container version: 6bfa3b19b4be (1 AUG 2021) (latest)
- Fargate version: 1.4.0 (latest)
- Container has 2 vCPUs, 4 Gb memory
Source: Docker Questions