Category : dask-distributed

I followed these instructions to deploy a Dask cluster on Kubernetes/Minikube with Helm. I installed and the deployed with the following command: helm install dask-chart dask/dask Running kubectl get services I see the scheduler, however the EXTERNAL-IP is none and I cannot connect to the scheduler: NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dask-chart-scheduler ClusterIP 10.107.222.251 ..

Read more

I have the Dask code below that submits N workers, where each worker is implemented in a Docker container: client.upload_file(‘/code/app/worker.py’) default_sums = client.map(process_asset_defaults, build_worker_args(req, numWorkers)) future_total_sum = client.submit(sum, default_sums) total_defaults_sum = future_total_sum.result() The problem is that in a development environment when I change the worker’s code I need to restart all the containers manually for ..

Read more

I need to run a scikit-learn RandomForestClassifier with multiple processes in parallel. For that, I’m looking into implementing a Dask scheduler with N workers, where the scheduler and each worker run in a separate Docker container. The client application, that also runs in a separate Docker container, will first connect to the scheduler and initiate ..

Read more