Category : dask

I have a question regarding an implementation of a Dockerfile on dask-docker. FROM continuumio/miniconda3:4.8.2 RUN conda install –yes -c conda-forge python==3.8 […] && rm -rf /opt/conda/pkgs COPY prepare.sh /usr/bin/prepare.sh RUN mkdir /opt/app ENTRYPOINT ["tini", "-g", "–", "/usr/bin/prepare.sh"] prepare.sh is just facilitating installation of additional packages via conda, pip and apt. There are two things I ..

Read more

I’m working with a Dask Cluster on GCP. I’m using this code to deploy it: from dask_cloudprovider.gcp import GCPCluster from dask.distributed import Client enviroment_vars = { ‘EXTRA_PIP_PACKAGES’: ‘"gcsfs"’ } cluster = GCPCluster( n_workers=32, docker_image=’daskdev/dask:2021.2.0′, env_vars=enviroment_vars, network=’my-network’, #filesystem_size=150, machine_type=’e2-standard-16′, projectid=’my-project-id’, zone=’us-central1-a’, on_host_maintenance="MIGRATE" client = Client(cluster) Then I read csv files, with the following code: import dask.dataframe ..

Read more

I am running a Docker compose file (docker-compose up –force-recreate) to start Dask scheduler, workers, and the Dask Jupyter notebook. I have added volumes as follows: notebook: image: daskdev/dask-notebook depends_on: – scheduler – worker ports: – "8888:8888" environment: DASK_SCHEDULER_ADDRESS: "tcp://scheduler:8786" volumes: – "~/Personal/python-workspace/notebooks:/home/jovyan/work" – "~/Personal/docker-volumes/tmp:/tmp" Whenever I try to create a file under the work ..

Read more