Category : aws-batch

My AWS batch jobs are stuck in runnable state for past 3-4 days. So i tried debugging into it, and i realized that batch jobs are not actually joining the underlying Amazon Elastic Container Service (Amazon ECS) cluster. So i logged into EC2 instance and ran sudo service docker status and it throws following error: ..

Read more

We have python code which does the following inside a docker container import boto3 import tarfile s3 = boto3.client(‘s3’) s3.download_file("dev-bucket", "test/model.tar.gz", "/opt/ml/model/model.tar.gz") tar = tarfile.open("/opt/ml/model/model.tar.gz", ‘r:gz’) tar.extractall(path="/opt/ml/model") However, the job fails while extracting with "OSError: [Errno 30] Read-only file system" . Complete stack trace is : Traceback (most recent call last): > File "inference.py", line ..

Read more

I would like to implement integration tests featuring Metaflow flows; i.e. running a flow from start to finish within a Docker container; and ideally this wouldn’t require substantial rewriting of the flows which contain @batch decorators on specific steps. On the s3 side I can achieve this by setting up a local s3 mocking server ..

Read more

Is is possible to run docker-in-docker on AWS batch? I have tried the approach of mounting the docker socket via the container properties: container_properties = <<CONTAINER_PROPERTIES { "command": ["docker", "run", "my container"], "image": "docker/docker", "jobRoleArn": "my-role", "memory": 2000, "vcpus": 1, "privileged": true, "mountPoints": [ { "sourceVolume": "/var/run/docker.sock", "containerPath": "/var/run/docker.sock", "readOnly": false } ] } However ..

Read more

I have a local dev environment running Docker containers. I have since migrated to using AWS batch to run these jobs and post results in my ‘prod’ environment. I would like to mirror this functionality locally. Essentially set up an endpoint that my local environment can post these jobs and wait for response. I know ..

Read more