How do I specify what version of CUDA libraries my code should be looking for in this container?

  cuda, docker, pytorch

So my workflow is gonna be a bit wonky, but I’m not permitted to use Docker, so I have Singularity instead.

I’m running some code that is giving me this error:

RuntimeError: nvrtc: error: failed to open libnvrtc-builtins.so.11.1.
  Make sure that libnvrtc-builtins.so.11.1 is installed correctly.
nvrtc compilation failed:

If more details are needed, I can provide.

I’m operating in a Singularity container built from a docker_hub image I created here: https://hub.docker.com/repository/docker/elavizadeh/imaginaire_cuda_10.2/builds
(Ignore the fact that the title says cuda_10.2, I’ll fix this all up if and when this works.)

Which is based on this dockerfile I created here: https://github.com/EvanLavizadeh/imaginaire-1/blob/master/Dockerfile

In that Dockerfile, I specify a base image "FROM nvcr.io/nvidia/pytorch:20.12-py3"
https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel_20-12.html#rel_20-12

I specifically chose that version of that base image because it claims it has Cuda toolkit 11.1.1 which I am hoping will contain libnvrtc-builtins.so.11.1.

But when I spin up my container, cuda reports version 11.0.221:

Singularity> nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0

Additionally, when I look at the files in /usr/local/cuda/lib64, I see libnvrtc-builtins.so.11.0 but not libnvrtc-builtins.so.11.1

I believe this is the source of my error but I don’t know how to correct it. I either need my code to use libnvrtc-builtins.so.11.0 instead of libnvrtc-builtins.so.11.1 or I need my container to build Cuda 11.1 and hopefully generate libnvrtc-builtins.so.11.0.

Any advice?

Source: Docker Questions

LEAVE A COMMENT