Docker overlay2: error walking file system: OSError [Errno 40] Too many levels of symbolic links

  docker, linux

Main app: uvicorn server on starlette (python) webapp

While I was trying to debug the error in the title (troubleshoot log is following below) running the below command at the host’s FS (/var/lib/docker/overlay2/[IMAGE_HASH_FOLDER]

find -L ./ -mindepth 15

I find the files involved in the loop.
Locally is the /usr/bin/X11 and at server I’m getting the following:

error walking file system: OSError [Errno 40] Too many levels of symbolic links: '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stderr'

and then from WatchGodReload:

WARNING: WatchGodReload detected file change in '['/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/3', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stderr', '/dev/fd/3', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stderr', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/4', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/1', '/dev/fd/5', '/dev/fd/6/dev/fd/4', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/2', '/dev/fd/6/dev/fd/6/dev/fd/1', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/1', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stdout', '/dev/fd/6/dev/fd/1', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/2', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/1', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/5', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/5', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/2', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/5', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/4', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/2', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/5', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stdout', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/4', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/3', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/3', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/1', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/5', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/3', '/dev/stdout', '/dev/fd/6/dev/fd/6/dev/fd/2', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/1', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/1', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stderr', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/5', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stdout', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/5', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stderr', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/4', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stdout', '/dev/fd/6/dev/fd/2', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/3', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/4', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/4', '/dev/fd/6/dev/fd/6/dev/fd/3', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stderr', '/dev/fd/6/dev/fd/6/dev/stderr', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/3', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/3', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/1', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stdout', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/4', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/2', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stdout', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/2', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/1', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/1', '/dev/fd/2', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/2', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stderr', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/5', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stderr', '/dev/fd/6/dev/fd/6/dev/stdout', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/3', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/2', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/stdout', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/4', '/dev/fd/6/dev/fd/3', '/dev/fd/6/dev/stdout', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stderr', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stdout', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/3', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/2', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/1', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stdout', '/dev/fd/6/dev/fd/6/dev/fd/4', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/4', '/dev/stderr', '/dev/fd/6/dev/fd/6/dev/fd/5', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/5', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/3', '/dev/fd/6/dev/stderr', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/5', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/2', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stderr', '/dev/fd/6/dev/fd/5', '/dev/fd/4', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/stderr', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/4', '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stdout', '/dev/fd/1']'. Reloading...

Either I run the container with the volume I usually bind:

UPLOADS_PATH=/var/opt/tmp
LOCAL_UPLOADS_PATH=/var/containers/TEST_UPLOADS

docker build --build-arg POSTGRES_USER --build-arg POSTGRES_PASSWORD --build-arg POSTGRES_SERVER --build-arg POSTGRES_DB --build-arg POSTGRES_PORT --build-arg UPLOADS_PATH --build-arg WHERE_AM_I --build-arg SESSION_SECRET --build-arg DO_YOU_WANT_USERS -t test .

docker run -d --name test_container -v ${LOCAL_UPLOADS_PATH}:${UPLOADS_PATH} -p 7000:7000 test

or without the binding, I still get the same error logs & the app is constantly restarting after every request.

How is possible to have such a loop (linked files?) inside the image?

UPDATE

The container was running smoothly until I’ve changed bcrypt library with pybcrypt and uvicorn with its cythonized version.

Much appreciate any suggestions on what to further explore.

P.S. I’ve also tried the docker system prune -a, and although there were deprecated stuff, nothing changed.

P. S. 2: @jordanvrtanoski I’ve separated the question as you’ve suggested.

Source: Docker Questions

LEAVE A COMMENT