Docker can’t find log file when starting docker image with environment variables set using docker run, no problem with docker start

  amazon-emr, bash, docker, jupyter, oauth

I’ve been trying to set up EMR with JupyterHub so that it’s configured to use GitHub oauth for login. I have seen in the docs that it’s not supported, but I thought I could see a way to make it work, but now I’m hitting an issue I don’t understand.

I have set up the EMR cluster and in order to configure it to run with oauth, I created a step that is run by AWS scriptrunner which is a bash script designed to update the config, install oauthentication using conda and then stop the container and start it back up using the environment variables that are required for oauth to work.

Once the config is updated, I run this function, which stops and starts the container:

restart_jupyterhub() {
  echo "Stopping JupyterHub"
  sudo docker stop "${JUPYTER_NOTEBOOK_ID}"
  echo "Starting Jupyterhub with new config"

But I get an unexpected error:

/usr/local/bin/jupyterhub: line 6: /var/log/jupyter/jupyter.log: No such file or directory

Then the container quits…

If instead of the docker run step, I use docker start "${JUPYTER_NOTEBOOK_ID}" the container starts fine (and the log file is there, along with the old logs which look like they were rotated out of the way), and I can even see the github oauth login prompt (so I’m fairly sure my config is mostly right), but since there are no environment variables for callback URL etc, it fails… so I know the file is there when I start up the container in this way, but why is it missing when I start a new copy of the image with docker run?

I tried to also change the entrypoint to touch the file first, but this gave me some weird errors about "File not found: unknown" and I gave up with that idea… but I would really take any solution at this point.

I managed to extract the DockerFile using the script here but it didn’t really give me any clue as to why the two different approaches give me a missing file…

I also tried using docker create to create the image first and see if I could start it, but I always seem to get the same issue.

Can anyone suggest why docker reports the file missing in one case, but not the other? I’m hoping it’s just something daft that I can’t see because I have been staring at it for too long…

Source: Docker Questions