Installing python packages from a private repo that works for both macOS and Ubuntu

Works on Ubuntu

This approach works for installing packages from a private repo when using docker-compose on an Ubuntu host system. The short summary is to set up SSH_PRIVATE_KEY as an environment variable that contains the contents of ~/ssh.id_rsa on the host. Next, forward the variable into docker via docker-compose:

docker-compose.yml:

version: '3.1'

services:
  some-service:
    args:
      - SSH_PRIVATE_KEY=${SSH_PRIVATE_KEY}
    ...
  ...

Dockerfile:

FROM python:3.7 as intermediate

WORKDIR /usr/src/app

ARG SSH_PRIVATE_KEY

RUN mkdir /root/.ssh/
RUN echo "${SSH_PRIVATE_KEY}" >> /root/.ssh/id_rsa && chmod 600 /root/.ssh/id_rsa

RUN touch /root/.ssh/known_hosts
RUN ssh-keyscan github.com >> /root/.ssh/known_hosts

RUN pip install --no-cache-dir git+ssh://[email protected]/<...>

However, this approach does not work on macOS because docker doesn’t receive the key. This can be verified by printing it out when running docker-compose build:

...
ARG SSH_PRIVATE_KEY
RUN echo "${SSH_PRIVATE_KEY}"
...

The key will print on Ubuntu but not for macOS. The follwoing error prints for macOS as well:

Command "git clone -q ssh://[email protected]/..." failed with error code 128 in None

Fails on both Ubuntu and macOS

So there is another approach to take and that is to use docker secrets.

docker-compose.yml:

version: '3.1'

secrets:
  ssh_private_key:
    file: ~/.ssh/id_rsa

services:
  some-service:
    secrets:
     - ssh_private_key

Dockerfile:

Replace:

RUN mkdir /root/.ssh/
RUN echo "${SSH_PRIVATE_KEY}" >> /root/.ssh/id_rsa && chmod 600 /root/.ssh/id_rsa

with

RUN mkdir /root/.ssh/ 
RUN ln -s /run/secrets/ssh_private_key /root/.ssh/id_rsa

However, this will not work generating the following error on both operating systems:

Permission denied (publickey).
fatal: Could not read from remote repository.

The expectation is that this approach would at least have worked for Ubuntu. When remoting into the docker image, the contents of /run/secrets/ssh_private_key can be displayed but not so for /root/.ssh/id_rsa. It’s possible the soft link isn’t being made and thus the cause for the failure. ¯_(ツ)_/¯

UNIX SOCKETS

Looking into this further UNIX sockets don’t work properly on Docker for macOS. Which seems to explain why things are working with the first approach listed above in Ubunutu only (I’m still not sure what’s wrong in the second approach).

Is there a solution to this problem that doesn’t involve running docker inside a vm like suggested in the last link above? Preferably a solution that can be shared across Ubuntu and macOS?

Source: StackOverflow