How can I build Python packages needing C on Amazon’s Lambda container?

  amazon-ami, aws-lambda, docker, pip, python-3.x

I am building a container for a Lambda that requires quite a few libraries to execute. One of which is cyquant which requires gcc to build. Others are in a private repository that require netrc credentials. Here’s my Dockerfile

FROM python:3.6 as intermediate
ARG NETRC_PATH
RUN echo "${NETRC_PATH}" > /root/.netrc
COPY requirements.txt /
WORKDIR /pip-packages
RUN pip install --upgrade pip 
RUN pip download -q -r /requirements.txt

FROM public.ecr.aws/lambda/python:3.6
COPY --from=intermediate /pip-packages/ /pip-packages/
RUN pip install --upgrade pip 
#RUN yum group install "Development Tools"
RUN yum install -y gcc gcc72-c++ python36-devel
RUN pip install --no-index --find-links=/pip-packages/ /pip-packages/*
COPY app.py ./
CMD ["app.handler"]

I’m basically just creating an intermediate container to download the pip files so that my credentials are not stored in the final container. I pipe in the NETRC file from my home directory and store it in root’s home for that container.

The build process is done on the lambda container because I figure building on the intermediate and copying the files is likely going to be problematic since there could be any number of differences in the C versions across the base images. Maybe I should build them in the intermediate though, and copy? I’m not sure if all I’d have to do is copy /usr/local/lib/python3.6/site-packages or if I’d need some other files.

I’m getting the follow error when pip installing

cyquant/dimensions.cpp:4:10: fatal error: Python.h: No such file or directory
 #include "Python.h"
          ^~~~~~~~~~
compilation terminated.
error: command 'gcc' failed with exit status 1

I figured yum installing python3-devel would solve this but it had no effect. Every answer I’ve seen on Stack Overflow or blogs says to install this. In fact I "knew" that was the right solution before I looked it up and when it didn’t work I checked and most every post agreed. But alas, it’s not solving the problem.

So how can I get Python.h on this container? Is the lambda/python:3.6 different from Amazon Linux to a degree that it’s not the same package management?

Also should I be doing the build process in the intermediate container instead or is this the right track?

Source: Docker Questions

LEAVE A COMMENT