- Bruno da Silva Valenga
A big problem when compiling Python libraries to use in our Lambda on AWS is using our computer (windows/macox/Linux) to assemble it. We starting to facing some issues because when you install a python package using pip, for example, Python knows that our machine has some libs that don't need to be recompiled because our machine will offer it to our library.
When we use Lambda, it can be a big issue because AWS's lambda image to run our function doesn't provide the same lib's that our machine offers us.
AWS uses some runtimes to run our Lambda, and you can check it out here. Also, you can set up a customise runtime, but I'll not cover it now.
So, how to solve this problem?
With a simple tool called Docker 😎
If you don't know what's Docker or never used it before, please leave this page now and read: What is a Container? ASAP.
We will use a docker image called lambic/lambda that replicates all the environment runtime where our Lambda runs. The developers define this docker image as a sandboxed local environment that almost identically replicates the live AWS Lambda environment, including installed software and libraries, file structure and permissions, environment variables, context objects and behaviours, and even the user and running process are the same.
They provide a lot of runtimes, not just for Python. You will find runtimes for all the Languages that Lambda offers you. You can check all the runtimes here.
Let's code. The difference: When we install some package, for example, Pillow using our computer with a simple pip install:
$ pip3 install Pillow
After installing the library in a specific local folder, we can see the library source code on the .venv local folder. We can see a standard package, but if we deploy it into our lambda function, we face some problems because our lambda runtime doesn't provide some libs.
$ ~/Projects/lambdas/img/.venv/lib/python3.8/site-packages/PIL > master ❯ ltotal 3640drwxr-xr-x 102 brunodasilvalenga staff 3.2K Oct 3 11:56 .drwxr-xr-x 28 brunodasilvalenga staff 896B Oct 3 11:56 ..-rw-r--r-- 1 brunodasilvalenga staff 2.9K Aug 10 2020 BdfFontFile.py-rw-r--r-- 1 brunodasilvalenga staff 14K Aug 10 2020 BlpImagePlugin.py-rw-r--r-- 1 brunodasilvalenga staff 14K Aug 10 2020 BmpImagePlugin.py-rw-r--r-- 1 brunodasilvalenga staff 1.5K Aug 10 2020 BufrStubImagePlugin.py-rw-r--r-- 1 brunodasilvalenga staff 2.7K Aug 10 2020 ContainerIO.py-rw-r--r-- 1 brunodasilvalenga staff 1.6K Aug 10 2020 CurImagePlugin.py-rw-r--r-- 1 brunodasilvalenga staff 2.2K Aug 10 2020 DcxImagePlugin.py-rw-r--r-- 1 brunodasilvalenga staff 5.3K Aug 10 2020 DdsImagePlugin.py-rw-r--r-- 1 brunodasilvalenga staff 12K Aug 10 2020 EpsImagePlugin.py...
And if we install the Python package using the docker image that we commented on before? Let's try:
docker run -it -w /work -v $(pwd):/work lambci/lambda:build-python3.8 "source .venv/bin/activate && pip3 install -r requirements.txt"
We can see that our folder contains different content, like this package libs.
$ ~/Projects/lambdas/img/.venv/lib/python3.8/site-packages/PIL/.libs > master ❯ ltotal 8664drwxr-xr-x 13 brunodasilvalenga staff 416B Aug 10 2020 .drwxr-xr-x 103 brunodasilvalenga staff 3.2K Oct 3 11:57 ..-rwxr-xr-x 1 brunodasilvalenga staff 1.2M Aug 10 2020 libfreetype-69f25d5e.so.6.17.1-rwxr-xr-x 1 brunodasilvalenga staff 245K Aug 10 2020 libjpeg-3b10b538.so.9.3.0-rwxr-xr-x 1 brunodasilvalenga staff 442K Aug 10 2020 liblcms2-a6801db4.so.2.0.8-rwxr-xr-x 1 brunodasilvalenga staff 212K Aug 10 2020 liblzma-6cd627ed.so.5.2.4-rwxr-xr-x 1 brunodasilvalenga staff 480K Aug 10 2020 libopenjp2-b3d7668a.so.2.3.1-rwxr-xr-x 1 brunodasilvalenga staff 277K Aug 10 2020 libpng16-bedcb7ea.so.16.37.0-rwxr-xr-x 1 brunodasilvalenga staff 646K Aug 10 2020 libtiff-bd1961ca.so.5.5.0-rwxr-xr-x 1 brunodasilvalenga staff 583K Aug 10 2020 libwebp-3a2aeecf.so.7.0.5-rwxr-xr-x 1 brunodasilvalenga staff 29K Aug 10 2020 libwebpdemux-e9ec482e.so.2.0.6-rwxr-xr-x 1 brunodasilvalenga staff 57K Aug 10 2020 libwebpmux-40630b44.so.3.0.4-rwxr-xr-x 1 brunodasilvalenga staff 86K Aug 10 2020 libz-a147dcb0.so.1.2.3
What happened here? Our docker image doesn't provide some SO lib's and Python needs them to compile the library. So Python will compile the libs for us and offer them like Python libs and not more SO libs.
If we update our Lambda with the new packages, we can see that our problem with lib packages was solved 🙌.