Took from here.
In present, I’m dealing a lot with AWS components and with AWS Lambda particularly. AWS Lambda supports several languages. There are five of them at the moment: Node.js, Java, Python, C# and Go. In my team we’re using Python.
As far as there are still two in-use versions of Python and each project requires its own set of libraries, a standard way for managing these dependencies is using the so-called virtual environment.
Traditionally, virtual environment was being created using the following approach:
The first command creates a clean distribution with Python3 in local directory
The second one activates virtual environment: changes environment variable
VIRTUAL_ENV, and removes
After that, commands
python3 works in context of distribution in directory
There is a
deactivate command to deactivate virtual environment and to return environment variables in their initial state.
The next step is installing libraries.
If a project is distributed in the form of a library, all required dependencies are specified in
For projects which are not libraries, dependencies are specified in special plain-text files.
They can be named in any way, but usually it’s
requirements.txt for general dependencies and
requirements-dev.txt for development dependencies.
Deploy AWS Lambda
To deploy the lambda source code, it must be packed to zip-archive within all dependencies. It’s pretty simple to do if everything is located in the same directory. We used the following commands for this:
After that it’s easy to pack the content of
_build directory and upload it to AWS S3.
The other way is to use AWS Serverless Application Model (SAM) extension, which automatizes and simplifies the process.
It’s actually what we’re using, but it’s another topic.
So, what’s wrong?
The approach described above leads to one serious pitfall and to some minor ones.
For specifying dependencies there is a special standard PEP 440, which allows to use conditional dependency versions, instead of fixing it to particular one. Usually, a version which provides backward-compatibility is used.
The problem is that in this case the latest available version is installed.
So, for equal conditions, two different in time invocations
pip install can lead to different results.
As for lambda, versions of libraries which are installed locally and versions of libraries which are packed into a zip can differ. Because these two installations happens in different moments of time. It can be a reason of hidden bugs and compatibility errors on cloud-side.
To solve this problem, dependency managers for some other languages uses the so-called lock-files.
For example, Composer for PHP uses
composer.lock, and NPM for Node.js uses
These files are generated and updated automatically during the installation of new dependencies or during the update of existing ones.
Lock-files contains information about specific versions of dependencies.
It allows the developer to be sure that installing dependencies from a particular lock-file produces the same result in different moments of time.
Usually, the information for checking consistency is stored along with the versions.
It helps to avoid installing spoofed version of libraries.
A minor issue of the traditional approach is that each virtual environment have to be created and activated manually. Moreover, it’s not clear which version of Python the developer should use.
Another minor issue is the manually-managed dependencies in
There was a situation when a developer installed the dependency locally,
but forgot to add it in the requirements file, which caused failing builds on other systems.
The pipenv project is aimed to get rid of these problems from the developer.
Pipfile as the dependency configuration file and
Pipfile.lock as the lock-file.
Pipfile already exists in a project, setting up a virtual environment and installing dependencies can be done with a single command:
If you want to set up a development environment, just add
If there is
Pipenv.lock, exact versions of dependencies are installed.
Pipenv + AWS Lambda
The pipenv tool is relatively new and is still under active development.
It doesn’t provide some features which are provided by
Some existing features could be removed in the future (for example).
For building a package for AWS Lambda it’s critical to have an ability to install dependencies to particular directory.
Pipenv doesn’t provide this feature out-of-the-box.
But, as usual, there are workarounds.
One of them is to write small script which copies sources of dependencies from virtual environment directory to target directory.
Another option is to generate
requirements.txt and install dependencies with
The second option can be achieved with one-liner (there are two commands, though):
pipenv lock -rgenerates a list of dependencies, which is compatible with the syntax of
pip install --target _build -r fileinstalls dependencies from a specified file to a target directory
pipenv runis required to run
pipinside virtual environment
Configuring virtual environment (in project root):
Build and deploy package for AWS Lambda:
The commands above could be placed to Makefile to be run with one command.