AWS Lambda layers were a game changer for me. When used properly, they can reduce deployment time dramatically and, in turn, reduce time to production in Serverless services with multiple AWS Lambda functions.
However, it's often a feature that's misunderstood and eventually not used.
This blog post covers AWS Lambda layers basics and how to build them with Docker & AWS CDK in Python.
A complete & working code can be found at my open source template project: https://github.com/ran-isenberg/aws-lambda-handler-cookbook
What is a Lambda Layer
AWS Lambda layers provide a convenient way to package libraries and other dependencies you can use with multiple Lambda functions.
A layer is a .zip file archive that can contain additional code or data. A layer can contain libraries, a custom runtime, data, or configuration files. Layers promote code sharing and separation of responsibilities so that you can iterate faster on writing business logic - AWS
Build a Layer with AWS CDK & Docker in Python with Poetry
Let's assume that our Serverless service project uses 'poetry' for dependency management and that the pyproject.toml file looks like this:
Our Goal
We want to bundle AWS Lambda Powertools, mypy-boto3-dynamodb other libraries under the [tools.poetry.dependencies] section into a Lambda layer to be consumed by all our lambda functions and deployed with AWS CDK.
Assumptions
You installed Docker.
You installed AWS CDK.
You use poetry as your Python dependency manager.
You use just one general pyproject.toml file for all the AWS Lambda functions. It includes all the requirements of all functions.
Prepare a .build Folder for CDK Deploy
Before we write any CDK code, we need to prepare the layer build folder.
We will store all the required build artifacts in a new folder: the '.build' folder, which will not be part of the code base. We will keep a folder for the Lambda layers artifacts and another folder that contains the service Lambda function code, ready to be packaged by AWS CDK.
Let's start with the layer.
First, we need to create a requirements.txt file from our pyproject.toml that will be used to create the layer itself. In this example, we use poetry, but pipenv is a valid option too.
We generate a requirements.txt from the [tool.poetry.dependencies] section of the toml file.
Unlike the [tool.poetry.dev-dependencies] section in the toml, the libraries in the [tool.poetry.dependencies] section are the libraries that the Lambda functions in the project require in runtime and must be uploaded to AWS.
Run the following commands to create the .build/common_layer folder and to generate the layer requirements.txt file inside it. As mentioned above, these files are not part of the code base and we use the .build folder to store custom build & deploy artifacts.
The poetry command will export the non development dependencies to the requirements.txt file.
The commands produce the following requirements.txt file in the '.build/common_layer' folder. The name 'common_layer' represents the layer purpose, the common usage Lambda layer in our service that all functions use.
Now, we prepare the Serverless service Lambda functions .build folder.
We can do this by running the following commands:
In our project, the Lambda function handlers code reside in the project's root folder under the 'service' folder. Before we deploy the Lambda functions to AWS, we copy the 'service' folder to the '.build/lambdas' folder, which will packed as a zip and uploaded to AWS S3 by the CDK code. The CDK code that creates the Lambda functions will take the Lambda handler code from the '.build/lambdas' folder.
We do this to preserve the import hierarchy in both AWS CDK and the Lambda function container environment after its deployment to AWS and to prevent runtime import issues such as:
This error occurs when building the function without copying it first to .build/lambdas but copying the code directly from the 'service' folder at the root level.
AWS CDK Code
Now that everything is in place let's write a CDK construct that creates a Lambda layer based on the prebuilt requirements.txt file and a Lambda function in Python 3.9 that uses the layer.
AWS CDK will spin up a Lambda layer builder container using Docker.
The container will look for a requirements.txt file, download all required libraries, zip them and upload it to AWS.
Once the layer object is created, you can add it to any Lambda function.
The AWS CDK construct 'MyConstruct' creates:
Basic Lambda role (lines 17-25).
Lambda layer (lines 27-34). CDK looks for a requirements.txt file in the entry folder ('.build/common_layer') and uses Python 3.9 as runtime compatibility.
Lambda function (lines 36-46). The function uses the layer in line 45 that it receives as an argument. The handler code and other modules are taken from the code folder '.build/lambdas.'
Deploy Time!
Now we can run CDK deploy and sit back while the resources are created.
We run CDK deploy command (and point it to the app.py file in the project) combined with the previous commands that create the .build folders.
In the AWS Lambda handler cookbook Serverless template project, we use a makefile to abstract away such details. Running 'make deploy' will run these three command together.
See the makefile here.
Use External ARN Layer
If you wish to use an externally built Lambda layer you can swap the '_build_common_layer' implementation to the following:
In this case we build a public layer built by the amazing AWS Lambda Powertools team.
Notice how we select a specific version (17) as part of the ARN.
Stay tuned for my next blog post where I present the AWS Lambda layer best practices.