AWS Lambda layers were a game changer for me. When used properly, they can reduce deployment time dramatically and, in turn, reduce time to production in Serverless services consisting of multiple AWS Lambda functions.
However, it's often a feature that's misunderstood and eventually not used.
This blog post covers AWS Lambda layers basics, the pros and cons, and my recommended best practices.
My previous post provided technical details and code examples on how to build a Lambda layer with AWS CDK in Python.
How to Build AWS Lambda Functions with External Dependencies
When you create an AWS Lambda function, you need to provide the handler code and its inner modules (logic layer, database handlers, etc.). However, more often than none, your code relies on external dependencies. When your Lambda function runs, it runs on an AWS container base image. The AWS-provided container image might not contain all your external dependencies. In that case, you need to provide them yourself.
You have four options:
Build & upload a Lambda container image that you build yourself, which includes handler code and all external dependencies and upload it to the Amazon ECR repository. See my previous blog on the subject.
Build & upload a ZIP file containing all Lambda code and the external dependencies per the AWS Lambda function. See code examples here.
Build & upload a ZIP file containing external dependencies as an AWS Lambda layer. See code examples and technical explanations here.
Use an existing Lambda layer ARN that contains the external dependencies. See example at the end of my previous post.
While options one and two are reasonable solutions, I'd like to focus on options three and four as they provide a unique optimization opportunity for your Serverless application.
Let's assume that your Serverless application has ten AWS Lambda functions and that your external dependencies weigh about 50 megabytes.
In addition, all external dependencies are shared among all 10 Lambda functions.
Now, let's recall the four options we have for uploading the dependencies to AWS.
In option one, you need to create a docker-compose file per Lambda function, each with its dependencies, entry point, and CMD. It's another resource that you maintain, upload to AWS ECR, and pay for its storage.
The bottom line is unless you consume dependancies that have an unzipped size of more than 250MB, I don't see any advantage to this mechanism.
In option two, which is the most common, we create a ZIP file per Lambda function. AWS CDK does it natively using a Docker builder container that creates the ZIP file.
These ZIP files are uploaded to your AWS account as AWS S3 assets.
However, since all ten functions share the same dependencies, Docker will bundle the same dependencies, weighing about 50 MB into the 10 Lambda functions.
That's 500 MB of the same dependencies being uploaded.
That's an extra time to deploy. Assuming you do this per environment; dev, test, stage, production, it can add up to even minutes, depending on your internet speed.
What if there was a way to upload these dependencies just once and accelerate the overall deployment time?
That's where Lambda layers come into play.
AWS Lambda Layers Pros
AWS Lambda layers are uploaded once and can be used by multiple Lambda functions.
Now let's recall the previous use case, where your Lambda functions have shared dependencies adding up to 50 MB.
Let's assume you create a layer out of those dependencies and set your Lambda functions to use it.
With this use case in mind, let's summarize the advantages that you gain by creating the Lambda layer:
Reduce Lambda function ZIP file size. Each Lambda ZIP file contains handler code and inner modules but no external dependencies.
Reduce overall time to production - each lambda ZIP file is smaller and takes less time to upload. Multiply it by the number of Lambda functions times the AWS accounts number (dev, test, production), and you are looking at a much shorter time to deploy from dev account to production. I've see cases where several minutes were shed of the total deployment time.
You can see and edit Lambda function code in the AWS console - A cool side effect. When you open the code section of the AWS Lambda function, you can view the code if the ZIP file size is under 3MB (according to console editor limits).
AWS Lambda Layers Cons
Let's go over Lambda layers cons:
Versioning - layers are versioned, and your function always consumes a specific version.
You can consume up to 5 Lambda layers in one Lambda function.
The total unzipped size of the function and all layers cannot exceed the unzipped deployment package size quota of 250 MB. https://docs.aws.amazon.com/lambda/latest/dg/invocation-layers.html.
You can use an externally built layer, i.e., layers consumed as ARNs and not built in the same project. However, they introduce additional challenges:
They are harder to consume and share - You can use layers created and deployed in a different account. Sharing is more complicated between AWS accounts, and in the end, you depend on an external ARN that you include in your code.
Manual updates - Layers are versioned. The version is part of the ARN. You need to be aware there's a new version and manually go and change the version to the latest version. Lambda layers don't have a package manager like Python and other languages have. Updates, as a result, become a manual endeavor.
Setting up a local developer environment is hard - you need to figure out what external dependencies the layers bring and install them locally to test your code and debug in the IDE. This can prove to be challenging, especially in cases where they are conflicts. A discrepancy between the local developer and Lambda function environments can lead to crashes and bugs.
Security. Using layers as an external ARN can expose you to a security risk as you are never sure what is bundled in the layer. Make sure to use AWS Inspector to mitigate this vulnerability.
AWS Lambda Layers Best Practices
Don't Use Layers with External ARN
As seen in the previous cons segment in section 4, external layers have many disadvantages that make them irrelevant and not worth it.
This brings me to an apparent resolution: build it yourself!
Use AWS Lambda Layers as a Deployment Optimization
The main takeaway is that you should use AWS Lambda layers as deployment optimization and build it in the same code repository that uses the layer.
Leave dependency management and versioning to proper package managers, i.e., in Python, use pip or poetry and use layers as a deployment optimization.
Building the layer in the same repository that uses it will diminish the lambda layer cons numbers 1 and 4. The Lambda functions will always use the latest version automatically, and every time you update the dependencies, a new layer is created. In addition, the developer environment will always reflect the same dependencies that your layer bundles since they are made from the same dependencies version.
Multiple Layers
Should you build multiple layers in the same service?
I'd argue against it as it brings added complexity and increases deployment time.
I suggest splitting the dependencies into multiple layers only if one function requires a drastically larger package (which increases cold start); otherwise, use your judgment and analyze the cold start impact with/without the extra dependency.
You can use AWS X-Ray tracing capabilities; read more here.
Security
And lastly, security. If you are still required to use Lambda layers by ARN, ensure you enable AWS inspector on your Lambda function and layers.
AWS Inspector will scan your layers for vulnerabilities and notify you if you need to take action.
Click here for more details.
AWS CDK Example
Want to learn how to build Lambda layers with AWS CDK? checkout my previous blog post.
Are you interested in a fully working CDK Serverless project that uses Lambda layers?
Checkout my open source template project: https://github.com/ran-isenberg/aws-lambda-handler-cookbook