I work in a cybersecurity company, CyberArk, and security is a concept that is constantly on my mind when I build SaaS services on AWS. It has to be—malicious actors are constantly looking for vulnerabilities, waiting for someone to slip up or for the perfect moment to launch a DDoS attack against your cloud service. As a result, we must prepare and secure our services. We can delegate most of the heavy lifting (at a cost) to AWS.
In this post, you will learn about the AWS Web Application Firewall (WAF), what it is for, tips, and insights for visibility, ownership, governance (and more) to protect your SaaS services.
This is the first of three posts in the series.
In the second post, I will provide AWS CDK code for WAF deployment and API Gateway association.
In the third post, we will review AWS Firewall Manager and how it allows the organization to manage AWS Application Web Firewall ACLs at scale in a centralized manner.
Table of Contents
You've Got to Secure Your SaaS Application
DDoS attacks, whether deployed by individuals or botnets, flood servers with requests and overwhelm them with traffic, which leads to the hosted services and sites being unavailable for users and visitors - Akamai
When you build public APIs or cloud-based applications, malicious actors will eventually attempt to disrupt and damage your customers' experience.
These attacks, known as distributed denial of service attacks (DDoS attacks), are carried out automatically. They are orchestrated by a network of compromised and remotely controlled machines, also known as bots, which are used to flood the target with an overwhelming amount of traffic.
A bot is a computer program that automates interactions with web properties over the Internet - CloudFlare
However, bots don't always launch full-scale attacks. They access sites for various reasons, including indexing (think about the Google search engine), which makes them "good" bots.
A bad bot is a computer program that either tries to steal data by scraping data or probing for vulnerabilities in preparation for an incoming attack. Luckily, we can protect ourselves from malicious bots while allowing the "good" bots to do their work. If you want to learn more about good and bad bots, check out CloudFlare's article.
Blocking DDoS attacks and malicious bots will maintain the integrity and performance of your SaaS service. Let's see how we can do that with AWS Web Application Firewall.
If you want to learn more about DDoS and the most famous DDoS attacks in history, check out this article.
AWS Web Application Firewall (WAF) Introduction
AWS WAF is a web application firewall that lets you monitor or block the HTTP(S) requests forwarded to your protected web application resources. You can protect the following resource types:
Amazon CloudFront distribution
Amazon API Gateway REST API
Application Load Balancer
AWS AppSync GraphQL API
Amazon Cognito user pool
AWS App Runner service
AWS Verified Access instance
As you can see from the list above, AWS WAF can protect both your container-based and Serverless services.
To understand how WAF provides an extra layer of security, we first need to understand how it works. Let's see how we can configure WAF to protect our SaaS service.
ACL and Rules
To use WAF with your AWS resources (as defined above), create a WAF access control list (ACL), define its rules, and associate the ACL with the resource you wish to protect.
The associated resources forward incoming requests to AWS WAF for inspection by the web ACL. In your web ACL, you create rules to define traffic patterns to look for in requests and to specify the actions to take on matching requests. - AWS
Each rule has action to run when matched. Action choices include the following:
Allow the requests to go to the protected resource for processing and response.
Block the requests.
Count the requests.
Run CAPTCHA or challenge checks against requests to verify human users and standard browser use.
AWS provides managed rules that are out of the box, and I highly suggest you use them as much as possible. These rules, constantly updated by AWS, are easy to use and there's really no reason to reinvent the wheel.
However, sometimes, we need to use custom rules.
A rule can have multiple conditions with an 'and,' 'or' or 'not' between them. A condition can use a regex to match HTTP header, rate-based element, block traffic originating from specific countries, or even an IP set (for example, block traffic that doesn't originate from your work VPN IP ranges). WAF also has the option to alter headers (transform) before examining them. You have many custom options; just be aware that the more advanced and custom you go, the more WCU you use (see WCU section).
Another important notion to remember is that rules have priority. WAF examines traffic from the top priority to the bottom, and when a rule matches its conditions, it performs the defined action, whatever it may be.
Lastly, you can enable CloudWatch metrics for your WAF rules. I suggest you enable them as they provide valuable insight into whether they make a difference (know what and why you're paying for!). For more information, check the AWS docs.
For other rules-building tips, check out this post and always consult with a security expert.
Now that we have the basics out of the way let's head over to my tips and tricks section, which discusses working with WAF and using it in an enterprise SaaS environment.
AWS WAF Tips & Tricks
In this section, I will provide insights and tips I've gathered using AWS WAF in my organization. The tips span team ownership, organizational governance, logs, and debugging issues in production and ongoing WAF ACL maintenance.
Ownership
Security experts, whether they are internal security architects or external security advisors, play a crucial role in analyzing your service's attack vector and constructing the set of ACL rules. The security architect also has the task of monitoring the overall adoption of WAF across the organization and the rules metrics over time (are they making an impact? do they match?).
Developers, on the other hand, are responsible for translating these rules into AWS CDK or Terraform IaC templates (more on that in the second post of the series). The developers will deploy the WAF ACL, associate it with their service resource (API Gateway), and add custom rules if needed.
If there's a WAF issue, customers are unable to connect to the service, etc., developers can view the WAF logs and traffic and resolve the incident with the help of the security architects.
This approach is a good balance between governance and team independence.
Governance
Speaking of governance, in the next post, we will discuss how to manage these rules in a centralized manner. Meanwhile, it's recommended to use a CDK construct or a Terraform template as a black-box architectural pattern containing all the recommended rules and deploy them across the organization. You can read more about this concept in my AWS.com article with Anton Aleksandrov "streamlining serverless governance by codifying architectural blueprints".
In my third article in the series, we will discuss how we take governance further and utilize AWS Firewall Manager for centralized security governance with AWS WAF.
Adding ACL Rules Over Time
Security changes and adapts over time. Your ACL rules will change. However, adding a new rule (especially with a 'block' action) can damage customers' experience or even block them when misconfigured. As such, having two ACL variants is essential: dev and production. As the name states, dev ACLs are used in the development accounts and are a great testing ground for testing new rules, blocking patterns, or other techniques. Try to think out of the box and simulate customer traffic as much as possible.
Once you are sure the new rules will not alter the customer's experience, add them to the production WAF ACL.
Mistakes can still occur, and unexpected customer flows can be hindered. As such, you need to plan a quick "break glass" method that the developers will train beforehand and know to initiate in case of a production issue. The break glass method can be as simple as a quick GitHub revert to remove or adjust the new ACL rule. Developers must know about a pending WAF change and how to debug it correctly.
Visibility & Logs
Developers must easily understand customers' issues that originate from WAF blocking their requests. Luckily, AWS WAF has logging support, which you must enable per ACL.
There are three possible WAF logging destinations:
S3 bucket
Kinesis Firehose
CloudWatch log group
Options one and two are good options for centralized monitoring, where all logs are sent to one AWS account, stored, parsed, and analyzed. The downside is that it requires you to build a data pipeline for those logs and then manage access for all developers in the organization, as they need means to debug their service production issues.
AWS has recently released a video showing how you can build such a pipeline:
However, my preferred option is option number three, using a local CloudWatch group, as it is simple and allows the developers to use CloudWatch insights and tools to debug the WAF logs on their account, so no special permissions are required. If you choose this option, set log retention time to 7-14 days at max to reduce log storage costs.
For more information, check out AWS documentation.
WAF WCU & Pricing
AWS WAF uses WCUs to calculate and control the operating resources that are required to run your rules, rule groups, and web ACLs. The WCU requirements for a rule group are determined by the rules that you define inside the rule group. The maximum capacity for a rule group is 5,000 WCUs. - AWS
While I'd rather add maximum security and build the perfect ACL with all the rules possible, it's not possible due to the 5000 WCU limit. You need to find the correct balance between security coverage and overall WCU. Try to use AWS-managed rules as much as possible, as they are more WCU-optimized and well-maintained.
Another critical issue is the cost. WAF is expensive as you pay for rules, ACLs, and traffic volume that goes through the ACL. For more details, check out the pricing calculator.
Summary
In this post, I introduce the AWS Web Application Firewall and explain why you should use it. We also covered the basics of ACLs and rules. Lastly, I shared some crucial tips that I've learned from using and updating WAF in a large enterprise.
In the next post in the series, I will provide AWS CDK code for WAF deployment and associate it with an API Gateway.
In that third post, we will review AWS Firewall Manager and how it allows the organization to manage AWS Application Web Firewall ACLs at scale.