D
Daniil Romashov
Guest
Daniil Romashov, Site Reliability Engineer (SRE)
A single client in a loop could potentially consume all of Lambda's concurrency. Therefore, we've implemented IP address-based rate limiting on the CloudFront edge. The number was taken from one week of logs. We then spent 16 days making final adjustments, including the test runner, a second route, notifications, and CI permissions.
The access token issuance route is behind CloudFront. The request is then sent to API Gateway, which in turn sends it to Lambda. This lambda function has 30 concurrent executions reserved per region. One client is stuck in a loop on the route, sending approximately 20 requests per second and consuming the entire budget. The remaining users are receiving 429 responses from Lambda. Without tokens, nothing will work.
I wrote this as a comment at the beginning of the waf.tf file. Then I started writing the rule itself. It was September 15, 2026.
There was only one thing we needed: one address shouldn't take everything. In other words, it's IP address rate limiting. And it's best to set that up at the edge, that is, in CloudFront, in front of API Gateway and Lambda.
I collected CloudFront logs for seven days, from September 8th to 15th. The logs contained 8,921 client addresses. For each address, the number of token path requests was counted at 5-minute intervals to identify peaks.
The most active regular address made 180 requests in 5 minutes. I also looked at the NAT address used for server rendering of the European dashboard. All the European users sit behind it. I was waiting for a big number there. The peak was 46. Abusive clients made from 468 to 7383 requests in 5 minutes. From 295 to 3888 in one minute.
There is a gap between 180 and 468. 300 fits into that gap. That is one request per second if it continues for a long time. A typical client wouldn't reach that point. The headroom over the most active of them is 1.67 times. The weakest of the bad ones is over the limit by 1.56 times. The strongest is 24.6 times.
The window is 5 minutes. The limit and window size are moved into locals, next to the log comments.
The repository does not contain information on how the logs are calculated. Neither the scripts nor the raw logs are available. Only the final figures are in the comment.
Rule type: rate_based_statement. IP aggregation key. This is the source address as seen from CloudFront. It does not use a forwarded address header and does not have its own custom key. The scope of the web ACL is CLOUDFRONT. Therefore, the ACL resides in the us-east-1 region and is linked to the distribution via
Not all requests are counted. The scope-down filter is constructed using and_statement and consists of three conditions. The path will be converted to lowercase and then match exactly. The method is GET. As will be explained later, the source address is not included in the QA runner's IP address set. Everything else is ignored, and the default action is "allow".
Names have been simplified here, and filter conditions are summarized in the comments.
If blocked, the WAF returns a 403 response by default. I configured a 429 response using custom_response. I did not specify a response body. The client will see the same code returned by the overloaded Lambda function: "Too many requests."
The general procedure for a WAF is as follows: First, set the rule to COUNT mode, monitor it for a week, and then enable blocking. I didn't do that. The first version of the rule included the block action and 429. None of the file versions have a count mode. The trial period has been replaced by the logs. I cannot judge whether this was sufficient.
There are no complete WAF logs because the
The alarm appeared 22 minutes after the rule. The metric is BlockedRequests in the AWS/WAFV2 namespace. Sum for 300 seconds, one period, and the threshold is greater than 0. One request is blocked in 5 minutes, and the alarm fires. It will clear by itself. ok_actions will be sent to the same SNS topic. Missing data is considered normal.
The catch lies in the Rule dimension. The metric_name from the rule's visibility_config is written there. The rule name does not work there, so I've left a note about this as a comment within the line.
To ensure notifications were delivered, I had to edit another repository. I added the sns
ublish permission for cloudwatch.amazonaws.com with the aws:SourceAccount condition to the SNS topic policy. This commit went in 7 seconds after the alarm commit.
Subsequently, the CI plan failed. The CI user was unable to read the SNS topic list. I removed the topic lookup through data and recorded the ARN as a string. I added 11 wafv2 actions on web ACLs and 6 actions on CloudWatch alarms for the CI user. The first commit with this rule was at 2:05 PM local time. All updates on the main branch were completed that same day, within two hours.
On September 24, E2E tests hit the limit at both the stage and the production site. All E2E tests, both browser-based and Node.js-based, come from the same address. A single test page requests tokens from within multiple web workers.
It was also possible to whitelist the runner's address and completely exclude it from the count. I did it a different way. The main rule uses not_statement to count everyone except the runner's set of IP addresses. A twin rule is placed nearby. It counts only this set, with the limit of 3000 in 5 minutes. That's 10 times the normal amount. The rules do not overlap. The runner has its own counter, which is not completely unlimited. The remaining clients still stay on 300.
It's not clear from the repository whether the tests stopped failing after the 3000 limit. The end-to-end pipeline history is in a different project.
A colleague, a frontend and infrastructure engineer, ported the rule to the stage. As mentioned in the comments, he mirrored the production rule. There are no alarms on the stage. A known client bug is already blocked there, and it is under investigation. Until the front-end bug is fixed, an alarm would be nothing more than noise.
On September 30th, the limit was put on the second route, GET requests to the marketplace profile. Again the CloudFront logs for the 7-day period from September 23rd to 30th. The most active address on the live site generated 60 requests in 5 minutes. On the stage 126, the QA runner 113. I set the same limit values, 300 and 3000. The rules and counters are separate. A second alarm has been added on the production site.
Each ACL has four rules with priority levels of 10, 20, 30, and 40. There are two ACLs. There are a total of 8 rate-based rules and 2 IP address sets.
Edge restrictions are not very effective if you can access the API directly and bypass CloudFront. On the stage this was closed by a colleague, not me. CloudFront adds a secret header to each request to the API origin. On the API Gateway side, a regional WAF rule allows GET of the token route only with this header.
Terraform automatically generates the secret, a 44-character random_password that does not contain special characters. This is stored in SSM as a SecureString. The lifecycle is set to ignore_changes on the value. Otherwise, manual rotations will be rolled back by the next apply. Rotation is manual. Add the new value to SSM, then update the WAF rule on the API side, and then deploy both.
My responsibility was CI. The stage plan failed on
Going back even further, on September 22nd, I gave the media services CI permissions on regional web ACLs with a specific suffix in the name, and permission to bind them to API Gateway stages. The regional rule itself lives in the media services code. I did not read it when writing this article.
The operation in the production environment cannot be verified from the repository. The number of blocked requests, addresses, and the number of times alarms were triggered are all stored in CloudWatch and the alarm history. I did not go there for this article. Furthermore, I have not confirmed whether the Lambda 429 error has been resolved for general users. And that was the whole reason for it. Furthermore, we still don't know who the abusive clients were.
This rule protects against a client falling into an infinite loop. Millions of requests were blocked by CloudFront and failed to reach Lambda. During this period, the limit of 300 was never changed.
Our own clients were the first to have their access blocked. On the stage, the issue was a frontend bug that was requesting tokens too frequently. The QA runner hit the limit in both the staging and production environments. I fixed the bug and added the runner to the whitelist. Now, the WAF only fires when actually subjected to a DDoS attack.
Therefore, the conclusion is as follows: A limit with an alarm on blocks catches not only attacks but also your own bugs. Before concluding that the block is an attack, check whose address it is
A single client in a loop could potentially consume all of Lambda's concurrency. Therefore, we've implemented IP address-based rate limiting on the CloudFront edge. The number was taken from one week of logs. We then spent 16 days making final adjustments, including the test runner, a second route, notifications, and CI permissions.
30 for everyone
The access token issuance route is behind CloudFront. The request is then sent to API Gateway, which in turn sends it to Lambda. This lambda function has 30 concurrent executions reserved per region. One client is stuck in a loop on the route, sending approximately 20 requests per second and consuming the entire budget. The remaining users are receiving 429 responses from Lambda. Without tokens, nothing will work.
I wrote this as a comment at the beginning of the waf.tf file. Then I started writing the rule itself. It was September 15, 2026.
There was only one thing we needed: one address shouldn't take everything. In other words, it's IP address rate limiting. And it's best to set that up at the edge, that is, in CloudFront, in front of API Gateway and Lambda.
Where did the number 300 come from?
I collected CloudFront logs for seven days, from September 8th to 15th. The logs contained 8,921 client addresses. For each address, the number of token path requests was counted at 5-minute intervals to identify peaks.
The most active regular address made 180 requests in 5 minutes. I also looked at the NAT address used for server rendering of the European dashboard. All the European users sit behind it. I was waiting for a big number there. The peak was 46. Abusive clients made from 468 to 7383 requests in 5 minutes. From 295 to 3888 in one minute.
There is a gap between 180 and 468. 300 fits into that gap. That is one request per second if it continues for a long time. A typical client wouldn't reach that point. The headroom over the most active of them is 1.67 times. The weakest of the bad ones is over the limit by 1.56 times. The strongest is 24.6 times.
The window is 5 minutes. The limit and window size are moved into locals, next to the log comments.
The repository does not contain information on how the logs are calculated. Neither the scripts nor the raw logs are available. Only the final figures are in the comment.
Rule
Rule type: rate_based_statement. IP aggregation key. This is the source address as seen from CloudFront. It does not use a forwarded address header and does not have its own custom key. The scope of the web ACL is CLOUDFRONT. Therefore, the ACL resides in the us-east-1 region and is linked to the distribution via
web_acl_id.Not all requests are counted. The scope-down filter is constructed using and_statement and consists of three conditions. The path will be converted to lowercase and then match exactly. The method is GET. As will be explained later, the source address is not included in the QA runner's IP address set. Everything else is ignored, and the default action is "allow".
Code:
rule {
name = "rate-limit-token"
priority = 10
action {
block {
custom_response {
response_code = 429
}
}
}
statement {
rate_based_statement {
limit = local.waf_token_rate_limit # 300
evaluation_window_sec = local.waf_token_rate_window # 300
aggregate_key_type = "IP"
scope_down_statement {
and_statement {
# uri_path EXACTLY <token path>, LOWERCASE
# method EXACTLY "GET"
# not_statement { ip_set_reference_statement { QA runner } }
}
}
}
}
visibility_config {
cloudwatch_metrics_enabled = true
metric_name = "acl-rate-limit-token"
sampled_requests_enabled = true
}
}Names have been simplified here, and filter conditions are summarized in the comments.
If blocked, the WAF returns a 403 response by default. I configured a 429 response using custom_response. I did not specify a response body. The client will see the same code returned by the overloaded Lambda function: "Too many requests."
Things I didn't do
The general procedure for a WAF is as follows: First, set the rule to COUNT mode, monitor it for a week, and then enable blocking. I didn't do that. The first version of the rule included the block action and 429. None of the file versions have a count mode. The trial period has been replaced by the logs. I cannot judge whether this was sufficient.
There are no complete WAF logs because the
aws_wafv2_web_acl_logging_configuration resource does not exist in the repository. CloudWatch metrics and request sampling are enabled for each rule and ACL itself. Blocking status needs to be analyzed on a sample-by-sample basis.Alarm
The alarm appeared 22 minutes after the rule. The metric is BlockedRequests in the AWS/WAFV2 namespace. Sum for 300 seconds, one period, and the threshold is greater than 0. One request is blocked in 5 minutes, and the alarm fires. It will clear by itself. ok_actions will be sent to the same SNS topic. Missing data is considered normal.
The catch lies in the Rule dimension. The metric_name from the rule's visibility_config is written there. The rule name does not work there, so I've left a note about this as a comment within the line.
Code:
dimensions = {
WebACL = aws_wafv2_web_acl.prod.name
Rule = "acl-rate-limit-token" # the rule metric_name, not the rule name
}To ensure notifications were delivered, I had to edit another repository. I added the sns
ublish permission for cloudwatch.amazonaws.com with the aws:SourceAccount condition to the SNS topic policy. This commit went in 7 seconds after the alarm commit.Subsequently, the CI plan failed. The CI user was unable to read the SNS topic list. I removed the topic lookup through data and recorded the ARN as a string. I added 11 wafv2 actions on web ACLs and 6 actions on CloudWatch alarms for the CI user. The first commit with this rule was at 2:05 PM local time. All updates on the main branch were completed that same day, within two hours.
QA Runner and Twin Rules
On September 24, E2E tests hit the limit at both the stage and the production site. All E2E tests, both browser-based and Node.js-based, come from the same address. A single test page requests tokens from within multiple web workers.
It was also possible to whitelist the runner's address and completely exclude it from the count. I did it a different way. The main rule uses not_statement to count everyone except the runner's set of IP addresses. A twin rule is placed nearby. It counts only this set, with the limit of 3000 in 5 minutes. That's 10 times the normal amount. The rules do not overlap. The runner has its own counter, which is not completely unlimited. The remaining clients still stay on 300.
It's not clear from the repository whether the tests stopped failing after the 3000 limit. The end-to-end pipeline history is in a different project.
Stage and second route
A colleague, a frontend and infrastructure engineer, ported the rule to the stage. As mentioned in the comments, he mirrored the production rule. There are no alarms on the stage. A known client bug is already blocked there, and it is under investigation. Until the front-end bug is fixed, an alarm would be nothing more than noise.
On September 30th, the limit was put on the second route, GET requests to the marketplace profile. Again the CloudFront logs for the 7-day period from September 23rd to 30th. The most active address on the live site generated 60 requests in 5 minutes. On the stage 126, the QA runner 113. I set the same limit values, 300 and 3000. The rules and counters are separate. A second alarm has been added on the production site.
Each ACL has four rules with priority levels of 10, 20, 30, and 40. There are two ACLs. There are a total of 8 rate-based rules and 2 IP address sets.
Bypassing CloudFront
Edge restrictions are not very effective if you can access the API directly and bypass CloudFront. On the stage this was closed by a colleague, not me. CloudFront adds a secret header to each request to the API origin. On the API Gateway side, a regional WAF rule allows GET of the token route only with this header.
Terraform automatically generates the secret, a 44-character random_password that does not contain special characters. This is stored in SSM as a SecureString. The lifecycle is set to ignore_changes on the value. Otherwise, manual rotations will be rolled back by the next apply. Rotation is manual. Add the new value to SSM, then update the WAF rule on the API side, and then deploy both.
My responsibility was CI. The stage plan failed on
ssm:GetParameter. I added seven SSM parameter actions and ssm:DescribeParameters to the CI user.Going back even further, on September 22nd, I gave the media services CI permissions on regional web ACLs with a specific suffix in the name, and permission to bind them to API Gateway stages. The regional rule itself lives in the media services code. I did not read it when writing this article.
Things we don't know
The operation in the production environment cannot be verified from the repository. The number of blocked requests, addresses, and the number of times alarms were triggered are all stored in CloudWatch and the alarm history. I did not go there for this article. Furthermore, I have not confirmed whether the Lambda 429 error has been resolved for general users. And that was the whole reason for it. Furthermore, we still don't know who the abusive clients were.
Result
This rule protects against a client falling into an infinite loop. Millions of requests were blocked by CloudFront and failed to reach Lambda. During this period, the limit of 300 was never changed.
Our own clients were the first to have their access blocked. On the stage, the issue was a frontend bug that was requesting tokens too frequently. The QA runner hit the limit in both the staging and production environments. I fixed the bug and added the runner to the whitelist. Now, the WAF only fires when actually subjected to a DDoS attack.
Therefore, the conclusion is as follows: A limit with an alarm on blocks catches not only attacks but also your own bugs. Before concluding that the block is an attack, check whose address it is