Both AWS and Cloudflare have released new bot mitigation features into their respective firewall products. Both releases provide additional features for filtering out unwanted bot traffic from reaching the application.
As Sébastien Stormacq, principal developer advocate at Amazon, shares "according to research done by the AWS Shield Threat Research Team, up to 51% of traffic heading into typical web applications originates from scripts running on machines, also known as bots." Bots can range from wanted, such as monitoring scripts or search engine crawlers, to unwanted, such as content scrapers.
The new AWS WAF Bot Control feature analyses the request metadata such as TLS handshakes, HTTP attributes, and IP addresses to identify the bot's purpose. It then categorizes the bots by type such as scraper, SEO, crawler, or site monitor. Once identified, the default action is to block bot traffic from unwanted bots. It is possible to customize the configuration to return a custom response by bot type or flag the traffic for downstream services by inserting a new header.
Two additional features have been added to AWS WAF managed rule groups, scope down statements and labelling. Scope down statements can be used to define what conditions cause a managed rule to run.
Labels are metadata that can be added to a request as a result of matching a rule statement. Labels remain available on the request for the duration of it being evaluated against the web ACL. Labels also emit CloudWatch metrics and will show up in WAF logs. This example matches a label that was added to the request locally within the same context as the rule:
Rule: {
Name: "match_rule",
Statement: {
LabelMatchStatement: {
Scope: "LABEL",
Key: "header:encoding:utf8"
}
},
RuleLabels: [
...generate_more_labels...
],
Action: { Block: {} }
}
Cloudflare's Super Bot Fight Mode extends the functionality released in 2019 with Bot Fight Mode. The bot solutions are now found in their own hub in the Firewall app under the Bots subtab. Users with a Pro plan now have access to a report that indicates traffic that is likely automated unwanted bots, likely human, and verified bots. This data can also be obtained via the GraphQL API. Bot traffic can be outright blocked or a challenge request can be issued.
Business users have access to Bot Analytics which helps visualize what traffic is being blocked. With this plan, requests will also be analysed by the machine learning engine which helps identify bots that are more sophisticated such as those that rotate their IPs.
Cloudflare's Enterprise Bot Management feature is built directly into their firewall. This allows for restricting bot protection to a particular path, similar to the scope down rules of AWS's WAF service. Cloudflare has also released early access to API Abuse Detection. This feature uses unsupervised learning to map out APIs and identify legitimate traffic to further mitigate traffic from unwanted bots.
AWS WAF Bot Control is available as a paid managed rule within all regions that AWS WAF is in. Cloudflare's bot protection is available to all plans, including the free tier, with differing feature sets by plan.