Many internet users experienced major website outages on July 2nd due to a misconfigured regular expression in CloudFlare’s Web Application Firewall. The misconfiguration lasted for approximately 30 minutes, in which downstream web sites and applications simply received 502 Bad Gateway messages.
Web Application Firewalls (WAFs) operate as a layer 7 network defense to inspect HTTP traffic as it passes through the network, but they cannot see if or how the application uses the data. As a result, WAFs run many pattern-matching techniques to detect possible threats within the input that is visible from the network.
In a technical follow-up, CloudFlare provided root cause details that the issue arose from a regular expression that consumed 100% CPU while looking for a security vulnerability in Sharepoint, CVE-2019-0604. The vulnerability manifested itself in the way that SharePoint provided data to an XMLSerializer. The root cause was attacker control of the XML base type, as described by security researcher Alvaro Munoz in a whitepaper, "Friday the 13th JSON Attacks." Due to the network locality of WAFs, many affected websites were not actually using Sharepoint.
The CloudFlare blog provides a working exploit to demonstrate the security issue. In this exploit, the attacker controls the deserialized type of ResourceDictionary, instructing it to fill itself with an object that controls the command line of a launched process:
<ResourceDictionary
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:System="clr-namespace:System;assembly=mscorlib"
xmlns:Diag="clr-namespace:System.Diagnostics;assembly=system">
<ObjectDataProvider x:Key="LaunchCalch" ObjectType="{x:Type Diag:Process}" MethodName="Start">
<ObjectDataProvider.MethodParameters>
<System:String>cmd.exe</System:String>
<System:String>/c calc.exe</System:String>
</ObjectDataProvider.MethodParameters>
</ObjectDataProvider>
</ResourceDictionary>
By understanding working attacks, defenders can see what they are actually defending, and software developers can establish the criticality of secure coding techniques. "Developers need to understand why a particular code snippet is insecure, and working attacks help demostrate the impact. Once developers know why, they are more receptive to language and framework specific coding guidelines on how to write secure code from the start," explains Dr. Matias Madou, co-founder of Secure Code Warrior who teaches developers secure coding through gamification.
The regular expression used by CloudFlare attempts to parse XML through regular expression backtracking:
(?:(?:\"|'|\]|\}|\\|\d|(?:nan|infinity|true|false|null|undefined|symbol|math)|\`|\-|\+)+[)]*;?((?:\s|-|~|!|{}|\|\||\+)*.*(?:.*=.*)))
RegEx101 features an online regular expression tester that demonstrates and visualizes the way in which the backtracking expression applies to the XML exploit payload.
The result of regular expression backtracking can be excessive CPU and memory consumption. The Open Web Application Security Product (OWASP) features a vulnerability category called Regular Expression Denial Of Service, which commonly occurs when users control part of a regular expression.
A discussion between Charity Majors (CTO of HoneyComb), Matthew Prince (founder & CEO of CloudFlare), and John Graham-Cumming (CTO of CloudFlare) provides additional details beyond the blog post. Majors compliments the work and practices of CloudFlare, "This [work] is hard tho. Kudos to you for the transparency."
At approximately 4,500 words, the blog post by John Graham-Cumming provides a rich, transparent description of the root cause, along with flowcharts of CloudFlare policies and technical performance analysis of regular expression backtracking.