Log4shell was one of the most critical vulnerabilities of the current history, affecting the vast majority of the Java applications deployed on the Web; the initial CVE being marked as Critical, the highest possible score. While everybody was probably thinking of a holiday gift, the past holiday season started with a lump of coal for many developers. Even in the aftermath of the event, some technology leaders considered the work of fixing this vulnerability to have "been some of the most intense days in my career" as mentioned by Martijn Verburg while reflecting on the events that took place after the discovery of the vulnerability.
Due to the critical nature of the current systems, classical approaches like Web Application Firewall (WAF) were not effective, so alternative approaches were required. The big players like AWS and Microsoft are using Java agents to hot fix these issues. A similar approach was taken by Contrast Security as well. In order to better understand why this approach would succeed where others failed, InfoQ reached out to Arshan Dabirsiaghi, co-founder and chief scientist of Contrast Security.
InfoQ: Thank you for taking the time to answer questions for our readers. What is Contrast Security’s mission and what are your day-to-day responsibilities?
Arshan Dabirsiaghi: Contrast’s mission is to empower developers to self-serve on application security. All the tools out there today require experts to run and experts to triage – mostly because they are full of noise. If security wants to keep pace with development, they need tools that are hyper-accurate, and that’s what drives us.
I wrote most of the original product, but today I serve as chief scientist. I guide all the functions in our business towards solving the problem of application security in a way that will allow developers to own security. To get there, we focus on easy-to-generate, hyper-accurate results, with clear storytelling that resonates.
InfoQ: Why was the scope of Log4Shell so much larger than other vulnerabilities?
Dabirsiaghi: Firstly is the ubiquity of the vulnerable library in the Java applications: our data suggests that it is referenced in 64% of the applications, and close to 40% actively using it. This is caused mostly by the limited options when it comes to Java logging libraries.
Secondly, the vulnerability is trivial to exploit: the vulnerable application just needs to log some data that end users provide, which is the main responsibility of a logging library, unfortunately. It was rare to see an application in our exploration that didn’t satisfy this condition. Attackers could just blindly throw a payload at a company’s perimeter applications, and through the spaghetti of modern distributed architectures, it would probably encounter a Java application running a vulnerable version of log4j2.
InfoQ: When things were urgent on Day 0, both AWS and Microsoft used a Java Agent to defend their systems? What are agents and why were they one of the first choices?
Dabirsiaghi: It’s worth noting that web application firewalls also quickly made rules, but they were quickly and thoroughly eviscerated (bypassed) by the community. The variable interpolation available within log4j made an already-difficult job of signaturing attacks impossible for perimeter controls like Web Application Firewalls. WAFs are also only good for HTTP – and modern apps don’t just have a classic three-tier architecture, anymore. HTTP is just one of many ways how apps are communicating.
Agents were the first choice for AWS and Microsoft because protecting against this attack with an agent is an extremely robust control that presents zero tradeoffs in terms of performance or stability. Simply preventing the dangerous Java Naming and Directory Interface lookup method from executing is enough to stop the attack.
Our own agent, Contrast Protect, has defended against this attack since 2018 through multiple layers of defense. We are just starting to scratch the surface of how instrumentation can prevent the exploitation of entire bug classes.
InfoQ: What are your takeaways after the Log4Shell madness?
Dabirsiaghi: As the guy who has been holding a "The End Is Nigh" sign, I'm excited that many people realize that application security is maybe the most important security, because it’s the one thing for which you intentionally program your firewalls to expose to the world. I’m more bullish than ever on an agent-centered approach to monitoring the security of applications (IAST and RASP). It’s only with agents that we can effectively and safely protect against many bug classes, and create a live inventory view of applications and their libraries in use.
InfoQ: How would you recommend improving the software development life cycle for a more robust ecosystem?
Dabirsiaghi: As a community, I’d venture to say that we generally do know how to create secure code, but it’s expensive and slow, and the people needed to do it are hard to find. It’s not a slight against the developers of this free, awesome utility to admit there’s not much security involved in the design of the major components of log4j2. That team very likely isn’t skilled sufficiently in that area, and that is a problem for all of us.
For example, security people understand the principle of least privilege. If that thinking was applied to log4j2, it’s likely that the JNDI features would have been opt-in, or packaged in a separate module entirely, and this vulnerability would have affected a vanishingly small percentage of applications instead of practically all of them.
Another problem nobody really talks about is that most of the tooling we use to test for security issues is built for scanning a whole application, not really a single component. I think we need new tools in this area, but who is going to buy licenses for them?
InfoQ: How do you think the OSS ecosystem will be impacted by this vulnerability? How about the corporate one?
Dabirsiaghi: I’m excited to see, because I’m an optimist. This vulnerability got attention like I haven’t seen before in appsec, and deservedly so, so there is good reason to hope.
The OSS ecosystem is mostly unpaid laborers doing passion projects. Given that, I don’t see any long-term changes to their behavior alone. However, all the revenue-producing consumers of their work could come to the table to enable that ecosystem to produce more secure code. They could contribute money, expertise, tooling, and standards in order to make OSS development look more like the secure development lifecycle you’d expect from a high functioning team inside one of the FAANG organizations working on a mission critical, highly visible project.
If this sounds unrealistic to you, well then, it sounds like you think it’s unrealistic to have secure open source components, and time will tell who is right – I just hope we don’t have too many Log4Shell incidents along the path in figuring it out.
I’d love to see some fundamental change because it is inevitable that such a vulnerability will happen again. This is why protection is crucial. And not just against 0-days. In fact, practitioners will privately admit that there is no way they will ever eradicate Log4Shell from within their borders. There’s just too much shadow IT, not enough inventorying capability, too many unmaintained apps, etc. We’re in this together with the OSS ecosystem and the damages are long-lasting.
In the aftermath of the Log4Shell winter storm, we better understand that traditional defense mechanisms, such as WAFs, deemed themselves useless against the highly polymorphic payloads an attacker could. So, rather than hoping the system is protected from outside by using the firewall as a moat, it was needed to actively patch the system from within. Java Agents proved to be a very efficient mechanism to accomplish this with similar approaches being followed at both Microsoft and AWS.