When DevOps Meets Security to Protect Software

I have now been working in cybersecurity for about two years and the lessons have been immense. During this time, my morning routine has largely remained unchanged; wake up, get some coffee, open my Spotify app, and get my daily dose of CyberWire Daily news. Credit where due, Dave Bittner and his team have done an amazing job with the show. The one thing that has remained constant over the years is the cyber attacks, the data breaches, and the massive thefts and sale of personally identifiable information of many people on the dark web.

We continue to hear the outcry from the cybersecurity world about the acute shortage of cyber talent. Many organizations are putting in place initiatives to try and fill this gap and even train and retain cybersecurity talent. The truth though is that the developers outnumber security professionals tremendously. So how did we get here? And more importantly, what is the industry doing to address these cybersecurity concerns of their platforms and products?

Historically in many organizations security has been treated as an afterthought. It has always been one of those checklist items at the end of the development and given the least priority and effort. The development team works on their software together with the operations team for deployment and maintenance. This was mostly successful, and became widely known as the DevOps process; a collaboration between the developers and the operations teams, working side by side to build, ship, and maintain software. However, security was not often a priority with DevOps. In addition, software development, deployment, and maintenance have continually increased in complexity at scale.

Nowadays, security can no longer be an afterthought, and this has become the general consensus among most professionals in the technology space. In fact, Hackerone has noted that fixing security defects in production is much more expensive than in development. It is becoming a standard practice in the SDLC to ensure security is a consideration during development.

Security and privacy are, more than ever, necessary components of all software. This is a challenge as it is not easy to change age-old processes overnight. But the change is needed as security incidents grow in volume and data breaches continue to be extremely expensive for organizations. A lot of work is being put in place to ensure security is ingrained in all products right from ideation. This is commonly referred to as shifting security left or DevSecOps.
Most companies are committed to churning out secure software to their customers. Currently, the speed at which new software is produced is lightning-fast. In many settings, developers are pushing new updates to products at hourly intervals. There needs to be an intricate balance between ensuring that the speed of innovation is not slowed down and that the security of products is not compromised. In recent years, there has been a huge push, even by governments, for better security in the SDLC processes. This has given rise to new practices in application and software supply chain security.

The Challenges

What are some of the challenges currently with how everything is set up? If organizations are committed to security and privacy, why are they still developing vulnerable software? This is not a question with a single answer. There are a lot of possibilities, but for the sake of this article, I will focus on a few points that most people tend to agree with.

The "Cybersecurity Tech Talent Crisis"

There have been reports and never-ending discussions about the shortage of experienced cybersecurity professionals. This is a contributing factor to the security challenges experienced by most organizations.

Advanced Persistent Threats

Nation-state-sponsored threat actors have become a normal thing in the recent past. These sophisticated actors continue to wreak havoc stealthily, breaching defenses and causing chaos along the way.

Attribution largely remains a complex topic, but most security researchers have pointed fingers at certain Asian states and Eastern European state actors. Andy Greenberg’s book Sandworm provides great details and insights into this with first-hand accounts from Robert M. Lee, a renowned Industrial control systems security engineer.

Ransomware Attacks

In the past couple of years, there has been an increase in ransomware attacks on various institutions and infrastructure. The crooks are into breaking into networks, encrypting data, and demanding colossal amounts of money in order to provide the decryption keys. Conti ransomware gang is perhaps the most prolific actor out there, affiliated with some high-profile and particularly damaging attacks.

Cloud Security Risks

Most of us probably still remember the Capital One hack. Cloud misconfigurations continue to present organizations with a huge security challenge. The cloud has been described by certain people as the wild west if proper guardrails are not properly put in place.

Software Supply Chain Attacks

This topic has attracted a lot of attention lately. Some of the common ways employed by bad actors to carry out these attacks include; compromising software building tools and/or infrastructure used for updates, as in the case of SolarWinds which is widely believed to have been conducted via a compromised FTP server, compromised code being baked into some hardware, or firmware, and stolen code signing certificates which are then used to sneak malicious applications into the download stores and package repositories.

DevSecOps and Software Supply Chain Security to the Rescue?

Is DevSecOps the answer to all the security challenges? Sadly, the answer is no, there is no single magic bullet that addresses all the security challenges faced by different organizations. However, it is agreeable that DevSecOps practices and securing the software supply chain play a pivotal role in greatly reducing software and application security risks.

DevSecOps advocates for knitting security within the software development process, from inception all the way to release. Security is considered at each stage of the development cycle. There are a lot of initiatives that have been put in place, to help drive this adoption. My two favorites, which are also big within the organization that I work for are:

• Automation of Security Testing: Building and automating security testing within CI/CD pipelines greatly reduces the human effort needed to review each code change. DAST (dynamic application security testing), SAST (static application security testing), IAST (interactive application security testing), and even SCA (software composition analysis) scans are pivotal in modern-day security scanning within pipelines. In most cases, if any security defect is detected, that particular build fails, and the deployment of potentially vulnerable software is subsequently stopped.


• Security Champions: This is by far my favorite practice. Perhaps it has different names in different organizations. However, training developers to have a security mindset has turned out to be highly beneficial, especially in helping to address the cybersecurity talent shortage problem.

In general, to implement DevSecOps, organizations often adopt a set of best practices that promote security throughout the software development lifecycle. Some key practices can be summarized as below:

Security as Code

Security should be integrated into the codebase and treated as code, with security policies and controls written as code and stored in version control. Code-driven configuration management tools like Puppet, Chef, and Ansible make it easy to set up standardized configurations across hundreds of servers. This is made possible by using common templates, minimizing the risk that bad actors can exploit one unpatched server. Further, this minimizes any differences between production, test, and development environments.

All of the configuration information for the managed environments is visible in a central repository and under version control. This means that when a vulnerability is reported in a software component, it is easy to identify which systems need to be patched. And it is easy to push patches out, too.

I was once part of a team that was tasked with ensuring cloud operational excellence. At the time, there was a problem with how virtual machines were spun up in the cloud environment. We needed a way to control which type and version of AMIs could be launched in the environment. Our solution was to create golden images according to the CIS Benchmark standards. We then implemented policies to restrict the creation of virtual machines to only our specific, golden Amazon Machine Images.

In the end, we had EC2 instances that adhered to best practices. To further lock things down, we had created Amazon CloudFormation templates, complete with which policies can be attached to the VMs. To achieve our goal, we leveraged these CloudFormation templates along with Service Control Policies (SCPs) to implement a standard and secure way of creating VMs. The point that I am trying to drive with this story is that code-driven configuration can be used to ensure automated, standard, and secure infrastructure deployment within environments.

Automated Security Testing

Automated security testing should be performed as part of the continuous integration and deployment (CI/CD) pipeline, with tests for vulnerabilities, code quality, and compliance. Using automated tools, it is often possible within an environment to set up and run automated pentests against a system as part of the automated test cycle.

During my stint as a cybersecurity engineer, I have been involved in setting up these tools within a pipeline whose sole purpose was to conduct automated fuzzing and scans on applications to search for any known OWAST Top 10 vulnerabilities. A popular open-source tool that can be used to achieve similar results is OWASP Zap.

At the time, this had two immediate results: The developers became more aware of the OWASP Top 10 vulnerabilities and actively tried to address them during development. On top of that, some of the routine and mundane tasks were taken off from the already stretched application security team. This is a simple example of just how automated testing can go a long way in addressing the software security challenges we currently face.

Continuous Monitoring

Continuous monitoring of applications and infrastructure is essential to detect and respond to security threats in real-time. A popular design pattern is the implementation of centralized logging within an environment. Logs from networks, infrastructure, and applications are collected into a single location for storage and analysis. This can provide teams with a consolidated view of all activity across the network, making it easier to detect, identify and respond to events proactively.

This article goes into great detail on some of the freely available solutions that are often used to implement centralized logging. Logging forms the foundation upon which metrics for monitoring can be set within an environment. The Target Breach from 2013 has often been used as a case study as to why proper investment in logging and monitoring is crucial.

Collaboration

Collaboration between development, operations, and security teams is critical to ensure security is considered throughout the development process. A lot of companies have embraced the agile way of work in a bid to give teams flexibility during software development, this further fosters collaboration within the teams.

DevSecOps is primarily intended to avoid slowing down the delivery pipeline. The DevSecOps methodology, being an evolution of DevOps, advocates for application developers, software release, and security teams to work together more efficiently, with more cooperation, in order to deliver applications with higher velocity while adhering to the security requirements. The end goal is to achieve delivery at scale with speed while ensuring collaboration and the free flow of ideas among the different involved teams.

Training and Awareness

Security training and awareness programs should be provided to all team members to ensure everyone understands their role in ensuring security. In my first job in cybersecurity, one of my key tasks was to implement a culture of secure coding within the organization. As a result, we decided to embark on massive training and awareness of the developers within the company. Our reasoning behind this decision was that if the simulated phishing tests carried out within most organizations usually work, then the same concept could be applied to the developers; train their eyes to spot the common problems.

This was quite the task, but in the end, we had two avenues to implement this. The first was to use commercial software that integrates with IDEs and scans code as the developers write, providing them with suggestions to address any security defects in the code. This was a huge success.

The second thing we did was implement regular training for the developers. Every fortnight, I would choose a topic, prepare some slides and perform a demo on how to leverage different security misconfigurations to compromise infrastructure. The two tools that were vital in achieving this were Portswigger WebSecurity Academy and Kontra security which provided a lot of practice on API and Web security misconfigurations.

At the time of my departure from the organization, this was a routine event and developers were more aware of certain common security misconfigurations. We also leveraged Capture the Flag events and provided some incentives to the winning teams to keep everyone motivated. This was one of the most successful initiatives I have undertaken in my career, and it was a win for both parties: the developers gained crucial knowledge, and the security team had some work taken off their plates.

Preventing Software Supply Chain Attacks

Any time I come across these words, my mind automatically goes back to the 2020 SolarWinds attack and the 2021 Log4j Vulnerability. Most of the people in the security world are quite familiar with these two, not only because of the damages they caused but also due to the fact that they struck right around the Christmas holidays! The conversations around software secure supply chain gained a lot of traction with these two incidents.

There was already much talk about the software supply chain, but there was little traction in terms of actually putting in some measures to address this problem. The whole frenzy that was caused by the Log4j vulnerability seems to have been the push that was needed for organizations to act. The US National Security Agency has been giving continuous advisories to developers and organizations on how to better address this problem. The problem with software supply chain compromise is not one that will be addressed overnight. To date, we still see reports of malicious Python or JavaScript packages, and even malicious applications finding their way to the Google Play Store.

I came across one of the simplest analogies describing software supply chains as I was reading blogs and attempting to gain more understanding about this topic, back in 2020. The author of the blog compared software supply chain attacks to the ancient kingdoms, where the enemy soldiers would poison the common water well, rendering every person in that village basically weak and unable to fight.

This is quite accurate with respect to supply chain attacks. Instead of an attacker compromising many different targets, they only have to taint the one common thing that many unknowing victims use. This makes an attacker’s work quite easy thereafter, and this is exactly what happened with SolarWinds, and later with the Log4j vulnerability. This is obviously extremely dangerous, in line with how software is currently made; there is a huge dependence on open-source libraries and packages.

This post by John P. Mello Jr. provides ten high-profile software supply chain attacks we can learn from, including that one staged on Okta in 2022. From the blog post, the compromise on both npm and Python Package Index (PyPI) alone affected over 700,000 customers. In this case, the attacker did not have to compromise each of the individual seven hundred thousand victims, they simply found a way to tamper with third-party software building components and enjoyed their loot. Like in the poisoned well analogy, the consequences are felt downstream by anyone who uses the compromised packages. Third-party risk assessment is now a whole thing in most organizations, however, this is still no defense for the poisoned well.

The big question is, how secure are these third-party libraries and packages? This is a topic that deserves its own article, there is simply so much to cover. With the problem highlighted, what are governments and private organizations doing to prevent these kinds of attacks from happening?

• Zero Trust Security Architectures (ZTA): This concept encourages us to always assume a breach is present and act as if the attackers are already in our environment. Cisco Systems and Microsoft have some amazing products for implementing ZTA through passwordless authentication and continuous monitoring of authenticated users.


• More organizations are opting to conduct regular third-party risk assessments. There have been cases in which certain organizations have been compromised by first compromising a less secure vendor or contractor, as was in the case of the famous Target breach.


• Enhanced security in build and update infrastructure should definitely be on top of the list. If properly implemented, attackers would technically not be able to tamper with and deliver vulnerable software or software updates to the downstream clients.


• Proper asset inventory, together with a comprehensive list of software bill of materials (SBOM). When Log4j hit back in 2021, many of us were in much bigger trouble because we had no idea where to look in our environments, as we didn't have an accurate SBOM for the various applications we were running. It is crucial to maintain this inventory because it’s impossible to defend and investigate that which you do not know.

DevSecOps plays a huge role in ensuring secure software development, this is becoming more apparent every day. Ensuring that software build materials remain safe is also key to preventing most software supply chain attacks. It remains to be seen how the threat landscape evolves, for now, however, we must focus on getting the basics right.

As cyber threats continue to evolve, it is essential to integrate security into the software development process. DevSecOps is a culture shift that promotes collaboration, shared responsibility, and continuous improvement, with security integrated into every stage of the development process. By adopting DevSecOps best practices, organizations can build more secure software faster and reduce the risk of security breaches, while also improving collaboration and reducing costs.