Using Cloud Native Buildpacks to Address Security Requirements for the Software Supply Chain

In recent weeks and months, a lot has been said about software supply chain security. Starting with a recent Presidential Executive order, the world’s favorite developer discussion board, OSS panel discussions, and many other places, software supply chain security — and the software Bill Of Materials (SBOM) are finding mainstream mention. 

A holistic overview of all the components that come together to deliver a product is defined as a supply chain. Unlike physical products, software development has a very loosely defined notion of supply chains. Several examples are available to help us analyze the parallels and understand how they're the same and how they're different. Take automobiles as an example – those assembling each car are only allowed to make use of carefully designed and selected components. This allows car manufacturers to control the quality of the vehicle produced. It also provides much more governance over the outcomes of the whole manufacturing process.

Examining the software supply chain (and attacks)

Now, take a typical software development process. The components that flow inward are never controlled or regulated strictly. Programmers are allowed to introduce non-standard libraries depending upon the current need of the project. There is no notion of a careful selection or security. Instead, the focus during software development is always on solving a technical challenge. This rather loose definition of the input components is the result of a rather loosely defined supply chain when it comes to software development. In addition to the actual development process itself, various other workflows such as staging, testing, and deployment tend to introduce newer (and more potentially untested) components into the lifecycle of a product. Software supply chain is a speed bump that comes along the DevOps industry’s superhighway – an industry that is disproportionately obsessed with velocity and speed of deployment. 

There are numerous ways to cause a software supply chain attack. Generally speaking, as software components move up the deployment pipeline, they are authenticated in order to work. These components should be able to access various internal pieces that comprise the host infrastructure. As an example, libraries that are used for testing and asserting various outcomes are given full access to the internet and a web client locally (typically in the form of a headless browser). As the test vector increases in complexity, more access to local systems such as the file system can be allowed. This results in an attack surface that is quite large. Now, imagine some malicious code being inserted into these libraries. When running in production these bad programs are over-authenticated and can then gain access to all the resources present, ultimately enabling them to execute an attack from within.

Move fast but don’t break things

Numerous DevOps principles position it squarely against the notions of common security concerns. Specifically, they are at odds with sensible defaults for supply chain security in software development lifecycles. Let's examine a few principles.

#1 - Fast and Frequent Releases

DevOps principles often advertise the ability to release M times a day and N times faster. This results in a much smaller fraction of time left over for auditing the components that go into building a new software application. The focus of software development teams always remains on lowering the rate of change failure. None of these metrics aim to directly verify the source or security of the software components that actually go into each release. Considering the bandwidth limitations that teams have, especially when releasing at a high frequency, it becomes impossible for software development teams to ensure that the use of new components are duly recorded and accounted for.

#2 - Transparent Build processes

Teams are constantly evolving build processes to be faster and more consistent. This is done by altering the build process in all stages such as development, testing, staging, and supporting releases in production. Many build tools make use of arbitrary install commands, perform dependency injections, and are directly responsible for provisioning the internal components within a piece of functional software. Adding to the complexity of a build process are notions of microservices and polyglot applications. Each service eventually spawns its own niche build and migration processes, which is all antithetical to a transparent and auditable build process. Eventually, build processes transform into little black boxes, causing sprawl and giving rise to a potentially larger attack surface.

#3 - Agile Adoption

Implementing DevOps at the workplace goes hand-in-hand with agile methodologies for software development. Agile practitioners always find it hard to allot latitude for non-development work. In several years of observing agile methodology in action,I  have never come across instances where the assignment of story points to discover and verify the source of software components have happened. Every bit of the agile process, namely, specification, estimation, progress, and tracking are very difficult to accomplish with fluid items such as verification of components and auditory regulations.

Looking for the weakest link: External Dependencies

In most organizations, deployment is akin to several dominoes, lined up in some benign pattern, and presents a path of least resistance from a developer workstation to production servers. All along this path, these are several distinct points at which supply chain attacks can originate. 

In particular, two areas are most susceptible to supply chain attacks. First, when developers decide to integrate components into the applications they are developing. This is comparable to an analogy where the magnitude of car safety is directly proportional to the use of certified parts in assembling it. If mechanics were allowed to use openly available parts, there can be no guarantee of standards or safety. Second, the build pipeline itself. There have been numerous cases where malicious artifacts have been inserted along the delivery pipeline and attackers have asserted themselves once they had access to over-provisioned environments. 

Developers – in particular open source contributors – are strongly opinionated. Solving technical problems is their primary purpose and they have limited patience and bandwidth for reading and understanding compliance checklists. They desire unbridled innovation and not layers upon layers of regulation. Even the slightest hint of finger-pointing or blame will cause their interest to wane, which is certain to lead to a net negative outcome for software projects. Is there a resolution for these opposing forces of code vs. code?

In its entirety, the solution for mitigating software supply chain attacks comes in many parts. And, we’re not talking Dilbert-esque AI-blockchain-hodge-podge-ML solutions either. As with all secure systems, it begins with educating developers about their responsibilities, and then finding ways to incorporate security best practices without disturbing their workflows. 

Introducing a Software Bill of Materials with Buildpacks

One piece of the solution is a Software Bill of Materials (SBOMs). Simply put, SBOMs are a comprehensive list of all components and dependencies that are a part of the application that is being run. Several software components come into contact with an application at various stages, and an SBOM can act as a storehouse for all this information. They help make a developer's choice of dependencies transparent, making it easier to audit and comply with any security regulations in the workplace. One of the earliest evidences of SBOMs in the wild are in the Java ecosystem, where Maven - a Java build tool - generated an optional Bill of Materials. Today’s most common build practices, which employ the use of Docker images as immutable artifacts, generally have a high opacity and don’t often allow developers to know the composition of an image.

A more modern way to generate container images is to make use of Cloud Native Buildpacks and the pack cli. Cloud native buildpacks are used to export OCI-compatible container images from application source code. The CNB is currently in the form of a detailed set of specifications that are in turn used to create buildpacks. These buildpacks begin with examining the source code, initiating a build process, and finally exporting a container image. The CNB specification adds value in two portions of a container lifecycle - creation and maintenance. Across the whole stack that comprises the base image, the dependencies, and the application runtime - a lot of routine security and updates are required. By introducing distinct layers in the architecture of the final image and employing the use of a container registry, cloud native buildpacks make maintenance hassle-free. At scale, thousands of images can be updated in a matter of seconds when built using the CNB spec, whereas it would take a long period of time to update traditionally-built container images.

In addition to these advantages, the Cloud Native Buildpacks specification requires the generation and storage of metadata pertaining to the internal build and export process. This metadata helps determine what each layer within the container image is composed of and the components and dependencies of the application. 

In particular, Paketo, which is an open source implementation of the Cloud Native Buildpack specification, provides a clean way to solve the problems commonly associated with the lack of transparency. Paketo Buildpacks also provide the foundation for a repeatable and secure build process. Combined, they help address the most critical problems associated with software supply chain security that we identified before.

$ pack inspect-image test-xchange
Inspecting image: test-xchange

REMOTE:
(not present)

LOCAL:

Stack: io.buildpacks.stacks.bionic

Base Image:
  Reference: bace2b890e3a242b9ad9c6e4e882e1dc471ee27a7db59ec19745efc7345e2a51
  Top Layer: sha256:863853c5b183a0b88e5a8cd6f1b37ca5d3b4c6bf10a4338ff45e01befdf737d6

Run Images:
  index.docker.io/paketobuildpacks/run:base-cnb
  gcr.io/paketo-buildpacks/run:base-cnb

Buildpacks:
  ID                               	VERSION
  paketo-buildpacks/node-engine    	0.1.5
  paketo-buildpacks/npm-install    	0.2.4
  paketo-buildpacks/npm-start      	0.0.3

Processes:
  TYPE             	SHELL    	COMMAND           	ARGS
  web (default)    	bash     	node server.js   	 

Automation is baked into the pack command. It requires no additional effort from developers. Internally, pack makes use of Paketo Buildpacks to export the container image. Therefore, Paketo takes a batteries-included approach towards software supply chain security. By making use of a structured format for metadata that conveys the precise components of an image, Paketo is a great fit for organizations that look to implement compliance and security policy. 

The Bill Of Materials that is generated will help engineering teams make informed decisions, thereby helping mitigate upstream vulnerabilities that can come from software supply chains. 

We can easily see an example of this by running the pack inspect-image command. 

The above screenshot is an examination of the bill of materials from a container image created for a JavaScript web application using the pack build command. It stores the metadata associated with each of the buildpacks used in creating the container image. Here’s a link to a gist which showcases yet another example bill of materials associated with a container image for a Java application. In addition to the kind of information you see above, the linked example contains information about license metadata, dependencies, and a host of other information that constitute a comprehensive Bill of Materials. 

The benefits of buildpacks: Automated BoMs

Paketo Buildpacks - with their Software Bill Of Materials - offer a starting point for improving the security around software supply chains. Employing Buildpacks in the DevOps workflow for your application development helps diffuse a lot of the tension arising due to security best practices. Added to that, by offering an automated way to generate and include SBOMs, Paketo Buildpacks are great for productivity. 

Software engineering teams, especially small teams and startups will really appreciate the unobtrusive manner in which buildpacks accomplish this, with no additional effort piggybacked on to developer workflows. It will also be of interest to DevOps practitioners to know that the Paketo community is working on standardizing the BoM across all the buildpacks, which is expected to bring enormous value addition for those looking to audit the source of the software components. 

Wrapping up

Just to reiterate, here is a summary of the problems identified in the previous sections and the exact nature of the solutions that Paketo Buildpacks provides.

First, the topic of what modern engineering teams measure. The most frequently employed DevOps metrics are release frequency, change volume, and deployment time. None of these metrics account specifically for creating an audit trail of the components used within these deployments or releases. Automatically generating software bills-of-materials during the build process, as is done with Buildpacks, is a great way of ensuring that releases are tightly coupled with the metadata about the components that are used. With increasing frequency of releases, the SBOMs will get updated that much frequently - thanks to baked in automation.

The SBOMs generated during each deployment are made available for any auditing needs that arise. Typically, these audits are carried out asynchronously or at the time of any security incident. The contrast here is between scenarios that have a bill-of-materials versus those that don't. Depending on the security and compliance requirements in place for various teams, a review of the SBOM will be required. Without a BoM, any audit is a much more difficult and time-consuming task.

Because Paketo Buildpacks automate the generation of software bill-of-materials, it does not affect the process of estimation and tracking that agile teams tend to follow during a sprint. Just as engineering teams will have a software artefact to showcase during (or at the end of) a sprint, and prepare for its release they will also have an SBOM associated with the specific release. The strong coupling between SBOMs and the build process employed by Paketo ensures that this metadata is made available with no drop in velocity.

About the Author

Ram Iyengar, Developer Advocate, Cloud Foundry Foundation, is an engineer by practice and an educator at heart, Ram was (cf) pushed into technology evangelism along his journey as a developer and hasn’t looked back. He enjoys helping engineering teams around the world discover new and creative ways to work.