The Cloud Native Computing Foundation (CNCF) has announced the addition of four new hosted projects over the past month: Docker's Notary project is designed to provide trust over digital content like container images using strong cryptographic signatures; NYU's Tandon School of Engineering The Update Framework (TUF) is an open source trust specification that Notary implements; Lyft's Envoy service mesh is a data plane proxy for microservice communication; and Uber's Jaeger tracing project enables request/data communication flow to be observed across a distributed system, such as a microservices-based application.
The Notary project, initially created by Docker in June 2015, is designed to provide high levels of trust over digital content using strong cryptographic signatures. For example, crytographically signing container images and associated metadata. In addition to ensuring the provenance of the software, it also provides guarantees that the content is not modified without approval of the author anywhere in the supply chain. This then allows higher level systems like Docker Enterprise Edition (EE) with Docker Content Trust (which uses Notary) to establish clear policy on the usage of content.
The Update Framework (TUF) is an open source specification that was written in 2009 by Professor Justin Cappos and developed further by members of the Professor Cappos's Secure Systems Lab at NYU's Tandon School of Engineering. This project was submitted to join the CNCF in partnership with Notary, as Notary is one of the most mature implementations of TUF. Notary/TUF provides both a client, and a pair of server applications to host signed metadata and perform limited online signing functions.
Figure 1. Notary/TUF singing and verification sequence diagram
Current examples of Notary usage includes: Docker uses Notary to implement Docker Content Trust and all of the docker trust subcommands; CoreOS's Quay, a container registry SaaS, is using Notary as a flexible library for trust and verification of container images and metadata; and LinuxKit is using Notary to distribute its kernels and system packages. Notary is already used in production environments beyond container distribution with Cloudflare integrating it into their PAL tool for container identity bootstrapping and Kolide using it to secure their autoupdater for the osquery tool.
Last month the CNCF also announced that Envoy would be the 11th hosted project. Originally built at Lyft to move their architecture away from a monolith, Envoy is a high-performance open source edge and service proxy that makes the network transparent to applications. Software Engineer Matt Klein led his team to design the technology to abstract most networking complexities from the application developer. Written in C++ for performance reasons, the Envoy out of process architecture can be used with any application, in any language or runtime; including HTTP/2 gRPC proxying, MongoDB filtering and rate limiting, and more.
Figure 2. Current Envoy usage at Lyft
Klein explained in a recent blog post, that Lyft's business is almost entirely based on open source technology.
Without [open source], it's unlikely that the ridesharing service we know and love would exist today. Given the large development effort that had gone into Envoy, and understanding that many other organizations face identical challenges when moving from a monolithic to microservice architecture, we wanted to give back to the larger community that had nurtured our own company growth. Therefore, we decided to proceed with open sourcing Envoy and working to build a community around it.
Currently Envoy has 78 contributors from at least 10 different organizations with primary maintainers working at Lyft and Google. Klein believes that "as a technology, Envoy has the opportunity to become a primary building block of modern service architectures." This belief is fast becoming a realisation, as organisations like Verizon are using leveraging Envoy within the Nelson automated container deployment platform, the Istio service mesh control plane project is fast gaining traction within the industry, and startup companies such as Datawire are building open source tooling like Ambassador on top of Envoy. Envoy complements the existing CNCF service mesh project, Linkerd, created by Buoyant.
Rounding up the recent project hosting announcements, the Jaeger distributed tracing project, initially created by Uber, will be the 12th hosted project in the CNCF. Jaeger uses an OpenTracing compatible data model and provides instrumentation libraries in Go, Java, Node and Python. OpenTracing is an existing CNCF project, and defines a vendor-neutral open standard for distributed tracing.
Figure 3. Jaeger architecture and usage at Uber
Uber began deploying Jaeger internally in 2015. It is now integrated into thousands of microservices and recording thousands of traces every second. The tracing system is also used by companies like Base CRM, Stagemonitor, and Symantec. Additionally, companies like Red Hat are active contributors to the project. Bryan Cantrill, CNCF Technical Oversight Committee representative and project sponsor, stated in a recent blog post that distributed tracing is core to providing observability within microservice-based systems:
One of the criticisms of microservice-based architectures is that they can become distributed monoliths: complicated, interdependent systems that tend to fail (or perform poorly) at once due to unforeseen interactions. In order to attack this problem, we must have the ability to follow code flow across services.
More details on Jaeger can be found in the Uber blog post "Evolving Distributed Tracing" written by Yuri Shkuro, which explains the history and reasons for the architectural choices made in Jaeger.
For additional information on the CNCF, the project website contains details of the charter, current membership, and hosted projects. The announcement section of the CNCF website also contains further details on the hosted projects mentioned within this news item.