New Relic announced that their agents, SDKs and integrations will be available under an open source license.
A significant contributor to Cloud Native Computing Foundation's (CNCF) sandbox project OpenTelemetry, New Relic has been in the business of developing multi-language application framework instrumentation since their founding more than a decade ago. They are open sourcing most of their agents, SDKs and integrations with some of the remaining agents to be open sourced in a phased manner starting in September.
InfoQ caught up with New Relic's principal software engineer, Ben Evans, director of software engineering, Sharr Creeden, and group VP of engineering, Greg Unrein regarding the open sourcing.
They talk about how instrumentation of applications is evolving, the company's recent announcement, how it dovetails with OpenTelemetry, and the future of open instrumentation. They also talk in-depth about how they are paying special attention to Diversity, Equity, & Inclusion (DE&I) in working with the community.
InfoQ: Let’s start with OpenTelemetry, which is a sandbox CNCF project. Can you talk about the recent open sourcing, the New Relic agents with that background, and why developers and architects should care about the announcement if they already have OpenTelemetry on their roadmap?
Ben Evans: OpenTelemetry is in the early stages of its life cycle. We anticipate that there will be a variety of reactions from different sectors of the market, depending on where they are with their own observability journey. Not everyone is going to move at the same speed, and that’s OK.
Open-sourcing our previously-proprietary agents is an initial milestone for us - it’s very much the beginning of the journey and not the end. Those teams who are already enthusiastically involved with OpenTelemetry may not be the audience for our open-sourced agents. By putting our agents out there, it helps us by opening up our development processes. It helps the community by providing an example of a mature instrumentation system. In this space there is often a lot of devil-in-the-detail, and we feel that we’ve solved a lot of problems over the years - and the community shouldn’t have to reinvent the wheel and waste resources on issues that we’ve already encountered.
InfoQ: Can you talk about projects (if any) that are being open sourced that might still be relevant as OpenTelemetry starts to gain more ground in the enterprise?
Greg Unrein: We currently have OpenTelemetry exporters for several languages available in open source, with more support in the coming months. Beyond our OpenTelemetry-related projects, there is a range of open source instrumentation projects that we've contributed to or initiated that are likely to remain relevant even as OpenTelemetry gains more ground in enterprises. Examples include the Micrometer registry, Kamon reporter, and Dropwizard reporter.
Since OpenTelemetry is in the early stages of its life cycle, we expect the agent projects we recently open sourced to remain relevant for a long time as well. Over time we expect more enterprises to adopt OpenTelemetry-based solutions. One of our goals is to make that transition from our open source projects to OpenTelemetry as seamless as possible. Some of the code from these New Relic projects could help accelerate OpenTelemetry maturity and adoption.
InfoQ: Can you comment on the current state of Application Performance Management (APM)? You talk about open instrumentation in your blog. What does that mean? Any recent paradigm shifts and any more in the horizon vis-a-vis APM?
Unrein: Open instrumentation is the idea that observability in general, and APM more specifically, work best when there is a thriving ecosystem of engineers and vendors sharing expertise by contributing improvements to methods and tools together. It means the engineers using open instrumentation have transferable knowledge and tools, can contribute improvements to specific tools through that expertise, and then share it with a wide range of other engineers and companies. That is one of the reasons we're so enthusiastic about OpenTelemetry.
This shift to open instrumentation is one of a few that have emerged from the adoption of DevOps and observability as practices that are critical to effective software development and operations. Another shift is the recognition by many engineers and technology leaders that it is important to synthesize telemetry from a variety of sources into a coherent picture contextualized by the business and user experiences that companies are delivering. This is true in most environments, and is particularly acute in many enterprises where complexity is often higher. Open instrumentation promises to simplify this in the long term, but the need exists today.
InfoQ: Containers and service mesh tend to incorporate the sidecar pattern. Is this pattern(s) complementary or orthogonal to the agents in general and those that are being open sourced in particular?
Unrein: It depends on the agent or instrumentation, but in most cases it is complementary to the sidecar pattern. The code-level agents provide deep APM-style visibility into a particular service (often a customer's bespoke service), while the instrumentation available from sidecars provides context and further visibility from outside the service's processes and containers. Agents can also automatically propagate distributed traces, which is important for end-to-end visibility. The agents and instrumentation that focus more on integrations or infrastructure can often be deployed in containers or as sidecars which similarly complement the APM-style visibility.
InfoQ: Diversity and Inclusion have become hot button issues. Can you comment on the current state of diversity amongst the contributors of the recently announced open source projects and how you plan on increasing it? Anything else you want to add?
Sharr Creeden: Diversity, Equity, & Inclusion (DE&I) have been central concerns for New Relic since the beginning of our company. Our foray into Open Source does not alleviate those concerns, in fact it amplifies them, because surveys show that open source contributors and developers often come from even less diverse backgrounds compared to the industry as a whole. We recognize that we’re responsible for being a part of the solution to improve these statistics and are investing in practices and behaviors that further our belief that Open Source should be a place where everyone can feel comfortable contributing.
Within our recently open sourced projects, we’ve thoughtfully designed some aspects to better appeal to a wider variety of contributors. We have adopted a principles-based code of conduct to describe the open culture we aspire to cultivate and we provide transparent project governance mechanisms to ensure expectations are understood upfront. We’ve designed our documentation to be accessible and adopted processes such as using labels to highlight where contributions are most needed or good first issues for a new contributor to take on. Ultimately our goal of these tactics is to lower the barrier to participation within our communities.Internally we’re focusing on training our open source developers on best practices to foster inclusion, for example on how to be welcoming and responsive with their interactions. We’re also encouraging our engineers to adopt a mentorship mindset when it comes to Open Source collaboration. New Relic has so much expertise we can share to elevate each other and the industry, and we recognize that providing intentional mentorship is one of the ways we can increase DE&I within the Open Source community.
New Relic has a dedicated DE&I team whose mission is “... to create and sustain an environment that is intentionally diverse, equitable, and inclusive so that we can help our customers create more perfect software, experiences and businesses.” In collaboration with our senior leadership and peer-company partners, this team has developed a strategy to further our internal DE&I efforts including evolving our sourcing and hiring practices and implementing technology to mitigate unconscious bias at each stage of the hiring process.
One result of our DE&I team’s work is that the talent acquisition processes at New Relic have undergone several changes designed to make our company, including our engineering teams, more diverse, equitable, and inclusive, with specific process changes designed to:
- Create inclusive job postings
- Diversify recruiting channels
- Deploy balanced interview teams
- Standardize our interview process
In summary, they talk about how the future is evolving into open instrumentation and OpenTelemetry. They go into lengths about how they are working with the community and paying special attention to Diversity, Equity, & Inclusion (DE&I) which is particularly lopsided in many open source projects.