A lead software engineer at New Relic wrote a summary of how DevOps tools and concepts are used and practised in the New Relic engineering team. It talks about the evolution of the DevOps role, the tools used, including their own product, and the visible benefits of this culture.
Nic Benders, currently the chief architect and VP of Engineering at New Relic, started the first Site Reliability Engineering (SRE) team there in 2012. Rather than being imposed, the engineers themselves opted to adopt the DevOps philosophy once they realized the benefits. In spite of this bottom-up origin, there were still traditional siloes to be broken down between dev and ops teams. Adoption of the right tools and processes was another hurdle that had to be crossed.
The creation of DevOps friendly roles was one of the first steps. In the early stages, the team tried out the concept of a Forward SRE (FSRE), which was a consultant role played by SREs. They were assigned to different product teams, sometimes into more than one simultaneously, to ensure things were running smoothly for that team. This evolved into the Product Reliability Engineer (PRE) role where the PRE was directly part of the team, unlike in the previous role. PREs formed an integral part of the team, with direct communication instead of via ticketing systems. In recent times, the engineering team has been part of a larger initiative called Project Upscale, which lets any engineer choose the team they want to work on.
Two technologies that help New Relic’s DevOps teams do their job well are Cassandra and Docker. Previously, data was stored in MySQL which was managed by a different team. Cassandra, being written in Java, can be explored and debugged in depth by developers too who want to look at the code. It can also be deployed and operated much more easily by the same developers. Docker helps developers to use a uniform command set against different technology stacks, as long as they are all packaged as Dockerfiles. The number of different stacks had started to increase when the main application was being re-architected into a service oriented architecture. Using Docker lowered the barriers between dev and ops since they could use the same command for operating the deployment units.
Two internal tools that helped Dockerization were called Centurion and Shipright. Shipright enables devs unfamiliar with Docker to easily generate Dockerfiles from base images and configuration commands, whereas Centurion makes it easy to deploy Docker containers to fleets of machines with environment specific configuration. The latter tool also ensures that the deployment environment remains the same on dev laptops as well as on the production machines, and is open source.
Some of the New Relic suite of tools are used by the New Relic DevOps teams themselves. These include Insights, a dashboard similar to Hygieia, for a single view of metrics and events from different sources across teams.
The benefits that New Relic got from the adoption of a DevOps approach include the removal of bottlenecks when there are dependencies between teams, visibility in the software delivery process, and a culture of documenting everything like runbooks and incident writeups.
