Released this month by Capital One, version 2.0 of the open source project Hygieia adds support for multiple teams and a corresponding consolidated view into a program-level dashboard, a Jenkins output publisher and a GitHub Webhook consumer. Hygieia integrates widgets from story tracking, repository, build, quality, deploy and monitoring tools on a single-pane-of-glass HTML client that enables an easy access to important DevOps metrics.
The new program-level dashboard helps aggregating multiple or larger teams with possibly different issue and story tracking tools, repositories and build tools. At each pipeline stage important metrics such as product health are shown:
Image source: capitalone.io
In addition, the pipeline speed and number of commits are displayed in the program view:
Hygieia dashboards are customizable: users can select VersionOne or Jira for story tracking, Subversion or GitHub as repositories, Jenkins/Hudson for builds, Selenium and SonarQube for quality, uDeploy and Jenkins for deployment. More plugins are available.
The image below shows an example of a team level dashboard:
Hygieia dashboards assist in achieving process transparency and therefore help establish feedback loops that are the underlying concept of lean and DevOps. They contain interactive elements which enable drill-down and linking to the connected tools.
InfoQ spoke to the Hygieia team at Capital One:
InfoQ: Could you please introduce yourself and describe your involvement in the creation of Hygieia.
My name is Tapabrata Pal (I go by “Topo”), and I am the community manager for Hygieia.
As part of our broader DevOps efforts at Capital One, we were looking at commercial and open source tools for dashboarding. There are some great tools out there – but none of them actually showed us what we needed to see -- the health of the continuous delivery pipeline – from code commit to production deployment with all the necessary information around health and quality of the software.
So my colleague, Amit Mawkin, and I worked on a Proof of Concept during summer of 2013. When the proof of concept was successful, we showed it to our development teams and our leaders and secured funding for the project. Development was done in an “inner source” manner following Open Source development process within Capital One. At Capital One, interest in and appetite for leveraging and contributing to open source has grown exponentially as we’ve been increasing our in-house engineering talent to develop our own software. We believe there are often huge advantages to using open source software over licensed products.
We released the project internally during late 2013, tested it out, refactored and redesigned multiple times. We then open sourced the project (named Hygieia) during the summer of 2015.
Today, I serve as the community manager for Hygieia. The core committers are Amit Mawkin and Jeffery Damick and myself – each of us represents a different line of business within Capital One.
InfoQ: How has Hygieia improved the Continuous Delivery process at Capital One?
Transparency is a key requirement to become successful in Continuous Delivery process in any organization -- transparency around each and every stage of the delivery pipeline. This is even more important when you have a large product that is co-developed by multiple agile teams. Fast feedback is another aspect of the transparency. Everyone involved in the product development should know about the pipeline health – the product owner, the developers, the test engineers, the operations engineers, the information security engineers, the release engineers and the management so and so forth.
With the Hygieia dashboard, we were able to increase transparency in the pipeline. Now, everyone involved can see (a) what stories the teams are working on, how many are in progress, how many are done (b) what codes are being committed, how frequently, and who is committing code (c) how many builds are taking place, are builds failing, what broke the build etc. (d) static code analysis result of the code, security scan results of the code, unit test case results and test coverage, functional test case results etc. (e) various environments and code deployment statuses in those environments. (f) a deployment pipeline showing what version of software is deployed in which environment and what failed.
In addition to this, we recently released a newer version of the product – you can actually see movement of each code commit through the different development stages – code commit, build, deployment to DEV, QA, INT etc. all the way to production. Each of these stages shows the number of commits waiting to go to the next stage – amplifying the wait-times between the stages. This helps the teams discuss how to reduce the wait time – we find that this is the right way to become lean and speed up a continuous delivery pipeline.
InfoQ: What are the main benefits and which roles in the organization benefit most?
There are many tools that build a continuous delivery pipeline. The tools may vary from product to product – the variations are dependent upon many factors including the technology stack of the product, teams’ choices, tools availability etc. All these tools have their own dashboards to show a small part of the pipeline information.
Hygieia shows the view of the whole pipeline – from story/requirement to code deployment in production. It shows the right amount of information (opinionated that is) so that the right people from the product team can take action to fix something that is broken. It shows quality information that can allow the product owner to decide if some quality improvement work needs to be included in the current of upcoming sprint. It also shows information that can help start a discussion around the pipeline design itself – are the test cases running for too long? Is there a process that is slowing down the flow? And I can go on and on…
Some of the things that Hygieia displays enabled us to bring in some positive changes to our delivery practices. For example, earlier during our internal launch of Hygieia, many teams’ dashboard showed poor unit test coverage and open critical code analysis issues. The teams then went ahead and made sure to include some of this improvement work in their regular sprint work and they got good support from the management.
As far as the consumers of the Hygieia, I can say that this dashboard helps everyone involved in software development: the management, product owner, developers, test engineers, application security engineers, release engineers, etc.
InfoQ: The deployment orchestration that is currently supported is uDeploy. Are there plans to provide other connectors?
Yes. With our Jenkins plugin, users who are deploying applications using Jenkins can publish application deployment status to Hygieia. Internally, we are in the process of designing the integration with Chef and Ansible.
Hygieia is tools agnostic – the architecture of the product is a plugin based architecture. We have released plugins for the tools that we use and that are very popular in the industry. If some team is using a different tool, a new plugin can be created very easily. We also provide REST API’s that can be invoked remotely to publish data to Hygieia. We encourage the community to contribute back to Hygieia.
InfoQ: Can you share an outlook into the feature roadmap?
We will continuously enhance the product adding new feature sets that are important to our community. We are working on integration with AppDynamics, Chef, Ansible along with few other testing tools.
The next two Hygieia enhancements will be around
(a) Environment monitoring where we want to see a lot of infrastructure related information on the dashboard and
(b) Showing the flow of features through the pipeline along with the code commit. If you think about the whole premise of Continuous Delivery – it’s about delivering business values faster maintaining good quality. So, the business would want to know what are they getting out of this.
I am a firm believer of dogfooding. Internally, many of our large internal product teams are using it actively. New users of Hygieia will notice it is an active and opinionated community and that is because our internal users give us lots of feedback. We develop and release new features on a regular basis.
InfoQ: Anything else you want to add?
At Capital One, we believe in open source as a way of doing business – and with good reason: it’s a powerful approach that lets people work together to build great things while realizing shared benefits. The banking industry is still ripe for reinvention, which is why Capital One is embracing open source to reimagine products and services and digital experiences to connect people and money in more meaningful ways.
