Lessons Learned about Developer Experience from the Platform Adoption Metrics @ Naranja

Introduction

As software engineers, we have a bias towards laziness that usually results in us being better, more productive, engineers for it keeps us trying to automate things and create abstractions to eliminate duplicate code. But this impulse - which is certainly noble - sometimes takes us very far without really understanding the true economic impact of our initiatives, which eventually is what stakeholders will look at in order to define their viability.

At the Developer Experience team at Naranja, we are devoted to offering solutions to our internal customers, to allow them to focus on business-driven developments, by removing the need to resolve platform issues, which would often need to be done over and over again.

We decided to achieve this goal by applying product management to our internal platforms, and after gathering some experiences, we understood the importance of having metrics that reflect the impact of our work and enable data-driven decisions,  and implemented a solution to gather them and provide visibility throughout.

In this article we will cover the definitions that we arrived at in order to align our organization around these concerns, and the outcomes that resulted from implementing a solution that measures our work based on them and from the fact of having that data available.

Context

Naranja is the main credit card issuer in Argentina, with more than 5 million clients, 9 million credit cards, 200 branches and agreements with more than 260,000 commercial partners. In 2017, it started a digital transformation process, with the mission of becoming the most loved fintech in Argentina. This implied a total renewal of both its existing systems, as well as their culture and development processes.

Tribes, Squads and Projects

Within this context, Naranja shaped it’s organization's with a structure inspired by the Spotify Model, with autonomous squads responsible for product delivery, grouped in tribes. Currently the IT area has more than 300 professionals, organized in 20+ squads, grouped in 5 tribes and two development centers (Buenos Aires and Córdoba). Their scope includes a mobile app, a website that includes self-management tools, a portal for commercial partners, and several other customer-facing applications and the services that power them. Each squad is responsible for the execution of one or more projects, which for the purposes of this initiative are equivalent to repositories in our SCM - GitLab.

Developer Experience

In order to support improving these teams’ efficiency, the Developer Experience area is organized itself in three different practices (Delivery, CloudOps and Development) which are responsible for agreeing with tribe leaders a solution roadmap, based on their prioritized development needs, and then their implementation, facilitating their adoption and providing support.

Products & Assets

For these solutions, the area adopted a product-based approach, treating each solution as a package that our clients can use with as much autonomy as possible, evolving them in response to our clients needs and even making clients part of the ideation process, with constant feedback. Products, conceptually, are a very important part of our metrics. Later on we’ll see how we use this concept as an aggregator of our indicators. Examples of our products are: Wiru (CI/CD solution on top of GitLab), Golden API (node.js REST API reference architecture), Zumo (UI Components).

Furthermore, our products are then composed of smaller units, that we will call assets. For example, Golden API is composed of many libraries and templates; each one of them is considered an asset. Wiru (CI/CD) is composed of jobs and templates. Again, each one of those is considered an asset.

Metrics

With these definitions, we aimed to try to understand how our different squads and tribes were using our assets in their projects. After some analysis and trying different approaches, we got to define three key concepts to measure: applicability, adoption and up-to-dateness. By applicability we mean the degree to which an asset or product applies to different projects, while adoption measures -within the frame of the applicability- which projects effectively adopted our solutions. Finally, up-to-dateness is an index that allows us to see if projects that actually adopted an asset are using the latest available version. Each one of these concepts deserves a more detailed explanation.

Applicability and Project Types

To properly understand applicability we first need to introduce another concept, namely "project types": these are abstract architecture patterns that we defined for Naranja at the software component level. Examples of project types include: "microservice", "frontend", "bff", "qa-automation", etc. Also, each project type may have variants, like the platform used (serverless or container-based). But it’s important to be clear that these types are agnostic to the actual implementation.

That said, our definition of applicability is associated with project types, as it is indeed resolved by a matrix that relates assets with project types, thus obtaining each asset’s applicability. For example, let’s suppose we have ten projects in our organization. Three of them are of the type "frontend", four "microservices" and three "bff". On the other hand, we have an asset, "naranja-angular-authentication-module", which is responsible for managing front-end authentication. This asset then we’d say has a 30% applicability (3 applicable projects / 10 total projects = 0.3) - for it can potentially be used in 3 projects (despite the fact of it being used or not, which we’ll cover later on).
There are also some details around how a project declares its types, that we’ll review along with some implementation details.

To complete the idea of applicability, another important definition that we found along the process is that of aggregated applicability. It took us some revisions to arrive at the conclusion that, when we try to understand applicability on a more global level -for example, to understand the applicability of a product as a whole- using averages did not properly reflect the overall usefulness.

Wiru (our CI/CD product) is a good example to explain this, as it is a product that takes care of a cross-cutting concern, it has some assets that apply to certain project types, and some that apply to others.

Using this example, assuming Wiru has three assets (one for microservices, another one for front-end and another one for BFFs), if we calculate its average, we’d obtain a 30.5% applicability since (10/3 + 10/4 + 10/3)/3 = 3.05. But if we observe carefully we’ll notice that Wiru, as a whole, applies to 100% of the total projects. We believe this is a more clear indicator of the possible reach a product can have. It’s basically determined by the fact that "at least" one of a product’s assets applies to a project.

Finally, it’s important to mention that some projects, because of their particular nature, will not fit within our established types. These are then categorized as "others" - but our products are generally targeted to solve problems within the standardized typologies. With this idea, we may finally clarify the applicability concept as a way to determine the reach of a product or asset within the context of an organization, and ultimately, the degree of alignment between our products and the needs of our clients.

Adoption

Once applicability is determined, we can measure adoption, which is again a relation between an asset and a project / repository. We can determine that a node.js project - for example - adopts an asset by observing it’s package.json and checking if the asset (if it’s a library) is being declared as a dependency. We can use a similar approach with our Wiru jobs - we’ll just look at a project's latest pipeline and determine if a certain Job is being run.

It’s important to note the adoption is measured on top of applicability. That is, again following the example, let’s suppose that from our three front-end projects, only two of them use the authentication module. This means that the adoption for that module is 66% (2 adopting projects / 3 applicable projects = 0.66).

We conclude by noting that, in this way, adoption gives us a measure of teams’ happiness with the solution we provided to a specific problem, since they decided to use it.

Up-to-dateness

Finally, within the frame of an asset’s adoption, we are interested in knowing which projects are leveraging its latest updates. We achieve this by looking at the used version and comparing it with the latest available version, thus obtaining an up-to-dateness index. We tried different models for this, following semantic versioning - where each component of the version may have a different weight. But since some products had different up-to-dateness policies, we decided to calculate this index by simply considering a project updated or not updated. This is done based on an asset level configuration, that indicates to look into the patch, minor or major version.

This metric is especially useful for teams to have visibility of the technical debt they may be accumulating within their developments, and to incentivize them to use the latest solutions.

Segments, Lifecycle, Asset Types and Categories

Before understanding the design of our implementation, I would like to introduce a few more definitions.

We found it very useful to define public segments, since each team may have different realities. For example there are some squads that do COBOL development, or other ones that are devoted to Information Security: the majority of the projects they work on are very different from our standard typologies. They are publics with specific stacks and needs that showed up once we inspected the above mentioned metrics. Therefore, having them grouped separately helped us to understand our data in context.
Another important dimension to consider is the lifecycle of a project - development, production, abandoned, etc. This allows us to further filter our views and refine our analysis.

And finally, a quick explanation on asset types and categories. By types, we mainly think of the way an asset is delivered to a project (think npm package, docker image, etc). It’s crucial for our automation tool to be able to determine if an asset is used in a project. Categories are then a way to classify assets within their types: within npm-packages we have some that are serverless framework plugins, or pulumi components. These enable some further filtering and grouping.

Implementation

Our implementation was mainly serverless: a lambda function will run every day to gather information from GitLab, process it and put the results into a CSV file in an S3 bucket.

One interesting implementation detail is that we are leveraging Spotify Backstage’s catalog-info.yml file schema to add meta-data to our repos. That’s where we obtain project types from, along with template usage, lifecycle stage, etc. Further information is collected either from GitLab’s APIs (branches, projects, pipelines, jobs, etc) or by accessing the repository files, parsing and analysing them (package.json, serverless.yaml, gitlab-ci.yml files).

Finally, we set up two main consumption methods for the data. One is a direct generic integration to Power BI via AWS Glue and AWS Athena, and another one is an API that performs specific queries to AWS Athena, and then returns them via a REST API - which is used by a widget within our Hygieia dashboards.

Visualization

We worked on two separated visualization strategies, one for each audience we tried to reach. On one hand we had the squads that use our assets, which were already using Hygieia dashboards as their overall metrics visualization solution. For them we designed a graphic that would allow them to quickly understand where their projects stood in terms of these three metrics:

And then also an interface that will let them drill down into the specific details:

On the other hand, we have our internal platform product owners, who need to understand how their products are used, from a product / asset point of view. For these we created some Power BI dashboards, with several interactive data tables and graphics, which allow us to perform different levels of analysis.

Outcomes

We started getting actual figures from this solution in November 2019. Since then, even though we are still iterating over some details, we saw some interesting outcomes - mostly related to how this data enables better conversations, by backing with facts what otherwise could probably end in an opinion war.

The first one is merely a consequence of the implementation process, but not least important: we got our whole organization on the same page regarding what we wanted to measure and the meaning of these numbers. And at the same time, a complete consistent catalog of products and assets was built, along with some basic data regarding all our projects, which was not centralized yet.

Then, having these metrics was a fundamental milestone for our negotiations with upper management. Since it is usually hard for us to convey the actual value that our platform provides -and even though a precise ROI report is something that we are still working on- having actual numbers of how our products are being used was key to come up with agreements on how to organize and allocate resources for these teams.

We found that this visibility is also useful at the squad level, since it allows every member of our internal community to clearly understand what is available for their projects to use and what other developers across the organization are doing in that regard. In this context, some tribe leads go as far as setting adoption or up-to-dateness levels as part of the objectives for the projects they oversee, as they encounter adherence to our internal platform improves their overall delivery and operational performance.

Applicability is key for us as platform product owners to identify opportunities of new products we could create. By having a clear landscape of where our products do not apply, we can then analyze the need of those uncovered projects and propose a solution for them. As a concrete example, by observing our data we realized that the Data Analytics tribe had several similar projects that were still not templated, and since they had a few more similar repositories to come, we teamed up with them to address the opportunity.

Also as platform product owners, we closely watch adoption rates to identify teams that may have not decided to use an asset. This helps us to proactively start a conversation with them to understand if the product does not resolve their specific use case or if maybe its usage was not quite clear and help them with the adoption. From these interactions, usually new ideas emerge for our teams to implement. On a more strategic level, this year we worked on a Tech Radar where every Tribe Lead and our Platform Leads were involved. The adoption rates we observed were utilized as an input to define the status of each product (whether it should be in the adopt, trial, assess or hold ring), thus shaping the overall technology roadmap.

Next Steps

To conclude, I’d like to mention some points that emerged during the process and that are not part of the current implementation, on which we will surely be working in the future.

An important one is trying to come up with a weighted model, which applies for both assets and projects. In terms of assets, we noticed that some of them may be more important than others within the context of a product. So in trying to determine if a product is completely adopted, we may want to somehow weigh its assets. This also applies to projects, since some of them may also be more relevant (for they have more traffic, or commits, or lines of code - we are still thinking about these).

Another one is then using these figures to feed an economic model that would eventually be able to determine the return on investment for each asset, since it is usually a function of how many times it has been used.

And finally, we have the idea of using these in some kind of correlation model, that allows us to analyze the possible impact of our adoption metrics in other development and operations metrics (deployment frequency, lead time, mean time to recovery, etc).

About the Author

Ignacio Cavina started his career as a freelance web developer back in 2005, and grew as a software consultant within Globant, where he eventually started focusing on helping organizations improve their Developer Experience. He has intervened as a key player in the development and adoption of internal platforms including Frontend, Backend and Serverless Infrastructure stacks, for companies like Southwest Airlines, LATAM Airlines and Naranja. With a background in film, TV and design, he aims to provide value by bringing user experience and communicational sensitivity to the enterprise software space. He lives in Córdoba, Argentina, where he is also part of a garage rock band with his son Galo.