Conducting DORA surveys in your company can help you reflect on how you are doing software delivery and operation. The way you design and run the surveys and how you analyze the results largely impact the benefits that you can get out of them. Carlo Beschi shared his experiences from doing DORA surveys at Agile Cambridge.
The first time Beschi ran a DORA survey was at ASOS in 2018. They were already doing things to get teams to own the operations (and not just the delivery) of their software. The purpose of the survey was to help validate their focus, steer their effort, and make a stronger case to allocate additional budget.
We did not do a good job at pitching the survey to our developers, nor at helping the teams’ managers understand "what was in it, for them", Beschi said. The completion rate was 30-35%, and the quality of data behind the enterprise report was not too solid, he added.
The second they ran the survey, they made sure it went out with proper communications and worked with the managers of the teams to get them directly involved:
The impact of this was huge, as well as helping with the "paperwork" some managers became champions for DORA and helped pitch the opportunity to their peers. We got a completion rate of 70%+ and a much more reliable enterprise report.
In 2022 at Treatwell, developers and teams did DORA quick checks where Beschi supported them in reading the results and discussing them:
Most of our teams are pretty good, compared to the industry average; the quick check does not really offer any solid insights, besides a boost of confidence and a tap on the shoulder - "keep doing what you are doing".
Teams that - for very valid reasons - are less good at modern engineering practices are a little bit in denial. The score they got was "you are also doing pretty well" while we know they have challenges with the software architecture, with some of their testing practices, with their ability to deploy small changes in production on a regular basis that they need to overcome.
They decided to do a full survey where they needed to build their own model to derive capability scores, with an additional manual effort to create the reports:
We struggled to find the time to create those reports, we got the teams to wait weeks, or months, for the reports, and the discussions of the same. We lost momentum. And then we lost sponsorship.
Beschi said the surveys are really powerful. They help you reflect on how you are doing software delivery and operations - how "modern" you are, how much you are working the way the best ones are, and how much you are lagging behind:
I’ve seen individual contributors having great aha moments, after answering questions from the survey. Also, I’ve seen teams being very grateful for the powerful mirror that DORA - and the report you can generate from a team’s answers to the survey - offers them.
Survey design, administration and analysis is a discipline itself, something you should not improvise on, Beschi mentioned. At ASOS, they used the official DORA survey and had a DORA-certified third party do the analysis for them. At Treatwell, they explored coming up with our own survey, which they ended up discarding after a few weeks of discovery and some spikes:
It’s not just about the topic you are asking about. It’s that the way you phrase your question/statement matters a lot.
They used the public DORA survey and developed their own model to translate the answers to capability scores. Beschi mentioned that it took quite some time for experienced people to have a decent one that felt reliable while much less sophisticated than the one DORA / Google uses.
InfoQ interviewed Carlo Beschi about the results of the DORA surveys.
InfoQ: What were the main insights you gained from doing the DORA surveys?
Carlo Beschi: Back at ASOS we felt that we needed to get better at Monitoring and Alerting, across the board. The survey confirmed that, and gave us the confidence to make a significant investment of money and time the year after in this area.
At Treatwell the survey confirmed that, with a few exceptions, we were in a solid place in terms of continuous delivery. That test automation was a mixed bag though. And, that the areas with the lowest scores were documentation and security.
We kind of knew all of this, but seeing it "validated" is very powerful. Security is at the top of the agenda for us in 2024, and I’ve seen great stuff happening in this area in the first half of the year.
InfoQ: What additional benefits did you get from building your own model and survey?
Beschi: Doing our own survey "in house" at Treatwell we got access to every single answer to every single question. You have the whole data set, to "slice and dice".
At ASOS, I had, let’s say, eight developers from a team, answering six questions on Documentation, and all I got back was, let’s say, a score of 3.5 on that capability. At Treatwell, I had access to all eight people’s answers, to each of the six questions.
When you go and look at the answers, you get additional insights. You will not know "who has answered what," but you can play back to the team within the reports and debrief sessions.
For example, by saying, "Hey guys, look at this … at this question that says It is easy for me to understand our technical documentation, 3 of you answered ‘strongly agree’ and 4 answered ‘disagree’ instead." There may be something there for you.
Learning why people feel this way could be powerful, and help you look at what you can do to change things.