Improving the feedback time of a continuous integration (CI) system and optimizing the test methods and classes resulted in more effective feedback for development teams. According to Tobias Geyer, CI systems are an important part of the development process and should be treated as such.
Tobias Geyer spoke about improving a continuous integration system as a tester at the Romanian Testing Conference 2023.
When Geyer started at the company, the continuous integration system was an old developer PC that had been put under a desk, and nobody was responsible for it, with no time to do maintenance. Being slow it resulted in very long feedback cycles and developers ignoring the feedback from the CI system altogether, he said.
A quick win mentioned to improve the feedback time was to skip build steps during business hours that weren’t needed all the time and limit them to a nightly build, Geyer said.
A problem that took more effort to fix was the disk I/O bottleneck. The build did read and write so much data that the hard disk couldn’t keep up. Geyer mentioned that he got in touch with their IT department and moved the CI system from the developer PC to virtual machines in the datacenter. That allowed them to fix the disk I/O issue and scale the system to two machines, enabling them to run more builds in parallel.
After figuring out which tests were fast and which were slow, they split them into categories and ran only a subset of the tests as part of their fast feedback CI builds. The slower tests were moved to a dedicated build that was run less frequently. That way they still got the full feedback but they got the majority of it way faster than before, Geyer explained.
Geyer described how they optimized the test methods and classes:
Once we identified which tests were slow, we treated them like technical debt. We created "test debt budgets" for developers to improve the tests, which they approached in different ways:
1. Test data got trimmed down so that only the relevant data was kept, which shortened the setup time for tests.
2. Mocking was introduced in the tests, making it obsolete to load any test data in the first place.
3. The product code was made more testable so that the same checks could be done as unit tests instead of integration tests.
Geyer concluded that it’s not necessary to have a deep understanding of the technical side of things to have an impact:
I can use my testing skills of measuring, experimenting and collaborating to make changes for the better happen, even if it means that someone else has to do the actual implementation work.
InfoQ interviewed Tobias Geyer about improving the CI system.
InfoQ: Can you give an example of a build step that was skipped?
Tobias Geyer: The most prominent example was our product obfuscation. Each product artifact gets obfuscated before it’s delivered to the customer, and obviously the obfuscated product needs to be tested. Unfortunately, the obfuscation takes at least 30 minutes. We’re skipping it during the day, at night the products get obfuscated and tested that way.
InfoQ: What major changes did you do to the building process and platform?
Geyer: We migrated our build system from Ant and Windows batch files to Gradle. That was done mostly by the developers in my team with me taking care of all the parts that had to deal with executing tests.
We made an obvious but important change by introducing a test CI system. This allowed us to prepare and test changes to the CI system (like plugin updates, new build nodes,...) without interrupting the normal development flow.
InfoQ: How did you encourage knowledge sharing about continuous integration among different teams?
Geyer: I looked for people in other parts of the company who worked with or on CI systems and had a similar tech stack. We had regular meetings where we discussed the latest changes and the problems we were facing. Quite often some team had already solved a problem and their solution could be re-used.
It was great to see what the others were doing and the results they had. There were cases where teams wanted to introduce a change but met pushback in their teams. Being able to say "This other team made good experiences with it" helped to convince people.
InfoQ: What’s your advice to teams that are unhappy with their CI solution?
Geyer: Approach it like any other software development project. Make a list of the issues that bother you and treat them like bugs. Which means: Prioritize them, analyze them and then collaborate on fixing them. I’d like to stress the "collaboration" point - talking to our IT department was crucial to get some of the issues fixed.