Developers writing e2e tests can make code testable, provide fast feedback, and prevent bugs. Wix worked with their product managers, developers, and QA engineers to transition from QA-only testing, to having developers write e2e tests to shift QA left and deliver faster.
Yevheniia Hlovatska, QA guild leader at Wix, shared her experiences from helping developers write good e2e tests, and QA engineers to drive such change in their teams at 2021 Spring OnlineTestConf.
Hlovatska explained what holds developers back to create end-to-end tests:
The most important reason is that developers don’t really like to do it. Of course, they see the value of writing tests. But how can it be compared to feature implementation for millions of users? Especially if they’ve already suffered from flaky tests before.
Because of these reasons, I see many developers with no motivation to write e2e tests, Hlovatska said. If you make them write those - you will end up with bad tests, killing your velocity instead of increasing it.
Wix is working on a transition from QA-only writing tests on Java to having developers writing e2e tests in JS. They started by talking to people in different roles about this approach:
Developers need approval and time to write e2e tests. We came to product managers and told them we wanted developers to write e2e tests. And you know what management usually hears in such cases? "Less features". It was very important to explain that lack of tests equals technical debt, which equals slower delivery and less features eventually. And good tests lead to higher velocity.
The second thing we did was talk to developers. They were "ok" to do it, and we started. With time we’ve realized that developers are not always experts in e2e tests. Combining lack of knowledge with bad experiences using e2e tests in the past, we’ve got flaky tests and a bad reputation for e2e tests in some cases. So we’ve started to build best practices and education to improve.
The next audience we talked to were QA engineers. Previously they were completely owning e2e tests, they had great expertise in test writing, test scenarios design and all other activities in test automation. So, to be honest, once we said "We want to have developers writing e2e tests now," some of them were pretty frustrated. For some of the QA engineers it sounded like, "We will become manual QA engineers from now on and will not do automation anymore". So we’ve explained that they are the ones who can drive a new approach with developers writing tests in their teams as experts and it can be the next step in their growth.
The first benefit that developers get from writing e2e tests is testable code. Thinking about e2e flows while developing your feature is like thinking about your future users and the actions they will do, Hlovatska said. Testable code leads to less bugs and gives more space for QA engineers to do proper testing.
The second benefit Hlovatska mentioned was that e2e tests written by developers during the feature implementation can provide a fast feedback loop about the quality of their code. They won’t need to ask and wait for testing to be finished to acknowledge there is a bug.
Having developers writing e2e tests allows to remove the bottleneck and "shift" QA left. By shifting QA left, you can prevent bugs instead of finding them, spend less time on waiting and deliver faster with fewer bugs in the end.
So it is not just developers but the whole team who benefits from such an approach, Hlovatska concluded.
InfoQ interviewed Yevheniia Hlovatska about developers doing end-to-end testing.
InfoQ: What tips do you have for enabling and supporting developers in doing end-to-end testing?
Yevheniia Hlovatska: Here are my tips:
- Design coverage
Good test scenario design is 50% of success for your automation.
- Tests must be short and focused. If your test covers many steps and touches many different features, you will create flakiness and increase the debugging time for your developers.
- Tests must always keep the "circumstances" in mind. In our case, we run thousands of tests in parallel and we need to design accordingly.
- Coverage must be analyzed and improved over time. It is important to constantly analyze your test results, bugs, latest production incidents, and other data to improve your tests.
- Teach
It is pretty hard to control every test that is written. A scalable strategy is educating developers on how to do good e2e tests.
- Review tests at first. Code reviews will give you a lot of information that can be used to improve both tests and processes around it.
- Create best practices. Test project structure, naming conventions, good waiters and many others.
- Build your processes. For instance, what to do if a test failed because of the bug.
- Change the mindset
If your developers already had a bad experience with e2e tests, it might be pretty hard to change their mind about it.
- Start with one good test. If you have dozens of flaky tests - skip them all and refactor one to be stable. If you don’t have tests - just create one good.
- Make it a habit. Become a living reminder that tests are a part of our culture now. When we run fast, it is easy to forget about tests we agreed to write.
- Show some numbers. Pick some metrics you can use and show them early as your "return on investment".
InfoQ: How do testers think about end-to-end testing? What are their concerns when developers start doing it instead of themselves?
Hlovatska: Usually, QA engineers feel like it is a step back for them. But in reality, it can become an evolution for your QA process and QA Engineers.
In Wix, we have a very high pace of development. Our usual ratio of developers to QA engineers is 10:1. So if QA is the only one responsible for e2e tests, we see either a bottleneck in delivery or we write fewer tests and we don’t have enough time to support them.
Once we moved test writing and maintenance to developers, it created space for new tasks. QA can grow their automation skills, teach developers and improve existing tests. They can also start working on an automation strategy and improve it.
Also, it frees some time to do deeper testing, explore products and find important bugs you’ve missed before.
And of course, QA engineers can still continue writing tests if they want to :)
InfoQ: What have you learned?
Hlovatska: I’ve learned that in order to succeed with this approach we can’t just ask developers to write e2e tests. We need to:
- Give them time to work on it
- Design and improve test scenarios for them
- Teach them
- Build a good process around it
And just let them do what they really like to do - write code :)
