Quality is not only about not releasing bugs to production, it is much more than that. Quality is a product that is user-friendly, easy to access and use, has high performance/short load time, and is about code that is stable and easy to maintain.
Jette Pedersen gave a talk about how to ensure good quality products at Swiss Testing Day 2020.
Testing at eBay Denmark is done on different levels to increase user experience, as Pedersen explained:
User testing, where we for example go out and talk to users and ask them to use/look at a prototype, will give us more insights of the users’ needs and thoughts, and whether or not they understand our thoughts with the product. Performance tests will tell us if our site performs within our expectations. A/B testing will tell us which variant of what we have built the user likes best.
Pedersen mentioned that to make sure they are testing the right things, they use test design techniques. An example is to use equivalence testing combined with boundary value analysis. This will make sure that you test all valid/invalid partitions only once, and that all boundaries on each partition are tested with the right values for the valid/invalid partitions.
Instead of trying to come up with a total solution from the start, eBay Denmark does a lot of experiments to find the right way to build products. They set up a hypothesis, for example:
If we offer our users a new privacy product
Then we will increase seller NPS
Because they can choose if they want to show their home address or not
Next, they tried out a fake door test to find out if users would buy the product. The test showed that the users wanted the product, so they built it and then started their A/B split testing to find the right layout.
InfoQ interviewed Jette Pedersen, senior QA software engineer at eBay Denmark, about ensuring quality, dealing with risks that impact quality, what they have learned about quality, and enabling quality thinking throughout the whole process.
InfoQ: How can we ensure good quality for our products?
Jette Pedersen: We need to think about quality from the beginning of a project or task. To ensure that we build something that the user can and will use, we do a lot of experimentation, but before we start that, we need to think about different things.
- Who do we build it for?
- Are there special demands for performance?
- What are the requirements, and are they known to us at the stage of the project we are in?
- Are there any UI changes?
- What is the risk assessment for the area we are working in?
- New security precautions should also be taken into consideration
All of this we talk through in a kick-off at the early start of the project.
When the product is being developed, we often use TDD and pair programming. This helps us to share knowledge and quickly overcome any obstacles that come along the way.
To be able to ensure that we get fast feedback if anything breaks in production, we also talk about if we should set up monitoring and which kind of test automation is needed.
A rule of thumb is that all high risk and some medium risk areas (if possible) should be covered by some kind of automated test.
InfoQ: What are the risks that can impact quality and how can we deal with them?
Pedersen: There are many different types of risks involved in product development.
- Do we have enough knowledge about the product area that we will be working in? If not - do we have access to an oracle or where can we find the answers?
- How much experience do we have with the framework used to develop the product?
- Have we worked in the area of code before?
- Is there a deadline? If so, oftentimes quality will suffer. We are lucky that we rarely have deadlines.
The way we deal with the risks is to try to identify as many as possible when we start a task e.g. at a task kickoff. When talking about which risks could impact the task, it also helps us to identify if the task is in a high, medium, or low risk area, and we can then take action to mitigate the risks. It could be that pair programming could be a solution. It could also be that we need more people to help out either with programming, testing or who have the needed knowledge in the area we are working in.
InfoQ: What have you learned about quality over the years?
Pedersen: You need to think of quality right from the beginning of a project. It is so much cheaper to find any issues in the beginning than later in the project.
Also using a risk-based approach gives us a really good indication of what we need to do to ensure that the quality is high enough. It also ensures that we do not spend too much time on ensuring the quality, if it is not necessary.
Today, ensuring the quality is not something that lies only within the role of a QA. If you get the whole team to take responsibility for the quality, then you will have much more success in delivering high quality to the users, and you do not have the problem of the QA being a bottleneck.
I have also learned that performance and usability are essential for the product. If the product does not perform or the user does not understand how to use our product, then it does not matter if the product has high quality. The user won’t use the product again.
We also have a high focus on automation, so that we are able to do continuous delivery at a fast pace.
InfoQ: What’s your advice for enabling quality thinking throughout the whole process?
Pedersen: Get your colleagues to agree on what quality is. Help them to create a mindset as a QA. Ask questions instead of giving the answers.
Communicate with each other throughout the whole project. The more we communicate and help each other out, whether that is when we code or when we test, the better the quality the product will have. Use a risk-based approach to understand how much quality assurance you need to do, and at what level. Make sure your colleagues know and understand how to do a risk assessment.
Have faith that your colleagues are just as capable of ensuring the quality as you are.
