Testing Games is Not a Game

Gaming is nowadays the biggest and most lucrative entertainment industry by far. And there is a reason for that. Video games are not only a piece of software; games are experiences. They are designed to convey a specific feeling. To drive the player to a specific and different mental space. A door to an alternative universe, a social network, an interactive book or movie. A living organism made of code. Video games have become the preferred hobby of the newest generations. We love video games!

That passion and love for video games is what makes most people think that being a professional game’s tester is a privilege and a luxury within the reach of very few lucky people; a task easy and delectable, even better than that, considering that you get paid for it. Nothing could be further from reality; a professional game tester’s life is complex and often hard, partly because of how the industry has evolved, but also due to the nature of video games itself.

Ever since the first game was created (Pong!) until what we now envision as a modern "video game", the complexity of these "software applications'' has been increasing exponentially. This entails a series of challenges and situations that force testers to become all-terrain superheroes, when it comes to being flexible, adaptable, and resilient. Moreover, testing video games goes beyond (or differs in general from) what Quality Assurance means and represents. It brings a new subset of responsibilities and skills intrinsic to the gaming industry. 

I would like to give you some insights about the industry, the role of the game tester, some thoughts on challenges, and finally, my learnings. Let’s go ahead, press the start button!

So what are the differences between a QA tester and a game’s tester?

These two roles are quite blurred in the industry, and sometimes there is no clear answer. It really depends on the company and their processes.

Most companies, I would say, have both roles in mind and care about the two, usually as different roles very well-defined in their scope of responsibilities. Some companies merge the responsibilities into just one role that takes over Quality as a whole.

So, we have both:

• QA testers dedicated exclusively to working within the usual QA spectrum that we all know: code, pipelines, automation, code quality, monitoring/observability, testing activities and such. This is the role that matches 100% with QA’s outside the gaming industry. The focus is to deliver a healthy application, free of bugs, in a timely manner. “The code works”.

• But also “game testers” (the ones I will talk about in this article), dedicated to interacting with/playing the game as close as a final user would do. It doesn’t mean this role doesn’t require technical skills, but its focus is much closely tied to the final experience that is delivered rather than technical endeavors. “The game feels nice”.

The truth is, in a video game the code could be 100% perfect with no flaws, but if the experience it delivers during playtime is poor or wrong, then the quality of the code means nothing. What if Mario could jump higher than originally implemented in Super Mario Bross? The game would be too easy and probably not engaging, even though the code behind the jumping mechanics works perfectly.

Testing video games is hard because the testing scope goes 4D (adding the expected emotions it should create in the player) and requires an active effort from the tester to be aware of the whole context, and not just a specific functionality. Graphics, sound, music, game flow, game design, game balancing, performance, storytelling, and on top of that, of course the game itself as a whole and its mechanics.

That is the biggest challenge: being able to remain focused on all these areas at once to understand if the game “feels right”.

I would say, nowadays the second biggest headache would be being able to find the vulnerabilities of the game. Gamers are cheaters by default and a big portion of them will try to exploit, cheat, or hack the game. And that of course will ruin the experience of other players and/or have a direct impact in your company’s revenue. 

The challenges of testing video games

Games are getting more and more complex with the years. And gamers are a very demanding public. For those titles labeled as AAA (tripleA, high-budget projects) we are expected to deliver novel mechanics, mind blowing gameplay and exotic plot-twists. With each iteration, testing all of these becomes harder and the established ways of working need to be assessed and tweaked. That is quite hard, taking in consideration there are so many different kinds of games that it would be almost impossible to unify the way a game tester works.  Committing to a general agreement on how to tackle testing processes, tools or even a job description with required skills is not feasible at all in the industry. From one game to another, from one game’s company to another, the required skills vary, the role changes.

Also, due to the pretty common overuse of test cases and testing documentation, alive games (those that have constant patches and versions over the years) grow exponentially and rather quickly into monsters. Game testers are usually forced to come up with better scoping techniques and risk/impact based testing. It opens up space for gaps where quality falls down with the consequent impact in gamers’ happiness.

It’s worth mentioning as well, for PC games, it is impossible to have a complete compatibility testing matrix. Can you imagine the amount of combinations of Hardware+OS+drivers+peripherals+networks possible in the market?

On a different note, it’s worth talking also about game testing as a career, and the industry itself. The biggest challenge for a game tester is to keep growing over the years, stay employed, and avoid getting burned out. As we mentioned already before, a game tester’s skills do not translate directly to a traditional QA tester role, so it is hard to transition out of the gaming industry.

Most game testers join the industry out of passion, and that also fuels their day-to-day work life. This is good and bad at the same time, since quite often newcomers tend to overlook working conditions, salaries, and hiring practices, just because they are to do what they love; their idealization of what testing games entails. To this add on the pride of working on X game or Y company.

Testing games; applying testing practices

Without even noticing, most game testers use an approach similar to the “automation pyramid” where we work from “units” to “e2e” tests. Let me give you an example:

Imagine a 3D world where you control a human-like avatar that can do several things: walk, run, jump, sit down, crouch, interact with objects, wear clothes, fight, swim… you name it. And there are also other things in this world: NPCs (other avatars controlled by the system), animals, items, rivers and seas, vehicles, a weather system, a landscape that can be altered...

How do we ensure that the whole game world works as expected? Thankfully, we have the power to load and run specific parts of the game in isolation, with game editors, and test only small sections of it as if it were a “unit test”.

First, we would load the avatar on an empty world. There’s just a plane to walk over and a sketch of the avatar that represents the player. At this point, we can just check that this draft of the character that we as players will use to interact with the rest of the game world works as expected. We will assume anything based on this skeleton can be, by extension, considered tested. This is where the fun begins: you add clothes and check if your avatar can wear them, then you add a horse and ride it. Then you add some mountains and check that you are able to ride uphill... then it rains! And so on.  

This part of the testing is the famous “jump a thousand times against the same wall” example that we use to explain how repetitive testing games is. A game tester’s mind really needs to be resilient.

We also have some more peculiar ways of testing involving users and communities. Whenever we have something that can be considered “playable”, it is a common practice in the gaming industry to run alpha/beta testing. The purpose of this kind of activity is to let users play our initial version and provide us with feedback about the game, bugs, improvements, or feelings. We also use them for performance purposes, to check if our server load behaves as we expected and the system is looking crisp, or to test the game in several different PC/Mobile configurations.

Automated regression testing on games

Let’s start by stating that it is indeed possible to automate regression testing for not-so-complex games or for games planned very cautiously and extremely well documented. Software is software, and there is always a way.

That said, having such a fine level of detail and planning never happens. Creating games is very agile when it comes to the design and ideation of the game, but not so much if we talk about final implementation of the code. This means that planning for a well-defined automation solution needs to wait until the game is almost done, which proves to not be useful or valuable at all unless your game will keep growing over the years… which is something you will not know until the game reaches market and the playerbase decides if they like it or not.

To be clear, I’m referring to automating the “game as an experience tests” point of view. The actual systems and software behind it can be definitely automated; usually with in-house software as there are too many company-based solutions in game companies.

For the rest of the game experience, we try to come up with “scenarios of truth” based on facts (“The player likes this or that”, “We want this level to be hard”, “It should be possible to buy a sword within the first hour of gameplay”) and the data that support these scenarios, and then we automate them in an e2e fashion.

Artificial intelligence is also definitely coming into our world to help with this. Imagine you need to test “Tetris” and you come up with an AI that plays the game for you and tells you, based on human-like parameters, how difficult the game would be based on UX personas, what the likelihood of someone using a booster is, or what the average engaging rate of a specific level is. This is already happening!

Examples of Video game bugs, glitches and exploits

Just so I can illustrate better the kinds of issues a game tester has to face, let me show you some more specific examples of defects that reached final users. Some of these deserve standalone articles and a very deep dive into the reasons behind it, like the very well-known “Corrupted blood plague” from World of Warcraft, or the hilarious “Missingno” from the vanilla Pokémon games.

But for now, let’s focus on the main kinds of issues a game tester finds, starting with issues related to “We never thought about it”, for example, “Fallout 4 flying glitch”

“What is the problem here?”, you might be thinking. Well you are not actually supposed to be able to fly in Fallout4. It completely breaks the game! But then, how is this possible?

Well, it is quite easy to understand once you have figured it out. Basically, while you play the game you are able to grab and move objects; that is an intended feature. You can lift these items (an engine in the picture) and move them around. You change its position vector in the game's world. So, what happens if you first jump over the item and then grab and move it? Exactly, you can now fly. The item will become your flying carpet. And why did this defect reach players? Mostly because nobody actually thought about it during development. This happens quite a lot; as said before, games are humongous. The number of things you can do are so many that it is impossible to go through every single case.

Then we have another usual foe, “The game engine does not work as expected”. Most companies nowadays use standardized game engines (unreal, unity, godot, cryengine, lumberyard…) to create the foundations of their games. And while this is great and makes our lives easier, it also means you do not have control over most of the bases of how the game world reacts. Texture glitches are quite common, and rendering is a pain. But also, the physics are quite badly understood and used. Together with the inability to have a compatibility matrix, it ensures fun when a glitch like the ones below occurs.

Of course, there are also bugs strictly related to bad coding, errors in the code. A good example would be the “Nuclear Gandhi'' bug from Civilization. It is very well explained in Wikipedia’s Nuclear Ghandi article: 

[…]each leader's artificial intelligence in Civilization had a parameter that described his or her aggression on a scale from 1 to 10, with 1 being the least aggressive and 10 being the most aggressive.[…] Indian leader Mahatma Gandhi was the only leader in the game with the lowest possible aggression rating of 1 and, as a result, was only able to wage defensive wars. Once the AI changed its government form to democracy, which was preferred by peaceful nations such as India, its aggression level decreased by 2. In the case of Gandhi, this would lead to a negative aggression level of −1. However, the aggression level was stored as an 8-bit unsigned integer variable that could only store values in the range from 0 to 255 (or 28-1). The negative value would therefore result in an integer overflow, with the value being stored as 255 and Gandhi becoming about 25 times more aggressive than the most aggressive leaders in the game. […] This led to India suddenly attacking other civilizations with nuclear missiles.

Finally, we must talk about balancing and metagame. While these issues are not “bugs” per se, they do have a very powerful impact on the players and thus, define if a game will succeed or not. To not spend too much time talking about this topic, let’s say there is a difference between what you had in mind when creating the game, how you thought the game should be played, and what the players found to be the most optimized way of playing the game. The more unbalanced your game is, the more defined the metagame is. In practice, this means there is one specific tactic that beats everything else and playing in a different way will result in a worse or poor experience. Game testers need to have a very keen eye for this and be able to provide feedback on how narrow the metagame is, what things are OP (overpowered), and how players could be affected by it. Sometimes only top-notch players will take advantage (making them even stronger!), and sometimes it is payusers in free-to-play games getting access to incredibly strong items or mechanics, which means non spenders will get angry.

In general, if the final users (players) do not like something about the game, that instantly becomes a bug,. and your testing point of view was probably incorrect.

What I’ve learned from testing games and final conclusion

I am grateful for my experience as a game tester. I’ve learnt so many things and I’ve had so much fun!

If I had to come up with a list, it would be something like this:

• Your customer should be your number one priority; anything you do should be focused on them and nothing else matters
• You must be open to change. You need to be flexible; “Be water, my friend”
• There is no “impossible”. Just something you need to look at from another perspective
• Testing is easier if you love what you do and what you’re building
• Teamwork is not optional. Without a team I would have failed at everything
• Keep trying, again and again and again. Do not surrender, do not forfeit

As a final note, bear in mind that the gaming industry is wild and grows furiously fast. I have presented you with my experiences and what I know from friends and former colleagues, but there is always much more to it. 

If you are as passionate about games as I am, and you are considering becoming a professional game tester, go for it! Beat the game!

About the Author

Ismael Sánchez Villar loves combining two of his passions, testing and video games, and making a life out of it. Happily working as agile test lead at Activision-Blizzard-King, he keeps working towards helping the QA community grow, thus growing himself in the process. During 2020, he participated as a speaker during Agile Testing Days and other festivals and conferences. He is quite active on LinkedIn and always looks for new ways of contributing and learning. 

“Challenge me!” - Ismael Sánchez