Considering Risk in Video Game QA

QA testing smaller titles from independent or small studios is a different animal from testing large-scale, AAA titles.

The game is smaller (sometimes), its player base is often smaller, and its budget can feel microscopic by comparison.

Unfortunately, the impact of a critical, game-breaking bug can be just as or even more devastating on a small studio’s livelihood as it is on a large one. Large studios have the development staff and PR departments to help them bounce back from problematic missed bugs -- small studios don’t always have the same resources to draw from.

So if you’re a small studio, how do you make sure that you’re getting the biggest testing bang out of your buck on a project?

By taking risk into account when designing your test plan, and setting your risk threshold to an appropriate level for your player base, project scope, and budget.

WHAT IS RISK?

Risk, at its core, is way to measure how “dangerous” an area of the game is -- or how likely it is to generate serious bugs.
 

Most methods of determining risk will take two factors into account:

When considering the effect of an issue, it’s important to consider both the user experience and the impact on the development studio and staff.

A bug may make it impossible for a player to play, but it also might result in a game getting kicked out of console certification test -- which can push back schedules or cost money. Or, it might result in thousands of support tickets flooding the system that need to be addressed, or a loss of player trust in the developer.

HOW DOES RISK PLAY INTO QA TESTING?

So, how do savvy QA professionals use risk awareness to design test plans? It all boils down to deciding what and when to cover to get the most return on testing time.

Video games are so complicated that it’s nearly impossible to test every possible permutation of factors. Even if every combination gets covered over a game’s total lifecycle, this will be a needlessly expensive process, and it will be difficult to re-check everything when a change to the build is made.

Risk helps QA prioritize what should be covered first, and what should be covered at all. If done well, using risk when planning how, when, and where to test can substantially reduce the impact of not being able to test everything all the time while keeping the test budget lean.

At The Behemoth's test house, The Research Centaur, we use past experience (we tend to look for more experienced testers, with an average of 7 years in QA), information from the developers about what is currently being built or changed, and quantitative rating methods to determine risk. We also flex our test plans on a continuous basis to target areas of the game undergoing the most recent changes, while still covering the core of the game over time.

LEAN DOESN'T HAVE TO BE SCARY

If your test team currently uses a “full coverage” method and you’d like to substantially reduce your test plan budget, you can get started with risk-based testing by asking yourself two questions for each item on your test plan.

Rating the items based on a scale (High, Medium and Low) will do for a start, and then order your test plan items from lowest risk to highest. A High - High item, for example, should be ranked above a Medium - High.

Then, decide which areas to cover first based on your risk evaluations.

A sample basic test plan with risk-based evaluations and priority incorporated.

Where the real test hours savings come in is cutting certain test items out, or covering the right test areas at the right times in the development cycle to ensure the best return on investment for your test hours.

This is where the process gets trickier, because it takes a lot of experience to be able to cut test items out safely and maximize benefit based on studio activity.

FINALLY, A WORD ON HOW TEST IMPACTS YOUR DEV TEAM & GAME

Another thing to consider is the time investment of your development team -- you don’t want your programmers twiddling their thumbs waiting for bugs as your testers hit areas of test randomly!

It’s beneficial to the entire team if your testers are able to get to high risk areas with defect clusters (or large numbers of related bugs) first, especially in areas your programmers are already looking at. This ensures any required changes that would affect your system design are incorporated as quickly as possible.  

A metaphor for helping visualize the benefits of this testing method is the ripples formed when a raindrop hits the surface of a pond. The ripples that form around the site of impact will be the largest and strongest, and they will even out as the concentric circles grow larger and reach toward the edges of the pond.

A change to a game build is like this droplet -- it will have the biggest impact in the changed system, and smaller impacts on every system connected to the change.

Risk determination based on system change is like a water droplet making ripples in a pond.

So if you have limited test hours, it makes sense to test at the site of the change first, and then work your way out to surrounding areas as you are able to. This way, you’re likely to find the most bugs first, and will check areas with less return on your investment as you have time.

As you can see, determination of risk and effective testing can be a complex affair, taking into account inherent risk as well as the workflow of the team.

And this article is a simplification -- there is a lot more that goes into effectively using risk to determine your test plan. If you’re unfamiliar with risk-based testing, though, this method will give you some ideas about where to look for bugs first.

If you’d like to learn more, leave questions for me in the comments!