On Quality Assurance

               There’s a common question asked in interviews for game industry positions, no matter what the discipline or level of seniority: “What’s your favorite game?”. Too often this question is asked as merely an icebreaker, a way to segue into what is perceived to be more relevant questions to the position itself, or even how well the individual would fit within the culture of the team. The more discerning interviewer will note a few aspects of the candidate’s response, namely if they are direct in answering the question rather than claiming something to effect of what their favorite game is right now, and what the game itself happens to be. It should come as no surprise that I have witnessed many interviewers make up their mind about a candidate solely from their response to this question, especially if they name a game that the interviewer either worked on, or also holds personally dear to them.

                Let’s touch on that last point for a moment - how easy it is to evaluate and sum up a person based on what they define as their favorite game, and the subtlety of how they objectively or subjectively determine what makes that game their favorite. What is going on here beyond a simple bridge of common interest? To choose a favorite game, out of the hundreds of thousands ever produced, is ultimately to make a judgment. A judgment such as this can be taken lightly or not, depending on the qualitative discernment skills of the person and the more objective or subjective they are in internally describing something meaningful to them – it is just as valid for someone to rattle off a list of gameplay features, or graphical fidelity, or cite the soundtrack with comparisons to other games collectively held in high esteem as it is for someone to exclaim how they would play the game for hours with their brother, yelling at each other and feeling accomplished when one beat the other. Both are valid because both are qualities unique to the game, both emergent from the effort and design of the game itself being made interactive by the player.

                 How then to come to terms with the monolithic term that is Quality Assurance? How then to even find qualified people to then make those judgments on behalf of the developers and of the players? To begin to answer this it is important to dissect the root definitions of Quality Assurance and how it is manifested across the industry, primarily in three schools of thought:

Quality Control

                It would be remiss to ignore the terminology that has emerged on the scene between these three disciplines. Control implies power, implies objective reasoning to maintain that power, and most importantly implies a sense of urgent defense. To draw another example, Air Traffic Controllers are named such as they aim to control the inherent chaos of the thousands of high-speed machines flying around with the potential to crash into each other – they are not Air Traffic Assurance as the complexity of the system and, most importantly, the time in which they can act, is by its nature reactive and tense.

While no one is in danger of a collision in the world of video games, there is the real danger of taking thousands of hours of developer efforts (not to mention perhaps millions of dollars) and releasing something substandard to the public, leading possibly to the closure of a studio and the endangering of many livelihoods. This danger necessitates a force that can act as its’ opposition – hence the combative nature, and hence Quality Control.

How does this manifest? In a broad stroke, almost purely objectively. The types of people you want to ensure Quality Control are those who are now at the point of objectively measuring your product against what else has previously been released that has been objectively defined by the market and by game journalism as high-quality. This is a very important function to ensure the continuation of a business and provides a high degree of separation from personal bias with the product to produce something competitive.

The unfortunate downside here is the complete elimination of subjective measurements of the title (as anyone out there in QA/QC knows, dropping a suggestion bug while near certification is not just pointless, it can be met with active ridicule) and placing the highest possible quality bar in the hands of the developers themselves – who by necessity have tunnel vision and personal bias. Some developers can handle that, some can’t, and that is a risk that can manifest itself in wildly unpredictable ways.

Quality Assurance

                Moving from the wholly objective, we enter Quality Assurance. Here we begin to see the injection of subjective measurements of a product, the most important being ‘is it fun?’, ‘does this feel right?’ and ‘does this make sense?’. It is the role of someone in Quality Assurance to be able to bridge the gap between the urgent objective and the theoretical subjective – to be familiar enough and think broadly enough across the vast potential player base to anticipate reactions and translate them into actionable goals.

That act of translation is QA’s greatest value, to objectify the subjective with sound judgment. The sense of urgency and control meanders depending on the risk and volatility of the game itself, so often you will find those in Quality Assurance doubling as agents of Quality Control in dire times. The tension of forcing these personnel into that role for a significant time runs the risk of the atrophy of their ability to step back for a moment, put themselves in the shoes of the player, and communicate the necessary steps back to the development team to resolve the issue. At the same time, the atrophy of being able to stem the tide of a volatile product can lead to personnel focused entirely on the theoretical – too detached from the realities of the defects presenting themselves to be able to address immediate concerns effectively.

To touch upon terminology again, there have been many pushes within the game industry to move away from the term Quality Control in favor of the term Quality Assurance, for a myriad of reasons that are curiously almost entirely subjective in nature. Assurance as a word implies a knowledge base exists of what the game is meant to be, and that those with the title can steer the ship of development effectively towards that definition. This sounds more nuanced, more progressive, and more appealing to those who do not wish to acknowledge the potential for volatility in their products and as such require those who can control it. Rather, preventative measures are viewed as paramount and all encompassing. While such measures are necessary, the nature of game development is complex enough to warrant both – a truly bad time awaits any team who has convinced themselves they are above that distinction.

Development Quality Assurance

                Perhaps because of the frequent lumping together of QA and QC, there has emerged a loosely defined group of Quality professionals that sit embedded within a development team (often engineering teams) who are looked to as a strange mix of arbiters of quality judgment, examples of technical proficiency and objective reasoners. These Development QA folks have perhaps the greatest level of subjectivity to their day-to-day efforts, though as we find ourselves going across the spectrum of Quality we find that the need to translate into objective notions plateaus here rather than drops off entirely.

                If there is a commonality between all members of Development QA, it is the valuation of long-term vision as a preventative measure and the management of their own judgments. The line blurs considerably as the Development QA professional becomes equally as competent as a gameplay engineer at their chosen programming language – able to not just identify an issue but fix it directly and in a way that anticipates further issues. When this line blurs, so too does the issue of domain and jurisdiction – often the bane of these QA teams is when they anticipate too much, or too broadly, and come in conflict with making the game. A common pitfall is always to explore the notion of perfection, severely reducing the ability for efforts such as rapid prototyping as the Development QA personnel become embittered at the instability of what is being presented to them.

                Development QA is uniquely posed to ask the questions that lead to the right questions, and as such a curious hierarchy emerges when they meet QA and QC teams that have been contracted in some form to also work on the game. The friction of ideals, between the objective and the subjective, manifests greatly here in the fields of prioritization, definition and root cause analysis of issues. The Development QA personnel may see the same crash on boot the QC team encounters yet can read the code well enough to determine it is an issue with say, an animation system. As the QC team would only understand that a crash is occurring (perhaps I’ll explore the differences of black/grey/white-box testing in a later blog post), their input can appear too surface level to the Development QA team to take seriously, or even take as baseline informative. The point here is to realize that at the end of it, a crash is still occurring on boot. The ultimate action of the QC team is to ensure it is known, while the ultimate action of the Development QA team is to ensure it never happens again.

Conclusion

                We return to the question again, “What is Quality Assurance?”. Having been fortunate enough to have worked across all schools of the Quality sphere, I have given a different answer at different points in my life. I’ve found the starting point must always be personal, “what does quality itself mean to me?” and I find that is more difficult to truthfully answer than many technical questions I’ve been asked in an interview. If I am to assure, control or develop Quality, I need to understand that no matter what my definitions, they will all be judgments. If I am someone then to pass judgment, should I not weigh each issue with the gravity it is due? Should we not seek quality professionals who ponder the same things?

                Find a whiteboard and at the top of it write your favorite game. Then under it, divide the board into two columns labeled “Objective” and “Subjective”. Start with “Objective” and list at least three things about the game. Then attempt to define each of those three things in objective terms. Then define those definitions. I’d recommend you stop when you run out of room, as you will always be able to dive deeper. Next, do the same thing for the “Subjective” column and go as far as you can. Finally, take a broad look at everything you wrote and ask yourself the most important question:

                “Which of these things make this my favorite game?”

                When you have answered this truthfully, draw a heavy line across the board, splitting those that fit your answer on the top, and those that didn’t make the cut on the bottom. That, dear reader, is Quality. That is an informed judgment, and that is what makes games stand the test of time.