How can game developers use statistics, even before a game is released, to improve gameplay? BioWare's QA director Phillip DeRosa explains how the Mass Effect creator has set up and executed code-based monitoring of key metrics to test, analyze, and refine its projects through playtesting.
This article is about creating a window into player activities by using passive tracking systems to measure and improve the player experience.
We’ll start by looking at simple example from the television show to illustrate how using target audiences (potential customers) along with an iterative process of experimentation and observation can help improve product quality. We will then go over the people, tools and techniques that went into building this capability at BioWare.
When I originally read this story in the “Tipping Point” by Malcolm Gladwell I thought it was a good simple real-world example to illustrate using target audiences to test product during the development process without getting into complex implementation details which I cover in the second part of this article. I’ve kept this part brief and would recommend reading the book.
“In the late 1960s a television producer named Joan Cooney set out to start an epidemic. Her target was three-, four-, and five-year-olds. Her agent of infection was television and the ‘virus’ she wanted to spread was literacy.” 1 Sesame Street was the show.
There were a couple of must have’s in this equation - a good idea and a team of passionate and talented people to make it a reality. The team did a lot of research and wanted to bring in an expert to help measure and improve the show’s quality. They hired Ed Palmer, a psychologist and pioneer in the research of television as an educational tool. He developed a simple test process to measure the show’s “stickiness” (engagement) that is its ability to hold the audience’s attention and aptly named it “The Distracter”.
In one example, the test audience was setup with an episode of Sesame Street which was setup next to a slide show intended to steal away their attention. The audience was observed and time-stamped notes were taken whenever a portion of the test audience lost interest. The information was then passed back to the development team for review and resulted in many improvements.
The process was simple – hire an expert, define a test process, test the product with the target audience, objectively observe and recording findings, review, improve and repeat as needed. The show’s longevity is a testament to its success after some 37 seasons and 4134 shows.
Now applying this to interactive entertainment software is a little more complex – since each experience is different (non-linear) we need to look at a large number of individual samples and track a considerable amount of data (events and status information), aggregate the information, analyze it and generate valuable information out of it since the experiences can vary widely.
1 Malcolm Gladwell, The Tipping Point: How Little Things Can Make A Big Difference ( Little, Brown and Company ), p.89
This section covers the team of cross-functional people required to get this from idea to reality.
The test expert is our version of Ed Palmer and the one that owns the process and is responsible for gathering, analyzing, and effectively communicating the information. He should have strong game design, statistical analysis and test development knowledge.
The internal customer is the requestor of the information and will usually be responsible for developing a portion of the product or providing feedback on it. In some cases this can be used to bring in customer perspectives on divisive issues.
The play testers are the carefully selected group that represents the target audience. Selecting play testers for the target audience is an important part of the process. I’ve listed some interesting articles on this subject at the end.
The play test coordinator is responsible for planning, organizing, and managing the play-test sessions according to the test expert’s requirements. This includes everything from finding the play-testers all the way to ensuring they are compensated.
The developers design and implement the tools and technology that allow for the efficient gathering and analyzing of all this information.
BioWare's upcoming Xbox 360 exclusive RPG Mass Effect
The task of gathering, aggregating, and analyzing all this information is much more effective with tools and technology support. Simply put this is a logging system. Pick a database, logging system and reporting tool and go crazy. If the preference is the buy rather than build there are a few companies that provide products and services in this area as well.
Time is a precious commodity and having the right tools and technology will help reduce iteration time, allow for gathering and analysis of large amounts of data and increase the accuracy of the information as well as the speed at which it is delivered.
Our first foray into this was a manual process to gather information on some of our previously released titles with the help of a few less-than-enthusiastic testers, a spreadsheet and an electronic stopwatch. This proved to be a valuable exercise and served as our proof of concept. We showed this around and saw enough value to proceed building this capability. Building the tools and technology fits into our departmental goal of moving low value time intensive tasks onto computers and allow our skilled people to focus on high value analysis tasks.
What questions would you like answered about the product? Many developers want to know how customers are using their products and what they like or don’t like about them. The passive nature of these tests is the reduced bias since there is very little interpretation required (it’s pure).
This is a good start point to help define the requirements. Draw up an example report and then worry about the implementation. It is important to develop a credible process – one that is objective, simple and understandable.
This paper focuses on tracking the player experience but logging systems like this can be used to track test coverage (a favorite of mine), performance and a variety of other elements.
In the above picture you’ll find the process de-mystified. The phases are self-explanatory. Test passes are often done in a controlled environment with a large set of testers on a specific build of the product with a clear goal. The hard part is actually defining what to measure.
In this example we wanted to know what activities players engaged in and how long they spent in each activity in a full playthrough. We had some internal documentation that contained valuable information about building RPG experiences however the percentages of time players spent in those were quite off. This allowed us to add accurate information about timing to our knowledge pool (and in the document).
The Y value is total time and the X value is the activity buckets.
In this example we wanted to know what special powers were being used by players and for how long. This can be used to balance powers and find out what appeals (and what doesn’t) to players. Identifying root cause and coming up with appropriate solutions is up to those responsible for crafting the experience.
The Y value is the number of times the power was used and the X value is the character’s level.
These are just a few examples of what a system like this can be used for. From a QA perspective I really enjoy Test Coverage reports which get us accurate and timely information on what has been tested and what hasn’t.
It’s important to note that this is just one perspective and isn’t meant to replace other tried and true methods such as user surveys, expert review and first hand observation. When the information from all these processes is correlated it should provide a more complete picture of the experience and possible issues.
Playing games may be seen as trivial but making great commercially successful games isn’t. To summarize this in a few bullet points:
Involve the customer early and throughout the development process to gather player experience feedback both passively and through surveys to get valuable insights on product quality.
Develop a process and set of tools that allow for effective and efficient gathering and analysis.
Need good ideas and good people.
It’s cheaper and easier to identify and fix issues earlier (an ounce of prevention is worth a pound of cure). I will continue to repeat this mantra in all my articles.
Use a variety of tools, techniques and experts in addition to a system like this.
As Thomas Edison said, “There is a way to do it better, find it!” Be on the lookout for best practices inside and outside the industry.
Invest time and money in building tools that improve you capability and capacity to measure product quality.
Do see your QA team as a valuable asset by supporting them, including them, being open minded and you’ll be surprised at the results you’ll get. Now go and give them a hug.
“21st Century Game Design: Designing for the Market” by Chris Bateman (Demographic Game Design)
“Just Who Are Our Customers?” by Chris Bateman
“Hearts, Clubs, Diamonds and Spades” by R.A. Bartle
“Using the RITE method to improve products; a definition and a case study” by Michael C. Medlock, Dennis Wixon, Mark Terrano, Ramon L. Romero, Bill Fulton
“Metrics in MMP Development and Operations” by Larry Mellon
Read more about:
FeaturesYou May Also Like
