How can I make my game more fun for more gamers?

This is the question for those who want to make games that are popular, not just critically acclaimed. One (glib) response is to "design the games better." Recently, the idea of applying psychological theories as a way of improving game design has been an increasingly popular topic in various industry publications and conferences. Given the potential of applying psychological theory to game design, I anticipate these ideas to become more frequent and more developed. While I think using psychological theories as aids to think about games and gamers is certainly useful, I think that psychology has much more to offer than theory. An enormous part of the value of psychology to games lies in psychological research methods (collecting data), not the theories themselves.

I should clarify some terms here. When I talk about "psychology," I do not mean the common perception of psychology--talking to counselors, lying on the Freudian couch, mental illness, etc. In academia, this kind of psychology is called "clinical psychology." In this paper, "psychology" refers to experimental psychology, which employs the scientific method in studying "normal populations functioning normally."

But before I talk about how psychological research methods can help improve games, I need to first explain more about how psychological theories are helpful, and the limitations they have with respect to game design.

Psychological theories can be useful, but data are more useful
All designers think about what people like, hate and want. Some designers may be consciously using theories from psychology as part of the process to evaluate what people want, but most designers probably just rely on their intuitive theories of what they perceive gamers want.

The risks of relying on intuitive psychology. What I call "Intuitive psychology" is the collection of thoughts, world views, 'folk wisdom' etc. that people use to try to understand and predict others. Some examples might make this clearer--one common intuitive psychological belief about attraction is that "opposites attract." However many people (and many of the same people) also believe the opposite, that "birds of a feather flock together." Both of these ideas have some merit and are probably true in some ways for most people. But given that they are clearly conflicting statements, it is unclear which statements to believe and act on--which statement is true? Or, more likely, when is each statement more likely to be true? Does the degree of truthfulness for these statements vary by people? By situation? By both? The problem with intuitive psychology is that many intuitions disagree with each other, and it is unclear which world view is more likely to be right, if either of them are at all. You're just trusting that the designer's theories are close enough to reality in that the design will be compelling.

The insufficiency of formal theories of psychology. Formal theories of psychology have been subjected to rigorous testing to see when they map onto reality, and when they do not. In order for a theory of psychology to gain any kind of acceptance, the advocates have to have battled with some success against peers who are actively attempting to show it to be incorrect or limited. This adversarial system of determining "truth" and reliable knowledge employs the scientific method of running experiments and collecting data. Because of this adversarial system, formal theories of psychology are more trustworthy than intuitive theories of psychology--you know that they are more than just one person's unsubstantiated opinion about what people want.

But while theories of psychology from academia can be quite useful as a lens to examine your game, their limitation is that they are typically too abstract to provide concrete action items at the level designers need. This lack of specificity in psychological theories hasn't really hurt designers too much, because in the most part designers (and people in general) have a decent enough idea of how to please people without needing formal theories. I think very few people had light bulbs go on when they learned that Skinner's theory of conditioning stipulates that people will do stuff for rewards. But the work that Skinner and others did on how to use rewards and punishers well in terms of acquisition and maintenance of behaviors can be enlightening. But academic theories of psychology don't get granular enough to tell us whether gamers find the handling of the Ferrari a bit too sensitive.

An example of why academic theories of psychology aren't enough is in order. Skinner's Behaviorism is probably one of the most well-defined and supported theories, and the easiest to apply to games. (In fact, John Hopson wrote an excellent article in Gamasutra in April 2001 demonstrating how to analyze your game through behaviorism's lens.) One of Hopson's examples is about how players in an RPG behave differently depending upon how close they are to reinforcement (e.g., going up a level, getting a new item, etc.). He talks about how if reinforcers are too infrequent, the player may lose motivation to get that next level. However, how often is not often enough? Or too often? (Who wants to level up every five seconds?) Both "too often" and "not often enough" will de-motivate the player. Designers need to find a 'sweet spot' between too often and not often enough that provides the optimal (or at least a sufficient) level of motivation for the player to keep trying to level up. Theory may help designers begin to ask the more pertinent questions, but no theory will tell you exactly how often a player should level up in three hours of play in a particular RPG games.

Beyond Theory: the value of collecting data with psychological methods
So I've argued that the psychological theories (both intuitive and academic) have limitations that prevent them from being either trustable or sufficiently detailed. Now I'm going to talk about what IS sufficiently trustworthy AND detailed--collecting data with psychological methods. Feedback gleaned via psychological testing methods can be an invaluable asset in refining game design.

As I said at the beginning of this paper, the central question for a designer who wants to make popular games is "how do I make my game more fun for more gamers?" and that a glib response is to "design the games better." Taking the glib answer seriously for a moment, how do you go about doing that? Presumably, designers are doing the best they can already. The Dilbertian "work smarter, not harder" is funny, but not helpful. The way to help designers is the same way you help people improve their work in all other disciplines--you provide them feedback that helps them learn what is good and not so good about their work, so that they can improve it.

Of course, designers get feedback all the time. In fact, I'm sure that many designers sometimes feel that they get too much feedback--it seems that everyone has an opinion about the design, that everyone is a "wannabe" designer (disguised as artists, programmers, publishing execs, etc.), as well as everyone's brother. But the opinions from others often contradict each other, and sometimes go against the opinions of the designer. So the designer is put in the difficult situation of knowing that their design isn't perfect, wanting to get feedback to improve it, and encountering feedback that makes sense, yet is often contradictory both with itself and with the designer's own judgment. This makes it difficult to know what feedback to act on. So the problem for many designers is not a lack of feedback, but an epistemological problem--whose opinion is worth overruling their own judgment? Whose opinion really represents what more gamers want?

Criteria for good feedback and a good feedback delivery system
Before launching into a more detailed analysis of common feedback loops and my proposed "better" one, I need to make my criteria explicit for what I consider "good" feedback and a good feedback delivery system. The addition of "delivery system" is necessary to provide context for the value (not just accuracy) of the feedback. The criteria are:

The first criterion is about the accuracy of the feedback which is critical; the rest are about how that feedback needs to be delivered if it is going to be useful, not merely true.

Common game design feedback systems and their limitations

There are many feedback systems that designers use (or, in some cases, been subjected to). Most designers, like authors, recognize that they need feedback on their work in order to improve it-- few authors have reason to believe that their work is of publishable quality without some revision based on feedback. I'm going to list the feedback systems of which I am aware, and discuss how good of a feedback delivery system it is. There are two main categories of feedback loops: feedback from professionals in the games industry, and from non-professionals (i.e., gamers). While these sources obviously affect each other, it is easier to talk about them separately

Feedback from Professionals in the games industry
There are two main sources of this kind of feedback:

So while feedback from professionals is the current bread and butter for most teams and definitely nails criteria two, three and four, it operates a great deal on faith and hope on criterion one--that the feedback from industry professionals accurately maps onto gamers' opinions. The reason this assumption is questionable is perhaps best illuminated by a simple thought experiment--how many games do you think a typical gamer tries or sees in a year? How many do you think a gaming industry professional tries or sees? They are probably different by a factor of ten or more. Gaming industry professionals are in the top 1 percent in knowledge about games, and their tastes may simply be way more developed (and esoteric) than typical gamers' tastes. While some professionals in the industry are probably amazingly good at predicting what gamers will like, which ones are they? How many think they are great at it, when others disagree?

So while feedback from industry professionals is necessary when designing the game, they may not be the best at evaluating whether gamers will like something. In the end, they can only speak for themselves.

Feedback from Non-professionals
Game teams are not unaware of the problem of their judgment not always mapping onto what most gamers really want. Because of this, they often try to get feedback from those who are more likely to give them more accurate feedback, and the obvious people to talk to are the gamers themselves. Some common ways that this is done are listed below, along with some analysis of how good a feedback system it is according to the four criteria.

This approach has potential to be useful, in that it involves listening to gamers who aren't in the industry. However, there are many pitfalls to this--It is often dubious as to how accurately the feedback represents gamers due to the situations themselves (only certain kind of people post messages, people feel pressured to say positive things, the people running the test often lack sufficient training in how to avoid biasing the participants, etc.), and the relatively small number of people. How to minimize these concerns and create a feedback system that works on all four criteria is discussed in the next section.

Designing a better feedback system

Up to this point, I've mostly been criticizing what is done. Now I need to show that I have a better solution. I'm going to outline some of the key factors that have allowed Microsoft to develop a feedback system that we think meets the four criteria that I set up for a "good" feedback system. We call this process of providing designers with feedback from real users on their designs "user-testing," and the people who do this job "user-testing specialists."

The importance of using principles of psychological testing. Experimental psychology has been studying how to get meaningful, representative data from people for over 70 years, and the process we use adheres to the main principles of good research. This is not to say that all psychological research is good research any more than to say that all code is good code; researchers vary in their ability to do good research the same way that not all programmers are good. But there are accepted tenets of research methodology that have been shown to yield information worth relying on, and our processes have been designed with those in mind. (For the sake of not boring you senseless, I'm not going to attempt to summarize 70 years of research on how to do research in this paper.) What I'm going to do instead is describe the day-to-day work that the user-testing group at Microsoft does for its dev teams (both first and third party).

The actual testing methods we use. The user-testing group provides three major services: usability testing, playtesting, and reviews. These services are described in detail below.

1. Usability research is typically associated with small sample observational studies. Over the course of 2-3 days, 6-9 participants come to Microsoft for individual 2-hour sessions. In a typical study, each participant spends some unstructured time exploring the game prior to attempting a set of very specific tasks. Common measures include: comments, behaviors, task times and error rates. Usability is an excellent method to discover problems that the dev team was unaware of, and to understand the thoughts and beliefs of the participant and how they affect their interaction with the game. This form of testing has been a part of the software industry for years and is a staple of the HCI (Human-Computer Interaction) field more so than psychology. However, methods used in HCI can be traced to psychological research methods and can essentially be characterized as a field of applied psychology.

2. Playtest research is typically associated with large, structured questionnaire studies that focus on the first hour of game play. The sample sizes are relatively large (25-35 people) in order to be able to compute reliable percentages. Each person gets just over 60 minutes to play the game and answer questions individually on a highly structured questionnaire. Participants rate the quality of the game and provide open-ended feedback on a wide variety of general and genre-specific questions. Playtest methods are best used to gauge participants' attitudes, preferences, and some kinds of behavior, like difficulty levels. This form of testing has a long history in psychology in the fields of attitudinal research and judgment and decision-making.

3. Reviews are just another version of feedback from a games industry professional. However, these reviews are potentially more valuable as the reviewers are user-testing specialists, who are arguably have more direct contact with real gamers playing games than other game professional. Their entire job is to watch users play games and listen to their complaints and praises. Furthermore, teams often repeat mistakes that other games have made, and thus experienced user-testing specialists can help teams avoid "known" mistakes.

The result of each of these services is a report is sent to the team which meticulously documents the problems along with recommendations on how to fix those problems. Our stance is that the development teams are the ones who decide if and how to fix the problems.

One noticeable absence in our services is "focus groups." Our belief (supported by research on focus groups) is that focus groups are excellent tools for generation (e.g., coming up with new ideas, processes, etc.), but are not very good for evaluations (e.g., whether the people like something or not). The group nature of the task interferes with getting individual opinions, which is essential for the ability to quantify the evaluations.

How this feedback system fares on the four criteria for a good feedback system. So, how does the way we do user-testing at Microsoft stack up to the four criteria? Pretty well (in my humble opinion). A recap of the criteria, and my evaluation of how we do on them is given below.

Vital statistics on the user-testing group at Microsoft

Group history: the usability portion of the user-testing group has been around in a limited fashion since Microsoft entered the games business in earnest, in 1995. Funding was at a very low level (one usability contractor and 30+ titles to support) until the Games Group began investing more heavily in 1998 with the introduction of the Playtest group. The usability and playtest group merged to form the user-testing group in 2000. The current user-testing processes have been relatively stable since 1997 (usability) and 1998 (playtest).

Current composition of user-testing group: 15 FT user-testing specialists, 3-5 contract specialists, 3 FT support staff. Almost all user-testing specialists have either two or more years of graduate training in experimental psychology, or equivalent experience in applied psychology and are gamers. All four founding members of the user-testing group are still with the group.

Amount of work: In 2001, we tested approximately 6500 participants in 235 different tests, on about 70 different games. 23 of those games were non-Microsoft products. In 2002, we expect to produce about 50 percent more than we did in 2001. From 1997 to Jan 2002, the group has produced 658 reports on 114 products (53 Microsoft, and 61 non-Microsoft products) representing the opinions of more than 15,000 hours of consumer reactions to games prior to their release.

Special thanks to Randy Pagulayan and Ramon Romero for their help editing this article.