Test your damn game: choosing participants

It is important to test your game with representative users. Talk to them, watch them play your game. You need their independent feedback, after all you are too close to your game.

Choosing the right participants is key to success

It is important to get the right participants for your user tests. You will be using the test feedback to change or improve your game. You should try to make sure the participants are the right audience for your game so you don’t waste time or money.

Would you trust anyones feedback on your game? It is important to match the feedback with the type of person who gave the feedback. For example: Who said the combat is boring? Is it someone who isn’t a fan of your game’s genre? Is it a super fan who has spent $1000 in a competitor’s title?

You might think that anyone is appropriate for UX or FTE testing because you expect the game will be intuitive enough for everyone. For most tests you want a specific audience.

Principles to identify your audience and your user testing participants

Quick! What is the target audience for your game? If you answered, “Males 15-25,” then you lose!

A demographic is not your audience. If this is how you select your participants then you are unlikely to get useful feedback. Simple demographic information is insufficient to know if a person will be a fan of your game. Defining and categorizing participants is key to getting trusted feedback from user testing.

A person’s interests and past behavior is a much better indicator if they will like your game. So, we can define an audience through two major factors: 1. interest, 2. behavior. An interest means the person likes this type of game (genre, theme, platform, etc).  Behavior means their experience or level of familiarity with this type of game. 

4 player categories based on their behavior

Here are a few ways of categorizing players based on their behavior and what types of user tests are appropriate for each category.

1) Average User

2) Power User

3) Novice User

4) Influencer, Maven

3 practical tips for creating participant requirements

Let’s review some practical tips for screening your participants based on their interests and behaviors.

1) Interests - focus on comparable titles

Focus on what comparable titles the participant as played. A comparable title is a game like yours that participants might have played. When someone has played a comparable title, it is a strong indicator that they like that type of game.

What makes a good comparable title?

2) Player Behavior - what did they do in the comp title?

When screening for player behavior you should focus on what their play-habits were with the comparable title. You may want to know how far they advanced, how much money they spent, how often they played, how competitive or social they were.

What did they do in the comparable title?

3) Don’t be too restrictive

Avoid being too restrictive with your requirements or you won’t have enough participants. Avoid requiring having played too many comparable titles.

Participant screening process

Before we run a user test with potential participants we need to determine their suitability for our test. We do this by asking them questions to see if they meet our requirements.

Overview of participant screening process

Participant Requirements

As discussed previously, the participant requirements are: the type of player you want for your test, comparable titles or interests, details of player behavior to identify player type.

Screening Survey

Use our requirements to create a screening survey. Be sure requirements are concrete. For example, know what level the player should have reached in each comparable titles.

Follow-up Screening

Participants who pass the first screening survey are asked follow-up questions. This additional screening step is to verify the participants are members of the target audience. You should verify they are not lying or misrepresenting their experience. Follow-up screening should double check key answers from their screening survey. It is ok if the answers are only roughly similar. Be sure to ask another question to show the participant has knowledge of the games they said they played. Failure to perform follow-up screening may result in having some of your participants being “duds”.

Sourcing Users

After you are prepared identifying requirements, creating screening survey, and follow up questions, then you need a way of driving traffic to your screening survey. Sourcing users will be covered in a future topic.

Example screening requirements

Let’s run through the process with an example RPG game. For this example we will focus on one comparable title, Marvel: Contest of Champions. This title was picked because of its similarity to our example RPG game: 1. similar reaction-based combat, 2. similar meta game of collecting and upgrading heroes. We want to run the user test with average users and power users. 

Our chosen follow up question is about the meta game. “What is your top champion's prestige?” The players in Marvel: Contest of Champions spend most of their time upgrading heroes, it is important. All non-novice players should be able to answer.

All of the following target level and champion prestige is appropriate at the time of writing in mid 2017.

Average user screening

We want users who have played about 3 months. We have chosen the main screening criteria of level. This creates a target level of 30-40. For follow-up screening, our expectations for an average users’ top champion prestige is 2500+, or 3500+ for paying users.

Power user screening

For power users, we want people who have reached near-end game. We have chosen a target level of 55+. As of writing, the current level cap is 60 and takes about a year+ to reach. For follow-up screening, our expectation is a top champion prestige of 3500+.

Lessons learned : screening questions that didn’t work for me

Learn from my mistakes! Here are a few lessons I learned.

Focus on genres instead of comparable titles

In my early screening surveys, I asked players to select genres that they played. This was instead of asking for a direct comparable title. Asking genres did not work well. The genres had different meanings and definitions to different people. The lack of clear and enforced app-store categorization did not help the issue either, with puzzle games being categorized as RPG games.

In further screening survey iterations I gave example games to help clarify the genres. The added examples didn’t seem to help much. 

Games you are currently playing

I had always asked what games the participants were currently playing. This always resulted in a wide range of responses, from one of many games someone was currently playing, to a laundry list of games the person had heard of in the past few years. So this question did not always help with participant selection.

Not clarifying play habits

It wasn’t until later iterations of the screening survey that I started to feel more in control of the survey participants. These surveys directly asked about comparable titles and follow up questions clarifying the participants play habits in the comparable titles.  The clarifying questions about the play habits was important. They yield much better results than a yes/no or a last played date.

Lack of follow-up

A lack of rigorous follow-up questions was also an early problem that resulted in having participants who clearly were unfamiliar with the game genre we were testing.

Conclusion

When conducting user tests we need to have the right participants so we can trust their feedback. There are two main factors to focus on when screening potential participants. The first factor is the participants interests. Are they interested in games similar to yours? Have they played games that are comparable to yours? The second factor is the person’s behavior playing comparable titles. Does their level of familiarity with the comparable title match the type of user test you will run?

When screening potential participants you will have them take a survey to collect enough information to determine if they are a good fit for your user test. You will ask follow up questions to double check the responses for participants you are interested in.

I hope these guidelines will make your next user test more productive. Get out there and test your damn game!