Asking the right questions - Moderating a great usability playtest

A usability playtest can be a powerful tool that — when employed by a team of experienced user researchers — developers can leverage in order to gain a lens on how players actually experience the game. This can be used to illuminate a wide array of facets of the player experience, from usability issues that leave players confused and frustrated, to deviations from the design intent whereby players don’t use a feature in the the way that the designers expected.

For those less familiar with the methodology, a usability playtest typically involves having a small group of players, independently playing through a game, while being carefully observed by researchers. The play session is typically followed by a retrospective interview, asking players about their understanding and experiences of the game. The aim is to identify elements of the game that the player struggles to understand, and help answer specific developer questions such as ‘Is the tutorial effective in teaching players the games combat mechanics?’, and ‘Are players aware of the stamina bar when playing?’.

Some of the playtesting facilities present at Player Research in Brighton. The usability playtesting room is depicted on the left.

Despite the immense value that iterative usability playtesting can offer to the games developers, running your own playtest isn’t without significant pitfalls. Gathering good data from your players can be very challenging, and there are various attributes that can have a significant impact on the quality of the data generated.

With this article I want to place particular focus on the player, and how their experiences with the researchers and the playtest environment can shape the quality of the data generated. Covering everything from what the playtesting room should look like, to the types of questions the researcher should employ to delve into the players experience with the game.

The Setting

In order to generate data that’s representative of how the player is likely to experience the game at home, it’s important that the playtest feels as organic as possible.

Our Studio Doesn’t Make the Game

In order to encourage the most natural, unbiased data, you want the play experience to be as similar as possible to how players would play the game at home. However, it’s common that big game developers cover their offices in merchandise and branding relating to the games they’ve produced. When coming into a playtest, players are likely to see this as a firm reminder that they’re playing this special, unreleased game in a very important game developer building.

Where possible, when designing your playtest lab, avoid any aesthetic features that might make the player feel as though they’re playing in a developers office. Don’t cover the walls with your studios logo, avoid filling the lobby with the games your studio has developed.

Furthermore, if members of the development team are present in the building, they should be hidden from the players — developers shouldn’t wear any t-shirts or branding that tie them to the games developer or publisher. Any conversations regarding the games development should be held out of earshot of the player.

I’m Not the Games Developer

It’s important that the player understands that they can openly criticise the game, without any consequence. Players that think they’re speaking to the games developer might think that the developer has something specific that they want to hear. They might see it as an opportunity to gush over their favourite game franchise, or think that they’ll be punished for saying something negative. Inversely, someone could also take it as an opportunity to behave negatively towards a developer who they’ve decided they dislike, holding them responsible for features they didn’t like in previous games.

With this in mind, it’s important that the player doesn’t think the interviewer is developing the game. Stress to the player that ‘We didn’t make the game, and it doesn’t matter to us whether you love it, or hate it. We just want you to be honest about your experience’.

The Playtest Shouldn’t Look Like a Lab

The playtest should feel as close to the players natural play environment as possible. That means that the room shouldn’t look like a show home, or a laboratory. What does the average living room in your country look like? Consider adding details that might make the room feel more comfortable, add realistic looking furniture to the room, a bookshelf with books, toys, etc.

Avoid Making Players Conscious That They’re Being Observed

Unless they’re a twitch streamer, the player isn’t going to be familiar with the idea of being recorded or observed and this may make the play environment feel unnatural. While it’s essential that players are made aware that they’re being recorded when consenting to the playtest, drawing attention to this fact only emphasises the unnatural playtest environment. This means not showing them behind the scenes. The player should never see the observation room, and where possible, the researchers should conceal all video/audio recording equipment (both software and hardware).

If a technical issue with this equipment arises, simply politely tell the player that there’s something you need to check and request that they wait outside the room. If the player is speaking too quietly for the recording, remedy the technical underpinning rather than asking the player to adjust their behaviour. If you want to ask the player about something you observed, simply approach it naturally (‘Could you tell me about the combat in the game?’) and avoid saying something that indicates you were watching (e.g. ‘I saw you were having difficulty with the sword, can you tell me about that?’)

Cameras Vs One-Way Mirrors

There’s much debate on the topic of how players should be observed. In my experience as a researcher — using two way mirrors within academic research, and camera observed rooms as a games user researcher — cameras place less emphasis on the fact that players are being observed in the moment. While all players are made aware that the session is being observed when consenting to take part in the research, there’s no benefit to making players overly conscious of this fact during the playtest.

One way mirrors leave the player wondering what might be happening on the other side of the mirror and draw persistent attention to the fact that the room is not like the rooms that the player normally plays their games in. Having one wall occupied by a mirror, gives the room a distinct, lab environment feel that players are likely to be totally unfamiliar with. Cameras on the other hand, can be discretely placed throughout the room without drawing persistent attention.

Building Rapport

As well as the unfamiliar setting, the interview process involves the player speaking to a complete stranger. If the player doesn’t feel comfortable speaking to the researcher, then it’s going to be difficult to get them to talk openly about their experiences with the game.

A Single Point of Contact

The moment that the player enters your building, they’re in an unfamiliar and potentially intimidating environment. It only contributes to this if the player also needs to learn who different people in your company are. As a result, it can be a more comfortable playtesting experience if the player has a single point of contact at your studio — a single researcher who greets them at the door, interviews them, and eventually debriefs them when they’re about to leave.

Perhaps most importantly, by providing this single point of contact you allow the opportunity for the researcher to build rapport with the player. If the first time the player sees the researcher is at the start of the interview, it’s very likely they won’t feel immediately comfortable speaking with them.

Use the Pre-interview to Build Rapport

It’s important that the playtester feels comfortable speaking to the researcher, ideally at the start of the interview. When the participant first arrives in the playtest area, you’ll want to greet them and thank them for coming in for the playtest, as well as getting them to sign a consent form, and NDA. This first interaction may be vital in communicating that you’re friendly and easy to talk to, so that when they enter the interview proper the player will be more comfortable speaking openly about their experience with you. Ask them about their the travel in to the studio, was it easy to find? Have they been in for a playtest before? If not, let them know not to worry. Make them aware of what you’ll be doing with them today, and that they can ask you if they have any questions.

At the start of the testing session, get players to talk about the games they’ve been playing recently. It’s good to have this on record and confirm the data that players have provided to you via online surveys/telephone, and it also helps them get into the flow of talking to you about their experiences with games.

Using Open Posture to Help Players Feel More Comfortable

Aggressive or closed posture (e.g. arms crossed, slouched back) can intimidate the player, or make them feel uncomfortable speaking to the researcher. Using open posture (e.g. hands in front of you, sat upright) can help make the interviewee feel comfortable, as well as giving the impression that you’re listening, and value their perspective.

You should also ensure that you’re sat at the same level as the interviewee, as a higher position might make them feel as if they’re of lower status to you, which may be intimidating. Additionally, ensure that there are no physical barriers between the two of you (e.g. a chair or your laptop), as this is likely to make dialogue feel awkward and potentially uncomfortable.

Using Non-Verbal Cues to Appear Attentive and Encourage Dialog

When interviewing the player it’s important that they get the impression that you’re both attending the conversation, and interested in what they have to say. However, you also don’t want to behave in any way that may seem as though you’re confirming what the player is saying. For instance, audibly saying things like ‘Yes’, ‘Mhm’, ‘Uh-huh’ may influence the players responses, and cause them to think that the researcher is agreeing with what they’re saying. Instead, you can audibly sound interested in what the player is saying by adjusting your intonation and tone when speaking. You can come across as attentive using non-verbal communication, such presenting eye contact, and gentle nodding as the player responds.

Adjusting the Perceived Status of the Researcher and Player to Make Conversation More Comfortable

Lowering the status of the researcher (so that we don’t seem hyper important, but approachable) can be a helpful technique used to make dialogue with the researcher more approachable and comfortable. Avoid non-verbal cues that might give the interviewee the impression you are of higher status (e.g. sitting with arms back, legs crossed), avoid any language that may impress your status on the participant (e.g. words like ‘researcher’ and referring to the player as a ‘participant’).

Additionally if a player seems reluctant to provide insight on their experience, you can consider raising their perceived status so that they feel their perspective is valued. Tell the player that you appreciate their views on the game. Give the interview signals that you’re listening and attentive to them (e.g. nodding, eye contact).

Playing the Game

The point in the playtest where the player actually players the game is of course, vital to understanding how the players experience, and there are a number of factors that may affect the quality of observational data generated during this time.

Leave the Player to Play the Game Alone

Wherever possible, leave the player to play the game on their own. Having a researcher peering over their shoulder and observing everything they do is likely to feel like a very unnatural experience, and it’s likely that this may change the players behaviour when playing the game. For instance, they might think that they should impress you by playing well, or make an explicit effort to show you something they think might be interesting, deviating from their natural play habits. Excluding exceptional circumstances, playtests should be observed remotely so that there’s less emphasis on the fact that they’re being watched.

Notably, it may be tempting to be present in the room to ask the player what they’re thinking, and what they’re trying to do, but these types of questions are likely to affect how the player thinks about the game. For instance, a simple question like ‘How does this work?’ might cause the player to ponder the capacity and limitations of a mechanic in ways that they wouldn’t naturally when playing the game at home.

However, there are exceptions to this rule. In some instances there can be value to more guided, think-aloud style sessions, and when playtesting with children it’s often necessary that a researcher remain in the room so as to supervise the session — both of these topics require more discussion than the scope of this article affords.

Avoid Abruptly Interrupting the Player

If the player were playing the game at home, they would be in a comfortable, safe environment, such as their bedroom, or the lounge. Crucially, the player won’t be suddenly interrupted at any point in time, it’s their space and it’s not likely a stranger is going to burst in and interrupt them. During a playtest, it can be make the player feel comfortable if they feel that they’re in control of their space. Instead of bursting into the room, knock and give the player an opportunity to answer. If they don’t answer (due to headphones, or another reason), then the knock at least gives them opportunity to be aware that someone is about to enter the room.

Of course, this also means that no one else in your studio should be disturbing the player during the session. Ensure that everyone else in your organisation is aware when a playtest is happening, and that people don’t simply swing the door open to take a look inside while the playtesting room is in use — a sign on the door that makes this clear can be very effective.

The Interview

Post-gameplay interview is where the researcher aims to determine the players understanding of the game. While some issues present themselves very visibly during gameplay observation, it’s often difficult to determine the root cause. Did the player not use the energy system because they didn’t understand it, or was it that they understood it, and deliberately opted not to use it?

Use an Interview Guide

It may sound obvious, but I can’t stress the value in having a fully prepared interview script which details both the questions you want to ask players, and potential probing questions that you might want to ask should the player not quite provide enough detail. For instance, the script might look a little like this.

Don’t Stick to Your Interview Guide

Despite the value of an interview guide, you need to be able to go off-script when probing into specific issues raised by the player. That means you willencounter things that you haven’t prepared potential follow up questions for, and you need to be prepared to ask further questions in order to encourage the player to provide more detail. In these instances it’s useful to have various types of question in mind.

It’s also important to allow the player to speak about the game out of sequence with the interview guide. The interview guide should only act as a checklist for things to cover, not a strict order that the conversation needs to adhere to. If the player jumps to something you had anticipated coming later in the conversation, simply adapt and allow the conversation to flow naturally, then revisit the other areas you wanted to discuss later.

Use Short and Concise Questioning

It’s important not to overwhelm the player with complex, or double barrelled questions. For instance avoid asking things like ‘Did you see the energy system? How did it work?’. Not only is this a poorly worded, leading question, it’s double barrelled and the player is likely to only answer part of what was asked. Ask questions one at a time, and ensure that the questions asked are clear and concise.

Don’t Let Players Project

It’s common that players will project their thoughts and feelings onto another, hypothetical player. Players might say something like, ‘I imagine someone younger might find it confusing…’. In these instances, it’s often the case that the player finds this particular mechanic confusing, but feels that expressing this is embarrassing, or uncomfortable.

As the researcher, it’s important to realign the interview to target the players own experiences so that you can determine how the player you’re interviewing experienced this feature. Try, ‘How was it for you?’.

Use Open Questions to Encourage Detail

Encourage the player to provide detail by asking open ended, rather than closed questions. For instance, rather than ‘Did you understand the energy mechanic?’, ask players ‘How did the ‘lightening bolt’ thing that you mentioned work?’. By using non-leading, open questions you can get valuable detail from participants that you wouldn’t otherwise attain.

Avoid Complex Terminology

Working within the game development industry likely means that the user researchers have picked up a wide variety of terms that might be used to refer to game elements. A tutorial might be referred to as a first time user experience, and words like ‘mechanic’ or ‘system’ might be used in reference to features present in the game. Players aren’t game developers, and it’s likely that they’ll be confused by these terms, repeated confusion and requests for clarification might make the interview feel stressful, or alternatively, the player may simply misinterpret the question and provide an equally confusing answer.

For example, ‘What did you think of the combat mechanics?’ could easily be more clearly worded as ‘What did you think of the combat?’

Speak the Players Language

When the player refers to the ‘energy system’ as the ‘lightening bolts’ it may be the case that they do not have a strong understanding of this mechanic. In this instance, the label ‘energy’ has an inferred function that the player may not be aware of, which may inform their understanding, or simply confuse the player. Using the terms that the player provides avoids giving anything away about the game, ensuring you can assess their understanding without influencing it.

Repeat the Players Answers

Sometimes the player might give an answer that’s complex, and inherently difficult to follow. In these instances it can be valuable clarify both the players’ and your own understanding by repeating the answer, back to the player.

It’s important that this repetition of the players responses sounds natural in conversation. It shouldn’t sound as if you’re repeating them word for word, as this is likely to make the player feel awkward and emphasise the fact that the session is being recorded.

Probe with Non-Leading Questions

In many instances the player will reveal something that indicates they don’t understand a particular element of the game. However, the player won’t provide enough detail for the researcher to fully understand what it is they’re missing, or what’s causing their confusion. This is an opportunity to follow up for more detail. Use non-leading, open questions which allow the player to explain what the scenario.

Employ a Mid-Session Interview for Longer Gameplay Sessions

In some instances, the playtest will involve having players play the game for long periods of time — beyond the 45minute mark in a single sitting. This can be problematic for the post-session interview, which relies on the players memory to recall their understanding of what happened, and what they were thinking earlier in the play session.

This is where it can be valuable to break the session into smaller chunks, both so that the player can get a break from the game, and so that the researchers can gain insight on the players experience.

Preparing mini-interviews that the researcher can employ throughout the session can be an effective way of ensuring that the interview process places less emphasis on the players ability to recall. Additionally, these shorter interviews can be used to examine the players experience of very specific elements of the game — for instance, a brief interview that follows the end of the tutorial segment can be effective in determining what the player understands, having just left the tutorial.

Think-aloud protocol — where the participant is encouraged to actively verbalise their cognitive processes while playing — is also something the researchers might consider. However, this has numerous various pros and cons (see further reading for more detail).

Understanding the Value of Silences

Periods of silence may appear awkward during the interview process, but in reality these moments can be incredibly valuable. These periods of silence give the player an opportunity to speak about anything related that they may have on their mind, that you have not directly asked about. It’s often the case that the player will speak about some misunderstanding they may never have touched upon if you had filled that silence with another question. Where you think the player may add detail, create brief pauses before following up.

Ensure That Players Feel Able to Criticise the Game

Sometimes players might be concerned about criticising a game because the game looks finished. For instance, why talk about fundamental design issues when they think it’s clear that the game is just about to launch? It can be helpful to make the player think that the game is in a more maleable state, so that their thoughts on the game are worthwhile.

Despite this, this does emphasise the fact that the player isn’t playing a normal video game, and that this playtest process is actually part of the games development process. Therefore, while it can be effective in encouraging players to provide more open criticism of their experience, it may be important to apply this technique selectively, and only with players that appear to present difficulty providing this type of criticism during the interview.

The Interviewing Researcher Shouldn’t Take Notes

Taking notes during the interview is essential for capturing the players self-reported experiences of the game. However, taking detailed notes is a demanding task, and as the interviewer, will result in it appearing as though you’re not engaged with the player. This is likely to emphasise the fact that the interview is being recorded, and disrupt the flow of dialogue between the interviewer and player.

Detailed notes should be taken by a second researcher, from an observation room. This way, the playtest can record highly detailed data on the players self-reported experience, without sacrificing the quality of the interview.

Providing Opportunity to Add Detail

It’s important that you provide persistent opportunity for the player to add detail. You can achieve this by simply asking the player every now and again if there was anything they’d like to talk about.

Start Broad, then Narrow In

Starting with broad questions such as ‘Could you tell me what you had to do in the game?’ and ‘Could you tell me about the gameplay?’ can be great for allowing the player to naturally speak about the game without being probed and re-directed. However, in some instances you’ll find yourself wanting to understand if they were using/aware of a particular system (in this example let’s say, a stamina bar), and this doesn’t come up in their responses. In these situations, it’s not clear if the player didn’t mention it because they weren’t aware of this mechanic, or they had some other reason for not mentioning it — such as simply forgetting about it, or not considering it important to mention. This is where you need to narrow down your questioning in order to make it clearer what you want from the player.

Here’s an example of how to narrow down into the players understanding of a particular mechanic. In this instance while the player is providing a clear account of the games core systems, the researcher is trying to narrow down whether they were aware of the stamina system.

If the player still doesn’t talk about the stamina mechanic, then it’s likely that they weren’t aware of the bar. At this point you could show them the screen and ask what it was, or refer to it directly. Players may say that they weren’t really sure, or indicate that they were aware of it, but just forgot to mention it when asked.

Be Aware That Players Don’t Know When They’re Confused

It’s common that you’ll observe players struggling with a particular feature, or mechanic, but when you ask players if they experienced any confusion when playing the game, they’ll tell you that everything was okay. You’ve just watched them struggle to drive a car in a straight line for 45 minutes, getting stuck in ways that you didn’t think imaginable, and yet the player will tell you that everything was fine.

It can get difficult to understand why this might be at first, but it often comes down to the player not recognising this as the type of thing you want them to talk about. The vehicle controls might have been difficult, but in the players eyes, nothing about them was confusing, and they blame their own ability for not being able to drive effectively.

In many instances, you can get more detail from the player on these issues by re-framing the question. While players may not consider what you observed as something that confused them, they may provide a different answer if you ask them ‘Was there anything in the game that you found quite difficult?’ or ‘Was there anything that you thought took quite a long time to do?’.

Often, the player will provide detail on their experiences which indicates a clear deviation from the games design intent. For instance:

Shut Up and Listen

This relates back to many of the points I’ve already raised, but I’ve heard if something is important, it’s worth repeating. While your questioning is present to guide the interview, it’s important that you allow the player to speak openly about the game, in their own words, without interruption or perceived pressure. That often means, even if the player is speaking about something you’re not interested in (e.g. when it doesn’t directly relate to a usability issue), simply letting them speak and waiting before redirecting the conversation.

Listening to the player communicates that you’re interested in what their saying, respect their thoughts, and value their perspective. It’s also important to remember that silences are among the most valuable tools in your arsenal as they allow the player opportunity to speak freely about the game in ways that might not have been encouraged by your interview guide.

Additional Reading & References