Asking the Right Questions - Moderating a Great Usability Playtest
A usability playtest can be a powerful tool that leveraged to gain a lens on how players actually experience your game. This article includes various tips to help ensure your own usability playest generates high quality, actionable data.
![](https://eu-images.contentstack.com/v3/assets/blt740a130ae3c5d529/bltba62518415cda0e2/652fe6ddbc479f8697ef691f/default-cubic.png?width=1280&auto=webp&quality=95&format=jpg&disable=upscale)
Asking the right questions - Moderating a great usability playtest
A usability playtest can be a powerful tool thatââŹĹ âââŹĹ when employed by a team of experienced user researchersââŹĹ âââŹĹ developers can leverage in order to gain a lens on how players actually experience the game. This can be used to illuminate a wide array of facets of the player experience, from usability issues that leave players confused and frustrated, to deviations from the design intent whereby players donât use a feature in the the way that the designers expected.
For those less familiar with the methodology, a usability playtest typically involves having a small group of players, independently playing through a game, while being carefully observed by researchers. The play session is typically followed by a retrospective interview, asking players about their understanding and experiences of the game. The aim is to identify elements of the game that the player struggles to understand, and help answer specific developer questions such as âIs the tutorial effective in teaching players the games combat mechanics?â, and âAre players aware of the stamina bar when playing?â.
![](https://cdn-images-1.medium.com/max/1600/0*HNMHdq9kgfoGJ6ri.png?width=700&auto=webp&quality=80&disable=upscale)
Some of the playtesting facilities present at Player Research in Brighton. The usability playtesting room is depicted on the left.
Despite the immense value that iterative usability playtesting can offer to the games developers, running your own playtest isnât without significant pitfalls. Gathering good data from your players can be very challenging, and there are various attributes that can have a significant impact on the quality of the data generated.
With this article I want to place particular focus on the player, and how their experiences with the researchers and the playtest environment can shape the quality of the data generated. Covering everything from what the playtesting room should look like, to the types of questions the researcher should employ to delve into the players experience with the game.
The Setting
In order to generate data thatâs representative of how the player is likely to experience the game at home, itâs important that the playtest feels as organic as possible.
Our Studio Doesnât Make the Game
In order to encourage the most natural, unbiased data, you want the play experience to be as similar as possible to how players would play the game at home. However, itâs common that big game developers cover their offices in merchandise and branding relating to the games theyâve produced. When coming into a playtest, players are likely to see this as a firm reminder that theyâre playing this special, unreleased game in a very important game developer building.
Covering the environment in merchandise and branding is likely to act a firm reminder that theyâre playing this special, unreleased game in a very important game developer building.
Where possible, when designing your playtest lab, avoid any aesthetic features that might make the player feel as though theyâre playing in a developers office. Donât cover the walls with your studios logo, avoid filling the lobby with the games your studio has developed.
Furthermore, if members of the development team are present in the building, they should be hidden from the playersââŹĹ âââŹĹ developers shouldnât wear any t-shirts or branding that tie them to the games developer or publisher. Any conversations regarding the games development should be held out of earshot of the player.
Iâm Not the Games Developer
Itâs important that the player understands that they can openly criticise the game, without any consequence. Players that think theyâre speaking to the games developer might think that the developer has something specific that they want to hear. They might see it as an opportunity to gush over their favourite game franchise, or think that theyâll be punished for saying something negative. Inversely, someone could also take it as an opportunity to behave negatively towards a developer who theyâve decided they dislike, holding them responsible for features they didnât like in previous games.
With this in mind, itâs important that the player doesnât think the interviewer is developing the game. Stress to the player that âWe didnât make the game, and it doesnât matter to us whether you love it, or hate it. We just want you to be honest about your experienceâ.
We didnât make the game, and it doesnât matter to us whether you love it, or hate it. We just want you to be honest about your experience.
The Playtest Shouldnât Look Like a Lab
The playtest should feel as close to the players natural play environment as possible. That means that the room shouldnât look like a show home, or a laboratory. What does the average living room in your country look like? Consider adding details that might make the room feel more comfortable, add realistic looking furniture to the room, a bookshelf with books, toys, etc.
Avoid Making Players Conscious That Theyâre Being Observed
Unless theyâre a twitch streamer, the player isnât going to be familiar with the idea of being recorded or observed and this may make the play environment feel unnatural. While itâs essential that players are made aware that theyâre being recorded when consenting to the playtest, drawing attention to this fact only emphasises the unnatural playtest environment. This means not showing them behind the scenes. The player should never see the observation room, and where possible, the researchers should conceal all video/audio recording equipment (both software and hardware).
While itâs essential that players are made aware that theyâre being recorded when consenting to the playtest, drawing attention to this fact only emphasises the unnatural playtest environment.
If a technical issue with this equipment arises, simply politely tell the player that thereâs something you need to check and request that they wait outside the room. If the player is speaking too quietly for the recording, remedy the technical underpinning rather than asking the player to adjust their behaviour. If you want to ask the player about something you observed, simply approach it naturally (âCould you tell me about the combat in the game?â) and avoid saying something that indicates you were watching (e.g. âI saw you were having difficulty with the sword, can you tell me about that?â)
Cameras Vs One-Way Mirrors
Thereâs much debate on the topic of how players should be observed. In my experience as a researcherââŹĹ âââŹĹ using two way mirrors within academic research, and camera observed rooms as a games user researcherââŹĹ âââŹĹ cameras place less emphasis on the fact that players are being observed in the moment. While all players are made aware that the session is being observed when consenting to take part in the research, thereâs no benefit to making players overly conscious of this fact during the playtest.
One way mirrors leave the player wondering what might be happening on the other side of the mirror and draw persistent attention to the fact that the room is not like the rooms that the player normally plays their games in. Having one wall occupied by a mirror, gives the room a distinct, lab environment feel that players are likely to be totally unfamiliar with. Cameras on the other hand, can be discretely placed throughout the room without drawing persistent attention.
One way mirrors leave the player wondering what might be happening on the other side of the mirror and draw persistent attention to the fact that the room is not like the rooms that the player normally plays their games in.
Building Rapport
As well as the unfamiliar setting, the interview process involves the player speaking to a complete stranger. If the player doesnât feel comfortable speaking to the researcher, then itâs going to be difficult to get them to talk openly about their experiences with the game.
A Single Point of Contact
The moment that the player enters your building, theyâre in an unfamiliar and potentially intimidating environment. It only contributes to this if the player also needs to learn who different people in your company are. As a result, it can be a more comfortable playtesting experience if the player has a single point of contact at your studioââŹĹ âââŹĹ a single researcher who greets them at the door, interviews them, and eventually debriefs them when theyâre about to leave.
Perhaps most importantly, by providing this single point of contact you allow the opportunity for the researcher to build rapport with the player. If the first time the player sees the researcher is at the start of the interview, itâs very likely they wonât feel immediately comfortable speaking with them.
Use the Pre-interview to Build Rapport
Itâs important that the playtester feels comfortable speaking to the researcher, ideally at the start of the interview. When the participant first arrives in the playtest area, youâll want to greet them and thank them for coming in for the playtest, as well as getting them to sign a consent form, and NDA. This first interaction may be vital in communicating that youâre friendly and easy to talk to, so that when they enter the interview proper the player will be more comfortable speaking openly about their experience with you. Ask them about their the travel in to the studio, was it easy to find? Have they been in for a playtest before? If not, let them know not to worry. Make them aware of what youâll be doing with them today, and that they can ask you if they have any questions.
The first interaction may be vital in communicating that youâre friendly and easy to talk to.
At the start of the testing session, get players to talk about the games theyâve been playing recently. Itâs good to have this on record and confirm the data that players have provided to you via online surveys/telephone, and it also helps them get into the flow of talking to you about their experiences with games.
Using Open Posture to Help Players Feel More Comfortable
Aggressive or closed posture (e.g. arms crossed, slouched back) can intimidate the player, or make them feel uncomfortable speaking to the researcher. Using open posture (e.g. hands in front of you, sat upright) can help make the interviewee feel comfortable, as well as giving the impression that youâre listening, and value their perspective.
You should also ensure that youâre sat at the same level as the interviewee, as a higher position might make them feel as if theyâre of lower status to you, which may be intimidating. Additionally, ensure that there are no physical barriers between the two of you (e.g. a chair or your laptop), as this is likely to make dialogue feel awkward and potentially uncomfortable.
Using Non-Verbal Cues to Appear Attentive and Encourage Dialog
When interviewing the player itâs important that they get the impression that youâre both attending the conversation, and interested in what they have to say. However, you also donât want to behave in any way that may seem as though youâre confirming what the player is saying. For instance, audibly saying things like âYesâ, âMhmâ, âUh-huhâ may influence the players responses, and cause them to think that the researcher is agreeing with what theyâre saying. Instead, you can audibly sound interested in what the player is saying by adjusting your intonation and tone when speaking. You can come across as attentive using non-verbal communication, such presenting eye contact, and gentle nodding as the player responds.
Adjusting the Perceived Status of the Researcher and Player to Make Conversation More Comfortable
Lowering the status of the researcher (so that we donât seem hyper important, but approachable) can be a helpful technique used to make dialogue with the researcher more approachable and comfortable. Avoid non-verbal cues that might give the interviewee the impression you are of higher status (e.g. sitting with arms back, legs crossed), avoid any language that may impress your status on the participant (e.g. words like âresearcherâ and referring to the player as a âparticipantâ).
Additionally if a player seems reluctant to provide insight on their experience, you can consider raising their perceived status so that they feel their perspective is valued. Tell the player that you appreciate their views on the game. Give the interview signals that youâre listening and attentive to them (e.g. nodding, eye contact).
Playing the Game
The point in the playtest where the player actually players the game is of course, vital to understanding how the players experience, and there are a number of factors that may affect the quality of observational data generated during this time.
Leave the Player to Play the Game Alone
Wherever possible, leave the player to play the game on their own. Having a researcher peering over their shoulder and observing everything they do is likely to feel like a very unnatural experience, and itâs likely that this may change the players behaviour when playing the game. For instance, they might think that they should impress you by playing well, or make an explicit effort to show you something they think might be interesting, deviating from their natural play habits. Excluding exceptional circumstances, playtests should be observed remotely so that thereâs less emphasis on the fact that theyâre being watched.
Notably, it may be tempting to be present in the room to ask the player what theyâre thinking, and what theyâre trying to do, but these types of questions are likely to affect how the player thinks about the game. For instance, a simple question like âHow does this work?â might cause the player to ponder the capacity and limitations of a mechanic in ways that they wouldnât naturally when playing the game at home.
It may be tempting to be present in the room to ask the player what theyâre thinking, and what theyâre trying to do, but these types of questions are likely to affect how the player thinks about the game.
However, there are exceptions to this rule. In some instances there can be value to more guided, think-aloud style sessions, and when playtesting with children itâs often necessary that a researcher remain in the room so as to supervise the sessionââŹĹ âââŹĹ both of these topics require more discussion than the scope of this article affords.
Avoid Abruptly Interrupting the Player
If the player were playing the game at home, they would be in a comfortable, safe environment, such as their bedroom, or the lounge. Crucially, the player wonât be suddenly interrupted at any point in time, itâs their space and itâs not likely a stranger is going to burst in and interrupt them. During a playtest, it can be make the player feel comfortable if they feel that theyâre in control of their space. Instead of bursting into the room, knock and give the player an opportunity to answer. If they donât answer (due to headphones, or another reason), then the knock at least gives them opportunity to be aware that someone is about to enter the room.
During a playtest, it can be make the player feel comfortable if they feel that theyâre in control of their space. Instead of bursting into the room, knock and give the player an opportunity to answer.
Of course, this also means that no one else in your studio should be disturbing the player during the session. Ensure that everyone else in your organisation is aware when a playtest is happening, and that people donât simply swing the door open to take a look inside while the playtesting room is in useââŹĹ âââŹĹ a sign on the door that makes this clear can be very effective.
The Interview
Post-gameplay interview is where the researcher aims to determine the players understanding of the game. While some issues present themselves very visibly during gameplay observation, itâs often difficult to determine the root cause. Did the player not use the energy system because they didnât understand it, or was it that they understood it, and deliberately opted not to use it?
Use an Interview Guide
It may sound obvious, but I canât stress the value in having a fully prepared interview script which details both the questions you want to ask players, and potential probing questions that you might want to ask should the player not quite provide enough detail. For instance, the script might look a little like this.
Researcher: Could you tell me what you needed to do in the game?Â
Researcher: Was there any combat?Â
Researcher: How did this work?
Researcher: What options did you have in combat?Â
Researcher: How did these work?
Researcher: Was there anything you found confusing?Â
Researcher: Was there anything that you found quite difficult?
Donât Stick to Your Interview Guide
Despite the value of an interview guide, you need to be able to go off-script when probing into specific issues raised by the player. That means you willencounter things that you havenât prepared potential follow up questions for, and you need to be prepared to ask further questions in order to encourage the player to provide more detail. In these instances itâs useful to have various types of question in mind.
Researcher: Could you tell me a little bit more about that?
Researcher: You said x, what did you mean by that?
Itâs also important to allow the player to speak about the game out of sequence with the interview guide. The interview guide should only act as a checklist for things to cover, not a strict order that the conversation needs to adhere to. If the player jumps to something you had anticipated coming later in the conversation, simply adapt and allow the conversation to flow naturally, then revisit the other areas you wanted to discuss later.
Use Short and Concise Questioning
Itâs important not to overwhelm the player with complex, or double barrelled questions. For instance avoid asking things like âDid you see the energy system? How did it work?â. Not only is this a poorly worded, leading question, itâs double barrelled and the player is likely to only answer part of what was asked. Ask questions one at a time, and ensure that the questions asked are clear and concise.
Donât Let Players Project
Itâs common that players will project their thoughts and feelings onto another, hypothetical player. Players might say something like, âI imagine someone younger might find it confusingâŚâ. In these instances, itâs often the case that the player finds this particular mechanic confusing, but feels that expressing this is embarrassing, or uncomfortable.
As the researcher, itâs important to realign the interview to target the players own experiences so that you can determine how the player youâre interviewing experienced this feature. Try, âHow was it for you?â.
Researcher: What did you think of the tutorial instructions in the game?Â
Player: They were fine, yeah. Though some of the language might be confusing for kids.
Researcher: But how was the language for you?Â
Player: I think⌠sometimes it could benefit from being a little more clear. It said things like âTap and Untapâ. Obviously I know what those words mean, but it could be a little confusing in the game.â
Use Open Questions to Encourage Detail
Encourage the player to provide detail by asking open ended, rather than closed questions. For instance, rather than âDid you understand the energy mechanic?â, ask players âHow did the âlightening boltâ thing that you mentioned work?â. By using non-leading, open questions you can get valuable detail from participants that you wouldnât otherwise attain.
Avoid Complex Terminology
Working within the game development industry likely means that the user researchers have picked up a wide variety of terms that might be used to refer to game elements. A tutorial might be referred to as a first time user experience, and words like âmechanicâ or âsystemâ might be used in reference to features present in the game. Players arenât game developers, and itâs likely that theyâll be confused by these terms, repeated confusion and requests for clarification might make the interview feel stressful, or alternatively, the player may simply misinterpret the question and provide an equally confusing answer.
For example, âWhat did you think of the combat mechanics?â could easily be more clearly worded as âWhat did you think of the combat?â
Speak the Players Language
When the player refers to the âenergy systemâ as the âlightening boltsâ it may be the case that they do not have a strong understanding of this mechanic. In this instance, the label âenergyâ has an inferred function that the player may not be aware of, which may inform their understanding, or simply confuse the player. Using the terms that the player provides avoids giving anything away about the game, ensuring you can assess their understanding without influencing it.
âYou mentioned the lightening bolts before, could you tell me about these?â
Repeat the Players Answers
Sometimes the player might give an answer thatâs complex, and inherently difficult to follow. In these instances it can be valuable clarify both the playersâ and your own understanding by repeating the answer, back to the player.
âSo, let me see if I understand correctly⌠The only way to find water was by sending out a robot scout?â
Itâs important that this repetition of the players responses sounds natural in conversation. It shouldnât sound as if youâre repeating them word for word, as this is likely to make the player feel awkward and emphasise the fact that the session is being recorded.
Probe with Non-Leading Questions
In many instances the player will reveal something that indicates they donât understand a particular element of the game. However, the player wonât provide enough detail for the researcher to fully understand what it is theyâre missing, or whatâs causing their confusion. This is an opportunity to follow up for more detail. Use non-leading, open questions which allow the player to explain what the scenario.
Researcher: âYou said this was confusing, what did you mean by that?â
Researcher: âWhat did you try?â
Researcher: âHow did you expect this to work?â
Employ a Mid-Session Interview for Longer Gameplay Sessions
In some instances, the playtest will involve having players play the game for long periods of timeââŹĹ âââŹĹ beyond the 45minute mark in a single sitting. This can be problematic for the post-session interview, which relies on the players memory to recall their understanding of what happened, and what they were thinking earlier in the play session.
This is where it can be valuable to break the session into smaller chunks, both so that the player can get a break from the game, and so that the researchers can gain insight on the players experience.
Preparing mini-interviews that the researcher can employ throughout the session can be an effective way of ensuring that the interview process places less emphasis on the players ability to recall. Additionally, these shorter interviews can be used to examine the players experience of very specific elements of the gameââŹĹ âââŹĹ for instance, a brief interview that follows the end of the tutorial segment can be effective in determining what the player understands, having just left the tutorial.
Preparing mini-interviews that the researcher can employ throughout the session can be an effective way of ensuring that the interview process places less emphasis on the players ability to recall.
Think-aloud protocolââŹĹ âââŹĹ where the participant is encouraged to actively verbalise their cognitive processes while playingââŹĹ âââŹĹ is also something the researchers might consider. However, this has numerous various pros and cons (see further reading for more detail).
Understanding the Value of Silences
Periods of silence may appear awkward during the interview process, but in reality these moments can be incredibly valuable. These periods of silence give the player an opportunity to speak about anything related that they may have on their mind, that you have not directly asked about. Itâs often the case that the player will speak about some misunderstanding they may never have touched upon if you had filled that silence with another question. Where you think the player may add detail, create brief pauses before following up.
Ensure That Players Feel Able to Criticise the Game
Sometimes players might be concerned about criticising a game because the game looks finished. For instance, why talk about fundamental design issues when they think itâs clear that the game is just about to launch? It can be helpful to make the player think that the game is in a more maleable state, so that their thoughts on the game are worthwhile.
âFeel free to be as critical as you want. Itâs still a demo.â
Despite this, this does emphasise the fact that the player isnât playing a normal video game, and that this playtest process is actually part of the games development process. Therefore, while it can be effective in encouraging players to provide more open criticism of their experience, it may be important to apply this technique selectively, and only with players that appear to present difficulty providing this type of criticism during the interview.
The Interviewing Researcher Shouldnât Take Notes
Taking notes during the interview is essential for capturing the players self-reported experiences of the game. However, taking detailed notes is a demanding task, and as the interviewer, will result in it appearing as though youâre not engaged with the player. This is likely to emphasise the fact that the interview is being recorded, and disrupt the flow of dialogue between the interviewer and player.
Detailed notes should be taken by a second researcher, from an observation room. This way, the playtest can record highly detailed data on the players self-reported experience, without sacrificing the quality of the interview.
Taking notes in the interview will emphasise the fact that the interview is being recorded, and disrupt the flow of dialogue between the interviewer and player.
Providing Opportunity to Add Detail
Itâs important that you provide persistent opportunity for the player to add detail. You can achieve this by simply asking the player every now and again if there was anything theyâd like to talk about.
Researcher: Was there anything else related to the gameplay that you wanted to mention?
Player: No, I donât think so.
Researcher: Nothing else that you found confusing or a little hard to understand?Â
[Pauses, intentionally allowed so that the player has opportunity to consider the question]
Player: Oh! There was this one thing, itâs not important, butâŚ
Start Broad, then Narrow In
Starting with broad questions such as âCould you tell me what you had to do in the game?â and âCould you tell me about the gameplay?â can be great for allowing the player to naturally speak about the game without being probed and re-directed. However, in some instances youâll find yourself wanting to understand if they were using/aware of a particular system (in this example letâs say, a stamina bar), and this doesnât come up in their responses. In these situations, itâs not clear if the player didnât mention it because they werenât aware of this mechanic, or they had some other reason for not mentioning itââŹĹ âââŹĹ such as simply forgetting about it, or not considering it important to mention. This is where you need to narrow down your questioning in order to make it clearer what you want from the player.
Hereâs an example of how to narrow down into the players understanding of a particular mechanic. In this instance while the player is providing a clear account of the games core systems, the researcher is trying to narrow down whether they were aware of the stamina system.
Researcher: What did you have to do in the game?
Researcher: Could you tell me about the gameplay
Researcher: You mentioned combat, how did that work?
Researcher: Were there any limitations to combat, or could you do anything you wanted to?
Researcher: Was there ever a reason you might need to stop attacking?
Researcher: You mentioned your health before, how did you know how much health you had at any one time?
Researcher: Was there anything else like the health bar that you had to be aware of during a fight?
If the player still doesnât talk about the stamina mechanic, then itâs likely that they werenât aware of the bar. At this point you could show them the screen and ask what it was, or refer to it directly. Players may say that they werenât really sure, or indicate that they were aware of it, but just forgot to mention it when asked.
Be Aware That Players Donât Know When Theyâre Confused
Itâs common that youâll observe players struggling with a particular feature, or mechanic, but when you ask players if they experienced any confusion when playing the game, theyâll tell you that everything was okay. Youâve just watched them struggle to drive a car in a straight line for 45 minutes, getting stuck in ways that you didnât think imaginable, and yet the player will tell you that everything was fine.
It can get difficult to understand why this might be at first, but it often comes down to the player not recognising this as the type of thing you want them to talk about. The vehicle controls might have been difficult, but in the players eyes, nothing about them was confusing, and they blame their own ability for not being able to drive effectively.
Youâve just watched the player struggle to drive a car in a straight line for 45 minutes, getting stuck in ways that you didnât think imaginable, and yet the player will tell you that everything was okayâŚ
In these instances, you can often get more detail from the player on these issues by re-framing the question.
In many instances, you can get more detail from the player on these issues by re-framing the question. While players may not consider what you observed as something that confused them, they may provide a different answer if you ask them âWas there anything in the game that you found quite difficult?â or âWas there anything that you thought took quite a long time to do?â.
Often, the player will provide detail on their experiences which indicates a clear deviation from the games design intent. For instance:
Player: âYeah actually⌠it took very long time whenever I wanted to change equipmentâ
or
Player: âIt was probably just me, but I found the driving quite difficult⌠it felt like I couldnât get there car to go where I wanted it toâ.
Shut Up and Listen
This relates back to many of the points Iâve already raised, but Iâve heard if something is important, itâs worth repeating. While your questioning is present to guide the interview, itâs important that you allow the player to speak openly about the game, in their own words, without interruption or perceived pressure. That often means, even if the player is speaking about something youâre not interested in (e.g. when it doesnât directly relate to a usability issue), simply letting them speak and waiting before redirecting the conversation.
Listening to the player communicates that youâre interested in what their saying, respect their thoughts, and value their perspective. Itâs also important to remember that silences are among the most valuable tools in your arsenal as they allow the player opportunity to speak freely about the game in ways that might not have been encouraged by your interview guide.
The first reaction of most people when they consider listening as a possible method for dealing with human beings is that listening cannot be sufficient in itself. Because it is passive, they feel, listening does not communicate anything to the speaker. Actually, nothing could be farther from the truth. By consistently listening to a speaker, you are conveying the idea that: âIâm interested in you as a person, and I think that what you feel is important. I respect your thoughts, and even if I donât agree with them, I know that they are valid for you. I feel sure that you have a contribution to make. Iâm not trying to change you or evaluate you. I just want to understand you. I think youâre worth listening to, and I want you to know that Iâm the kind of a person you can talk to. (Rogers, 1957)
Additional Reading & References
Bromley, S. (2017). Interviewing Players. In Games user research (pp 163â173). Oxford: Oxford University Press.
Drachen, A., Mirza-Babaei, P., & Nacke, L. (2017). Games user research. Oxford: Oxford University Press.
Knoll, T. (2017). Think Aloud Protocol. In Games user research (pp 189â201). Oxford: Oxford University Press.
Mixed Methods Podcast. (2018). Retrieved from https://www.mixed-methods.org/
Rogers, C. and Farson, R.E. (1957). Active Listening, Chicago: University of Chicago
Read more about:
BlogsAbout the Author(s)
You May Also Like