Tanya X Short is director of Canadian independent studio Kitfox Games, developers of Shattered Planet and the upcoming Moon Hunters. She's a frequent blogger on Gamasutra.

Hey you, designer reading this! 

You don't have to act like you know what you're doing. You can drop the act. You know full and well that you're not qualified for your job, your game is terrible, and it's all your fault. You've failed! Congrats!

At least, that's how it can feel after a playtest. Putting your unfinished work in the hands of a stranger can make you feel incredibly vulnerable. You already know the game hasn't achieved its potential; having randos point that out to you can be incredibly frustrating. Even the nicest, most excited playtester can ruin your day, because it feels like every suggestion and idea is a passive-aggressive comment on the ways the game is disappointing... in between all of the bugs, problems, and other ways it's clearly, actually failing. 

Some of the most entertaining bugs are found in online play, which makes feedback double the fun.

Any experienced designer will emphasize over and over how important playtesting is – it's cited as crucial for the success of virtually every successful game ever made. But it starts to feel a bit like snake oil when you know for a fact that lots of unsuccessful games also performed lots of playtests. It's not a cure-all, clearly. But running and digesting playtests is a skill, and one which will dramatically increase your usefulness as a designer.

There's always an excuse not to playtest. It's too early in the cycle to learn anything useful, or too late. There's a major bug that will ruin it. The next build will be better. There's no time in the schedule for it. Yadda yadda. 

Stop lying to yourself. You know you need to playtest every few weeks. Kitfox playtests as soon as we have a prototype, even while it's ugly and janky. Yes, it hurts and slows down production. Yes, it's important. Plus it has the side-benefit of making sure we 'work clean' and don't let game-breaking bugs linger for months*, but that's a different article.

How to Playtest: Imposter's Edition

There are many guides to best practices in playtesting, especially to solve usability, but most of them can be boiled down to “let the player play”. 

Before the test, articulate your assumptions and blind spots, so that you can work to disprove them. For example, “I think this level takes 10 minutes to solve. I think this challenge takes 2-3 tries for someone familiar with this genre. I think the Witch is the most appealing character.”

Then, when the test starts, don't explain, interrupt, or suggest. Don't answer the playtester’s questions immediately -- write them down. Take notes to refer to later. Ask questions only to clarify the player's intent where it's uncertain what they are trying to accomplish (“What are you trying to do right now?”), or afterwards.

Procedural generation is not for the faint of heart. The feedback is... not usually this clear.

Designers are used to creating, persuading, and collaborating. It’s what we’re good at. But a playtest is not the time for that. Here, you are only a watcher. Observe. Soak up the experience your game offers from a fresh perspective. Understand what the player’s trying to do, and why they're trying to do it.

We recently invited roughly a couple of thousand backers to try an early beta of Moon Hunters. It was the first time players were able to play the game for longer than ~15 minutes, and from the comfort of their own homes. And as the designer, it was torturous.

It was fine. The team's morale was high as we saw players finally really get into it, unlock things, and play without us over their shoulders. And, of course, there were large problems, aside from bugs. People didn’t understand the core story, the premise of the game, the antagonist, the controls.. but that meant it went according to plan. The point of a playtest is to challenge. It challenges developer assumptions, priorities, and future plans, almost as if the game itself stands up before you, shouting, “You were wrong! You were wrong in so many ways!”

Staying hands-off is particularly difficult if you aren't confident in your design. My impulse to interrupt is sometimes rooted in the sense that unless the player understands, and has fun, I am a failure and my game is worthless... but that isn't true! In fact, it's almost guaranteed that players will be confused, misunderstand, and be a bit miserable for the first few playtests – your game and your design skills will be better than ever after you figure out what's gone wrong.

"Playtester #4121, your suggestion to put some text here is very helpful. Thank you!"

Metrics are a good tool too, but it's not a shortcut, since rather than watching over a player's shoulder, you'll instead need to schedule in time to figure out what the metrics mean, and compare them to your assumptions and predictions. A thousand economists can implement a million metrics each, but without a designer to interpret it, it's just numbers. The human mind is uniquely set up for pattern recognition – put both yours and the tester's to work. Don't let your insecurities get in the way.

How to Take Feedback: Imposter's Edition

Designers with thin skins can find it tempting to throw away anything that doesn't align with their existing view of the game, because otherwise, it invalidates their position as Supreme Designer and Visionary. This generally happens inside the designer's head, so it can be completely invisible to anyone else except over longer periods of time, as others' ideas crash up against mental shields and fall away time and again, leaving the designs standing untouched. And it's true that some feedback is genuinely not useful, and costs of implementation are a real concern.

On the other side of the coin, designers without enough self-confidence can find it tempting to take every piece of feedback, because they are so open-minded that they've started to doubt the premise of their design. After several rounds of playtesting, the game can change direction so many times, the team can become dizzy and lose focus of their goals. And it's true that some feedback is genuinely gold and deserves immediate attention.

As I haven't seen many game designer-specific guides to taking feedback, here's some best practices that will help you open your heart to the right kinds of feedback, and hopefully help you choose which feedback to take, and how. Feedback isn't like medicine to be swallowed in regular doses – it's closer to a strange misshapen fruit, requiring you to carefully peel away thorny layers before devouring the nutritious flesh. It takes work.

Asking users to vote on options is feedback "easy mode". Players helped us pick this dance. But is that really feedback? It's certainly not as useful as criticism...

Types of Feedback

To reiterate, the things the tester does is much more important than the things the player says. However, the things they say can also be useful if processed correctly. 

We try to wring as much out of the player as possible before they forget, internalize, or otherwise lose the impression the game just gave them. 

Ideally much the feedback is collected and recorded during the actual play session, but that may be difficult if the player is remote, shy, speaks a different language, or otherwise has trouble communicating on the fly. Our most recent playtest questionnaire for Moon Hunters looked like this, if you’d like to use it. Metrics or observers can also record statistics like playtime, death count, choices made, etc.

Regardless of how it’s collected, here are three major types of feedback players give, and how best to use them.

Criticism

“I didn’t like ____.” 

This is the part you’re dreading, but in testers over 13 it’s unfortunately not really as common as you’d think, even when your game isn’t much fun. They’re more likely to give it when the creators are far away (i.e. beyond a computer screen, in another room, etc), but there are just too many nice people in the world for it to be as common as it is in Steam user reviews.

How to use it:

Observation

“This feels like _____.” or “This works this way because ______!” 

Tester observations are sometimes wrong, obvious, or completely bizarre. This is the most useful type of feedback, and the primary reason why I feel blessed to be a designer in the age of Let’s Plays. In single-player games it’s difficult to get testers comfortable enough to make observations at all, never mind continuously say what they think they’re seeing. At most reply, “Interesting!” or “Tell me more!”, no matter whether they’re right, wrong, or off the wall.

How to use it: the same way you use your own observations of their play -- match it up to your assumptions and expectations. Interpret the differences. Where they’re wrong, does their misunderstanding (or overactive imagination) hurt your intended experience? If not, maybe don’t worry about it, but keep it in mind for future decisions. If so, figure out where it came from and fix it.

Suggestions (Expansions)

“It be cool if…” or “What if…”

People generally give suggestions in order to build on a part of the game that already excites them. Something in the game connects with them and they can’t help but react. With game enthusiasts, this often means they’re reminded of a memory or past experience, and it spurs them on to bubble over with what they would call ‘ideas’ and what I would call ‘comparisons’.

How to use it: many of these suggestions are wildly out of scope and may even equate to a genre shift. Generally you can add these to your long-term wishlist of potential ways to expand the game and otherwise set them aside. Sometimes you can throw them out as “Oh, what they actually wanted was X and I’m making Y.”

However, it’s dangerous to dismiss them out of hand, as you may find a pattern of mismatched expectations. If many testers eagerly suggest capturing monsters in your game, maybe there’s something accidentally Pokemon-esque being communicated, and this may translate into long-term disappointment if it’s not addressed, in one way or another.

Suggestions (Fixes)

“It’d be great if this thing here instead worked like _____.”

It’s easy to assume that testers make suggestions because they’re trying to fix something broken in your terrible game, and occasionally this is the case. This is the least common type of suggestion, in my experience, because unhappy people don’t tend to be imaginative. If something frustrates someone, they’re much more likely to avoid it entirely, and focus on what they enjoy. Even so, it sometimes happens, if you find a particularly diplomatic or experienced tester.

How to use it: 

 Summary

Imposters can often be terrible at taking feedback, oscillating wildly between taking none of it and taking all of it. But with preparation, discipline and careful emotional distance, it’s a skill that turns an unfinished feature (or set of features) from an impossible tangle into a solvable design problem. Good luck!

*Hahahaha! I wish. But really, it’s a great goal to strive for. Ask me in 2017 if we managed this with our next game.

Further Reading

Playtesting, according to the Knowledge Guru

Five Tips for Better Playtesting

Playtesting in Game Design Process, by Backyard Game Design

Be sure to check out Tanya's previous piece, "The secrets to scrappiness: Fighting to survive as an indie studio."