Sebastian Long is a User Researcher at Player Research, a video games playtesting and user research studio based in Brighton, UK.

Playtesting is now a mainstay of game development studios all over the world - observing real people play games, throughout development, in order to evaluate how the experience compares to the designers’ vision. With the insight from watching and analysing players, developers question ‘what is understandable?’, ‘what needs improvement?’, or simply ‘is this the experience we want to deliver?’.

Finding the budget and time for playtesting is far easier for large studios, but what about smaller indie teams with rapid dev-cycles and modest scope? Is playtesting and user research compatible with indie game development? Does it deliver sufficiently valuable insights to provide a return on investment?

Being a four-man startup ourselves, we know first-hand the pressure that small teams are under. Despite the challenges, we’ve assisted many small studios in achieving their design vision, helping them produce chart-topping and award-winning games. Recently one such studio, two-person team Micro Macro Games, put the value of playtesting and user research to the test on Morphopolis: a fantastical 2D hidden object and puzzle indie game recently released on mobile and desktop.

With real comments and reactions from Morphopolis’ developer and publisher in this article, we hope that we can dispel a few myths about user research and playtesting on indie titles.

The Game

Morphopolis is a hidden object puzzle game that narrates the journey of a minuscule insect undergoing transformation; taking control of other insects and growing in size in order to interact with the world. Players scuttle their insect around a rich and vibrant microscopic environment, collecting and puzzling through a number of overlapping chapters. With only 8 weeks until release, Morphopolis developers Micro Macro Games and publisher Kumotion brought a near-final build to us at Player Research and tasked our team of researchers with uncovering the title’s usability and player experience, and to find out if Morphopolis was creating the experience intended by the developers.

Why Playtest?

Game publisher Kumotion were the first to raise the possibility of playtesting for Morphopolis. In particular, Steve Stopps at Kumotion says that he “knew that getting the difficulty just right would be a challenge. Not the puzzle design so much, more the less-apparent usability issues. There are always mechanics and information that we take for granted as developers, and we quickly become blind to issues that may impact the game experience. So I knew that usability testing would play an important role in getting the balance right. We wanted the puzzles to be difficult by design, not as a result of usability issues.”

Despite Kumotion’s keenness to open Morphopolis up to external critique, developers Dan Walters and Ceri Williams did have reservations, "When Steve shared that he wanted to run Morphopolis through a round of formal usability testing, to be honest I wasn’t sure how much value it would add,” says Micro Macro’s Dan, "It is not that I don’t believe in usability testing, but by that point Ceri and I had watched many, many people play our game, so we felt we had a good handle on what challenges the game presented, and how to resolve them. However, we both felt it wouldn’t hurt, and would likely simply reinforce what we already knew.”

Indie teams like Micro Macro Games are tightly knit, and are often able to keep a close eye on every aspect of the game. As such, the feeling that Dan and Ceri had of ‘having seen the issues’ is, we have found, quite common. So how did it hold up?

Before we answer that, lets look at how Player Research approach the playtesting process, to ensure developers get insightful information they can trust.

The Playtest Preparation

Playtesting should begin with the answer to two key questions:

What do the dev team want to take away from the playtest session?

Simply asking ‘what do you want to find out?’ can often be extremely difficult, as what you want to know is often a complete unknown, but we encourage developers to be as specific as possible when defining what they want to know.

For Micro Macro games, their key ‘unknowns’ covered both usability and user experience, as developer Dan explains, “Being a hidden object puzzle game, Morphopolis is, by its nature, a difficult game to gauge usability since part of the challenge in a game like this is giving the player space to figure things out. So initially we had questions around possible issues with the interface design that confused people or blocked progress. Then we were interested in looking for the balance of challenge and frustration: Did players have enough information to figure out the hidden objects and puzzles for themselves?”.

Who is your target audience, and what games might they have enjoyed?

Pinpointing an audience is one of the hardest things for a developer to do. Often developers will say ’this game is for everyone’, but who is everyone, really? Often “everyone” turns out to be a smaller group than the developers originally claim. Thankfully, Micro Macro Games were happy to pinpoint a specific target:

"Fans of point and click adventures like Machinarium, Botanicula, The Tiny Bang Story. Given the pacing of the game we expect Morphopolis to have most appeal to female players over 25”.

Having such a clear answer to the question of ‘who the game is for’ not only helps with recruiting the right playtesters, but will also assist in interpreting and analysing the results.

The Playtest Recruit

Finding the right players for playtesting is vitally important and ensures that players have a representative amount of knowledge of the genre, and approximately reflect the interests and abilities of the target audience. We recruit from a large database of real players, and do not use friends, family, serial testers or industry insiders who may know too much, go easy on a game, or have biased opinions.

For Morphopolis, 6 players were recruited with a 50/50 male to female split. A range of players who enjoy puzzle-type, narrative-type and point-and-click adventure style games were recruited - all of whom had at least a moderate level of experience with touch screen devices. The idea of gaining enough valuable information from only 6 people may be a surprise to some, however, if the recruit and playtest is setup and performed correctly, then much can be learned from 6 people.

The Playtest

Our playtests started with the player being welcomed into our dedicated playtest lounge (with strategically-placed cameras and audio capture equipment). The player signed an NDA, followed by a warm up pre-interview to learn more about the player and their gaming experience, and to set the player at ease. An NDA is vital for pre-release games, and we also stress the importance of making the player feel at home; we want each player to play as they would if that had just installed a game on their own device.

After this initial interview the player was left on their own with the game. They weren’t given any prompting beyond simply ‘start with chapter one, and knock on the door if you need anything'. Leaving the player alone at this point is important,  very few people are comfortable being watched by another person in the room - a fact that is most relevant for studios running internal playtesting. It can also be awkward to sit in the same room as someone who is struggling with a game, particularly your game, and resist the urge to assist them, especially when asked. By leaving the players alone and watching via the cameras from a separate room, we are able to let the player’s actions do the talking, take notes, and follow up with a later interview to gather subjective impressions, and clarify observed behaviour.

We always encourage developers to attend playtesting at Player Research. There is nothing like watching real players interact with your game. In the case of Morphopolis, time, money, and geographical constraints left the developers unable to attend; but with the relative ease of access to streaming video technology (with password protection) nowadays, developers can often still watch playtest sessions live.

As we watch the sessions, questions are raised about the designers’ intent - “Is the player intended to explore here?”, “Should the player be searching for this long?” “What is this UI element designed to communicate?”. Our focus is always on ensuring the game is achieving the underlying experience the developers intended.

After 40 minutes of play a researcher returns to the room to conduct a structured end-interview, beginning by exploring players’ understanding of the game. Then, once we have a baseline of the game elements players do and don’t understand, we question the usability of the game - the controls, the interface, and so on. Finally, we move onto the player experience itself - fun, enjoyment, frustration. Throughout the final interview we also take the opportunity to raise any issues we observed while the player was playing - being very careful not to lead the player and influence their answers, and remaining as neutral as possible. This interview approach allows us to build up a picture of the whole player experience, and where the issues are occurring.

After the interview, the player is paid, and leaves. Choosing to pay players is deliberate: we want players to arrive on schedule, and we want very specific people - often those who won’t play pre-release games simply for the experience.

The Analysis

After the playtests it’s time for analysis. Playtests generate a rich pool of data; hours of gameplay footage, interviews, pages of notes, and it has to all be brought together, understood, key problems extracted, irrelevances eliminated, issues ranked (sometimes with heated discussions), and then all this has to be communicated quickly to the developers. For a playtest the size of Morphopolis, this analysis and accompanying report takes one day for two researchers. Such a short turn-around is crucial for the fast-moving games industry.

The speed of feedback was extremely important for Steve at Kumotion, he wanted to playtest while there was still time to make changes; “It was important that we did the test when we still had time to act on the results. So many developers perform usability testing at the 11th hour, and expect it to be a warm pat on the back - telling them what a great job they’ve done. In my experience the results can often be more of a cold wake-up call, showing lots of things you need to fix to improve the experience. So we planned for this, and had time set aside to react to the feedback.”

Player Research analyse playtest results in isolation, without the development team involved. We believe it is important that we can discuss and critique without interjection, and are also hard on ourselves - questioning the accuracy and importance of the findings we produce. By bringing together a team with mixed backgrounds - interaction design, computer science, psychology, and human factors - each of the Player Research team has a unique perspective on the problem and potential solutions.

At the end the of the analysis Micro Macro Games and publisher Kumotion were provided with a 28 page written report, presented alongside full videos of the playtest sessions.

The Results

What were the results? As we don’t have space for the full report here, we asked the team at Micro Macro Games to pinpoint the issues and findings that they found most valuable. This was their reply:

“We’ve chosen three issues from Player Research’s report for the article, all of which we’d not anticipated prior to the playtesting. Firstly, a great example was Morphopolis’ incompatibility with colourblind players. It sounds obvious in retrospect, but given that 10% of the male population are colourblind, not adjusting the design could have caused frustration for a significant number of our players.”

The main offender for colourblind-unsuitable design was the first puzzle in chapter one. Players have to match the colours of movable pods with the static underlying element in the flowerhead. Simple perhaps, unless you’re colourblind. Micro Macro Games’ Ceri Williams explains, “This was immediately flagged by Player Research, and the solution they suggested was simple: we just needed to represent the puzzle mechanics by using colours and shapes. Since this was an early puzzle, this approach had the added benefit of making the puzzle mechanics more obvious to all players.”

Micro Macro Games also note the importance of an issue highlighting a discrepancy in how players expected to move their insect, and how the game actually handled the input, as Dan explains. “Another unexpected issue was the way that the playtesters wanted to interact and move their insect when the path forked. We knew how to interact as we had written the mechanic. However, the players were not armed with this knowledge, and all of them struggled.”

“In the way we had implemented the [path forking] mechanic, players needed to move fully past the fork in the path, then change direction to move along the new path”

“We thought there was nothing wrong with this. But playtesting showed that players just wanted to just drag to where they wanted the insect to go.”

“When we saw this in the report it was obvious, and that is the point. Many of the elements of the report seemed obvious, in hindsight, and very sensible. But as developers we were simply too close to the game and became blind to these issues. That is why having this external perspective was so valuable.”

“Finally, another interesting result concerned timing. We have a puzzle later in the game, where glow-bugs flash, and the player has to remember the sequence, then tap the glow-bugs in the remembered order correctly. Of course, part of the puzzle is figuring out what to do.“

“In the playtesting build we had a delay between two of the bug flashes - but this delay turned out to be a fraction too long. This extra pause confused players and led to misunderstandings on how to solve the puzzle.”

Players, incorrectly thinking that the flashing sequence to be repeated had stopped, began to tap the glow-bugs, but found other glow-bugs highlighting when they did so. Resultantly, players were entirely confused about the structure of the puzzle, and how to solve it.

To solve this, Dan and Ceri altered the structure of the puzzle, trimming the gap between the glow-bug flashes, and adding both audio and visual feedback when players tapped glow-bugs incorrectly - helping players to understand the structure of the puzzle.

These were just three of around 40 issues highlighted by the Player Research team.

Conclusion

Returning to our original question: is user research compatible with indie games, and does it provide them with valuable, actionable insights that deliver a return on investment? We’ll let Morphopolis’ developers Dan and Ceri have the last word:

“We are committed to our players, and want our game to be challenging, not frustrating; at the same time, we don’t want the game to be too easy, and I believe the playtesting revealed issues that have really helped us achieve this. One of the fears associated with playtesting is that it leads to a ‘dumbing down’ of the game or a process of ‘design by committee’. We did not find this to be the case. Rather the feedback we received highlighted valid, and unexpected, issues.”

“We were pleasantly surprised by how useful playtesting proved to be, and we’ve acted on nearly everything in the report. Our game is still challenging, but we believe it is now challenging for all the right reasons and early reviews from the press and players seem to show this too.”

Morphopolis is now available for iOS, Android, Blackberry, Windows 7/8, OSX, and is coming soon to Kindle Fire and Windows Phone. More details online at Micro Macro Games.

Article written by Seb Long and Dr. Ben Lewis-Evans, with contributions from Dr. Graham McAllister & Alistair Gray (Player Research), Ceri Williams & Dan Walters (Micro Macro Games) and Steve Stopps (Kumotion).