Better Games from Diaries and Data

Harvey Owen is a Games User Researcher at Player Research in Brighton, UK. Through playtesting and player experience evaluation, Player Research work to ensure games meet their designers' intent.

Diary studies are a research methodology that provides rich, meaningful data about how players interact with a game over a long period. Diary studies give players an opportunity to record their thoughts and feelings about an unfinished game in the moment. This diary tool gives the developers a truly nuanced and deep understanding of players’ motivations over time, their moment-to-moment reactions over weeks and even months, and unrivalled feedback on how best to improve the game from a player experience perspective. Despite these opportunities for insights that are far greater and more actionable than analytics or early access feedback alone, studios are rarely aware of diary studies, relying all too often on cold, impersonal metrics as the sole method of ‘understanding’ their players. Without the addition of qualitative data about how players feel to supplement analytics data about what they did, any ‘insight’ into players’ engagement, monetisation and retention risks being shallow, unfounded and potentially outright false.

What is a Diary Study? Simply speaking, a diary study or ‘play diary’ is a series of short questionnaires, completed each time a player completes a session of the game, while going about their normal life. Data is collected over time, across multiple players, and paired with data about their in-game progression to form a picture of their experience - insights from which can be used to balance the in-game economy, difficulty, tutorialisation, and even to better understand the game’s potential audience.

Why should I run a Diary Study? Diary studies provide access to detailed information about players’ experience over a much longer period than a playtest, which typically last an hour or two. Tracking players’ journeys, the player learns the game in their own time, in their natural environment, going about their everyday lives. This not only allows them to play further into the game, but also to develop their thoughts and feelings more fully, and provide deeper and more meaningful feedback. Diary studies, therefore, offer a very different perspective to other research methods (like playtesting). Players’ stories emerge from the data - threads of repeated frustrations, or hard-earned victories over difficult levels -  a whole new understanding of the impact of game mechanics, difficulty curves, and in-game economies; all revealed in the player’s own words.

“I have absolutely no idea what needs to be done for the Focus Bonus.“ -Day 6

“Still have no idea what Focus Bonus means. Only ever get 0 for it.“ -Day 10

“I still have no idea how the Focus Bonus mechanic works” - Day 11

- Except from a real player diary; the ‘Focus Bonus’ was vital to balance of the in-game economy

A game might be rigorously playtested, issues hammered out, developers made happy, then launched; but only when the game is in people’s living rooms, or perhaps being played on the bus, do the niggles and frustrations set in. Conceivably, to save space, the game’s playtest sessions were conducted with players sitting close to screens; only when played at home in the living room is it clear that that text is far too small. Subtitles and menus are tiny and illegible - almost impossible to comfortably read from the other side of a room. Issues like this are ubiquitous, and aren’t limited to usability - “the boss at level 10 is impossible”, “the next sword upgrade needs too much grinding” - these are the types of potential issues that diary studies excel at identifying.

When should I carry out a Diary Study? Diary studies are most effective when a game is near launch, has enough content present for players to engage with, but has enough development time remaining to make those vital changes. Diaries don’t need thousands or even hundreds of players - 20 to 40 is typical. They should be completely run and actioned before cash is injected into user acquisition - so as to ensure those expensive acquired players have the best possible experience.

Analytics and diary studies studies provide complementary types of data - quantitative (numeric), and qualitative (conversational). Analytics bring the hard data from hundreds, thousands, even millions of players, yet lack context - but when supported by diary studies, the combination can provide deeper insights about what players do, why they were doing that, and - critically - what they enjoyed of it. All too often developers and studios rely on analytics alone as a ‘foolproof’ way of finding out where latent issues exist within their games.

In order for a diary study to be most effective, developers can set questions or research areas beforehand. If diaries are being filled in many times per day, as in some cases, players won’t have the time nor patience for long-winded responses - so ensuring you’re only asking exactly what you need to know is vital. Traditional iterative playtesting can be incredibly valuable for highlighting the ‘known unknowns’ that the diary study can be used to explore in the longer-term.

“I’m Trying to enter a MEGARACE and told to buy a new car. I did that as instructed, only to be presented with the same situation. Lame.” - Day 13

- An excerpt from a real player diary; saving for a new car took several days, but the newly unlocked car type wasn’t clear, leading the player to purchase the wrong one, blocking her progression.

How do the players record their Diary entries?  The method by which diary data is collected varies, with no one method being a clear standout. Here at Player Research we have used a dedicated smartphone diary entry app, and also had players fill out a diary entry using an online survey tool - each of which was transmitted to us in real time, and allowed researchers to communicate back to players if required. The most effective methods are the most seamless, and least intrusive - perhaps even tools the player is already familiar with. The more effort a player needs to go to in order to fill in the diary, the higher rate of drop out will be seen, and increases the chance of players simply not bothering to complete a diary entry after a play session.

If the game being tested is on a smartphone, use a method that is accessible easily and quickly through the device, that poses as little interruption as possible. Ideally, the filling in the diary shouldn’t be a burden, so if as much as possible is auto-filled (such as date, timestamp, location in game, location of play, etc.) then it will reduce strain on the player.

Asking quantitative questions about the players’ experience (scoring things like difficulty, or enjoyment out of 7, for example) can allow researchers to map out players’ reported experience over time, and even compare between and across players to better understand their journey over time.

"I continued with my quest to kill a Goblin, but I have yet to find a single one" - Day 5

An excerpt from a real player diary; this player (and many others) couldn’t complete this quest, added 3 to 4 sessions into the game, because they’d yet to see the Goblin Castle.

Why not use Diary Studies all the time? Having harped on about the benefits of diary studies and where they can and do provide actionable results in games user research, it’s only fair that their limitations are also outlined.

One of the most commonly cited downsides of diary studies is the fact that they are labour intensive - requiring a lot of man hours, and a larger bill than traditional playtesting, especially if outsourced. The fact that diary studies are a much larger budgetary cost than analytics packages, yet cover a far shorter period of time might raise questions about their ROI; as we’ve discussed, these are questions that are easily answered by the quality of insight generated.

“I have only one challenge left - it takes a ridiculously long time to level. I have to make the [item] 10 times and they take about a minute each. This is super annoying.”

- An excerpt from a real player diary; players were given algorithmically-generated objectives, but some were more successful than others.

Diary studies have additional complications, like how should we keep players playing? How do we handle dropouts? Build security? How do we reward players for their time in keeping a diary? Managing and encouraging motivation to continue playing a game throughout the duration of a study is often a difficult task. Players are often being rewarded for giving their time to participate, so it can be difficult to know their true reasons for continuing (are they just in it for the money?). Players are always going to do what they are incentivised to do, so by giving them an opportunity to drop out early and still get paid, then they are likely to do that, however offering no incentive at all would likely see players not bothering to fill their diaries and dropping out as soon as it becomes inconvenient.

Despite these challenges, diary studies provide rich, actionable data from real players in a setting that is entirely natural. Playing in such a way that mimics how they would play the game upon its release allows them to fully develop their thoughts, attitudes and feelings towards the game, as well as highlighting latent issues that the game may have across all aspects of the experience. Although not perfect, the use of diary studies, particularly when combined with in-game telemetry, allows for unprecedented insight into issues with engagement, retention and monetisation that may otherwise never be revealed by analytics alone.

Learn more about Player Research:

Website: http://www.playerresearch.com

Twitter: http://www.twitter.com/playerresearch

Authors Note

This article was co-authored with Seb Long, also of Player Research, who provided excellent guidance.