What I Learned About UX Research After Leaving The Game Industry

(originally posted on the Handrail Blog via Justin Wingard)

I’ll never forget the last research effort I took part in before leaving the game industry.  We had around 30 internal developers for 3 days straight.  They were asked to play our game from the very beginning, write down any feedback or bugs they found, and email them to us.  No other instructions were given and there was limited facilitation.

Yeah, it was a complete mess.

Some participants sent us a paragraph or two, others sent 10 page dissertations.  Some focused on bugs, others focused on feedback.  Everyone organized their feedback differently, there would be one email with firm bullet points and another that was a pure stream of consciousness.

Almost nobody gave us hard quantifiable information.  The process of simply aggregating the feedback into anything intelligible took a week.  It was a month before we came up with action items based on the feedback.

If I ran studies like this at my current job, I probably wouldn’t stay employed long.

Working at a design agency I’ve had the chance to work with some large companies with highly formalized approaches to collecting user feedback.  It has given me an appreciation for just how much the game industry could gain from adopting similar methodologies.

Here are some of my quick takeaways:

Lesson 1 – Keep a Tight Focus

It was so rare for us to have time allocated for user research that the temptation was always there to study multiple systems all at once.  However, this is problematic.  Trying to study multiple systems simultaneously makes it difficult to draw accurate conclusions.  The efficacy of one system may be impacted by the shortcomings of another.

To put it in gaming terms, it would be hard to judge changes to the movement controls in your MOBA if you simultaneously made changes to the pathfinding system.  The two systems are interdependent.

In the above example, a stable build with a predictable pathfinding model should be chosen as a base for testing the movement controls.  The development team could then take an iterative approach to testing.  Once a week that build could be tested with a new set of updates to the movement controls.  No other changes would be included to the build such that feedback results could be accurately compared 1 to 1.  In very little time, a dominant control method would emerge from the research.

Changes to a pathfinding system can have a far-reaching impact on gameplay and should be controlled for in research.

Looking back on my days in gaming, I wish we had taken this approach on all major game systems.  I wish we had run weekly tests that were very focused on specific game systems.  This would have allowed us to isolate the good and the bad from the target system, it would have created a more rapid feedback loop which would have saved development time, and it would have ultimately resulted in better products.

Lesson 2 – Understand Your Personas and Recruit Appropriately

When it came to recruiting for user research, management would typically let us do studies with whichever internal candidates had the lightest workload.  This isn’t too surprising, development studios are notorious for feature creep, and spare time is a rare commodity.  The problem is that this gave us a mix of hardcore and casual gamers, genre experts and newbies.  We would effectively have wildly different personas participating in the same studies, and being evaluated as a single audience

As most readers probably know, a genre expert typically isn’t a great candidate for running the tutorial, and a casual player isn’t a great candidate for PvP balancing.  Your genre expert is going to blow through the tutorial without breaking a sweat, and your casual player is going be completely lost on most PvP topics.

I can only imagine the reaction of a non-MOBA player if posed the question, “How do you balance split-lane pushing champions given the tendency of this mechanic to polarize gameplay?”

If I could go back and do it again, I would develop personas that are more representative of our target audience.  These personas could serve as a baseline for how to recruit for our studies.  A tutorial study might focus on our casual gamer personas while a PvP study would focus on our hardcore gamer personas.

I would also recruit from outside of the studio to get fresh eyes on new systems, and ensure that all personas are represented.  Not surprisingly, it can be hard to find casual players at a game studio.

Personas are used as representations of key segments of your target audience

Unfortunately, I know there would have been big hurdles with this.  The biggest challenge with this would have been getting management buy in; they wouldn’t be able to simply pick the lowest workload developers.  They would fear features dropping out of the final product because of the lost productivity.

Also, they would have serious concerns about showing game content to people outside the company, even if they had an NDA.  In many ways, the biggest challenge is convincing management of the importance of UX Research.

Lesson 3 – Have a Clear Plan on Gathering Feedback

Gamers love sharing their opinion.  They will tell you exactly what they think of a design, often time in extreme detail.

This is can create some real problems.  If you don’t have a clear plan and means for gathering feedback things will get out of hand quickly.  Imagine trying to divine a fun factor rating out of a 2 page, stream of consciousness style, feedback email.  It’s very difficult, and you’re likely to introduce your own bias.

But this is also a huge asset.  It virtually guarantees that any research effort is going to generate a tremendous amount of data.  The challenge becomes creating a research protocol that helps you to sort and organize the information you gather.  Facilitation also plays an important role here.  You can’t expect participants to scroll through a lengthy study and fill out forms in the middle of a research session.  It breaks them out of the play experience, and will likely impact their feedback.

Research software tools such as Handrail can help you plan and organize your studies

In retrospect, I wish we had spent more time building better feedback forms.  We could have asked a variety of more specific questions, blending qualitative and quantitative feedback.  We should have used a software package to help facilitate our user research.  This would have allowed us to focus on data analysis, as opposed to data parsing.

Perhaps most importantly, we could have performed contextual inquiry, watching individuals play the game while collecting notes.  This would allow us to ask exploratory questions on specific areas of game content, digging into the key questions, and finding more valuable action items.

Taken together, these two changes would not only have saved us enormous time in our data analysis, but would have also have led to more accurate research.

Conclusions

Of course, it’s easy for me say all of this in retrospect.  The hardest part of game development is simply time management.  Your mind is typically so buried in the moment that you barely have time to strategize and plan.  It’s easy to delay research efforts for a few weeks so developers have time to fix a few more bugs.  It makes sense, right?  Why test something if you already know there are problems?

This thinking is, of course, short sighted.  One of the key value propositions behind UX Research is that it will save more time than it will cost.  It’s much faster and cheaper to prototype and test your game in a low-fidelity form before asking your engineering team to make it a digital reality.  A common pushback on this is typically, “But a paper prototype is not representative of the experience we plan to create.  It might score low because it doesn’t have the production values of a final product.”  However, statements like this are very problematic.

Minimal 3D – A low fidelity runner prototype included with Unity’s 2D+3D Runner Engine

A strong game system should be able to stand on its own without beautiful artwork or sound.  High production values will not transform poor mechanics into good mechanics.  Rather, they will transform great mechanics into a great user experience.

The value of research isn’t diminished by the state of the product, that state just needs to be accounted for in analysis.  For every development studio, early research of core game systems should be a standard process. This process is the key to how research can save a company time and money, and create better products.

I feel like I’ve learned so much in my transition out games, and at the same time, it’s become clear to me that I still have a lot to learn.

Share with us:

How does your company approach UX research? 

Is it a formal part of your development process, or a side note, reserved for idle time?