This year was the first time I have had the privilege to attend the Game Developers Conference in San Francisco, and also importantly as a Games User Researcher the associated Games User Research Summit. I had been lucky enough to be selected to talk and somewhat naively I had vague ideas of writing up my impressions of attending the conference on a day to day basis.
Welp, thanks to the wonderful people I met there keeping me busy into the early hours every single day that (happily) never happened. But before these memories fade I wanted to get some of my impressions down. These impressions are based on my notes and the notoriously bad recording device of human memory. So apologies in advance if when describing a talk I get something wrong. Please feel free to correct me. The content below was first posted on my blog on a day by day basis, but has been combined for this post. Where possible the titles below are linked to the talk on the GDC Vault.
Day Zero – Sunday – Hello San Francisco
This is not directly conference related, but the first day there (Sunday) was mainly spent exploring the city of San Francisco. Highlights were enjoying a long walk out to Castro Street via as many comic and RPG gaming stores as I could find. Then taking it easy by heading back through town to the waterfront, until finally registering at GDC where I had found that I had made it big with my name spelled out in cardboard for all to see at the speaker registration.
Day One – Monday – A Day of Stories
After a lovely breakfast at The Grove, it was time for talks!:
First up was Computers Are Terrible Storytellers – Let’s Give the Humans a Shot by Stephen Hood from Storium.
As part of the narrative track Stephen laid out his “narrative wormhole” of computer based storytelling where the computer can only accept the limited input that it has been designed to receive. As an alternative Stephen promoted allowing players to improvise and make their own stories. To do so he laid out six techniques to help players ; 1. Sharing, let players tell stories together, 2. Leave whitespace, a gap for players to fill, 3. Creative constraints, give players a framework and structure to work from, 4. Use dramatic economies, stories told with limitations (such as point totals that are spent) on players so they have to make interesting choices, 5. Provide setup, give players writing prompts and they can build worlds, and 6. Provide pacing, set an end point for players to work towards.
This was followed by Narrative Review Process – Getting Useful Feedback on Your Story by Jonathan Dankoff from Ubisoft Montreal.
This talk has already been covered well by Polygon but in essence Jonathan, a fellow Games User Researcher, laid out a method for working to improve narrative in games and put forward an argument as to why this was important (i.e. current feedback on narrative often too late in development, leading to a lack of iteration, the feedback that is received on story is often informal, and to gain stronger buy in to narrative across the development team). The method involved forming teams of 6-10 people, half of which were writers and half of which were other developers on the project (e.g. designers, leads, not writers). This team then had time to give notes on a very early treatment of the narrative beat, which was then collated and analysed, put into a report, presented to the writer, discussed one on one between the researcher and writer, and then discussed in a group with a focus on finding solutions and understanding comments. The success of this method is yet unproven as the games it has been used on are not yet out, but Jonathan was able to say it had helped writers find issues they didn't know were problems, and to work through issues they knew were problems but couldn't solve in isolation. As well as increasing buy in across the team and allowing for more early iteration to occur. Sounds like a win to me.
After lunch it was time for Strategic Community Communication – Managing Your Channels Without Running Aground by Maurice Tan from Deep Silver.
In this talk Maurice covered the difficult process of choosing which social media channels to spread your precious time and resources into. With the assistance of plenty of humour Maurice laid out the importance of focusing on the utility a channel will give you, the goals you have for the channel, the role the channel will play, and understanding your own resource limitations. He also stressed the importance of monitoring success, but not letting metrics rule your life. His final message was that it was important to be mindful, to not spread yourself too thin. Not just for yourself or for your fans, but also for your organisation. It is important that organisations buy into and understand what a community manager can do for them, and without this support communication is much less likely to be successful no matter the channel.
Next was Meaningful Play: Monetising F2P Without “The Pinch” by Scott Rigby from Immersyve.
To me this talk felt a bit like a sales pitch, and I don’t really find self-determination theory particularly compelling. To me, self-determination theory is a nice idea that has its merits, but its proponents tend to see the world as very black and white (or as very intrinsic and extrinsic perhaps?). HOWEVER, this aside, Scott argued that some F2P models can make players feel shamed and duped in the name of short term profit, which I can agree is unlikely to be very productive in the long term. Even if I am less certain that it is because specific “needs” are being violated. That said, autonomy, relatedness, and competence, which Scott promotes as basic psychological needs, are motivating for sure and there is certainly more room for games in the F2P space to grow to help meet (for example, by providing opportunities for players to be altruistic, cooperate, and for others to benefit when others spend in a F2P game), rather than suppress (for example by taking advantage of spenders by increasing prices after several purchases) these desires. He also discussed the fact that metrics can tell you what is happening, but not necessary why things are happening (e.g. there may be some time lag between motivation being lost and drop offs occurring in metrics). Essentially Scott’s message came down to the fact that players do not like feeling like they are out of control, that they are being controlled, or generally being screwed with. As Scott says, players are not dumb, and in the long-term treating players poorly will hurt F2P as a model.
I then had to meet some people, so I missed the last talks of the day and instead checked out some of the games in the West Hall. The train jam games in particular were interesting to check out and see exactly what could be done in 56 hours on a train. Please do check them out on the website, they are good fun!
Day Two – Tuesday – Games User Research everywhere!
On day two it was time to get up early and get a bus out to the Games User Research Summit held in Sony’s impressive offices in San Mateo. The Games User Research Summit is an annual opportunity for those interested in Games User Research to gather, network, and share their experiences (assuming they can make it to San Francisco that is). The summit was well attended and organised and included two tracks. As such, I wasn't able to see everything, but I did manage to catch a bunch of interesting talks. I should also say that you can see the official photos on Flickr, which were taken by Tiffany Young:
First up was a welcoming speech from the appropriately purple clothed Jordan Lynn (Games User Researcher at Volition and an organiser of the summit), which was to the point and well put. Then it was on to Rage Quitting: Predictors and Long Term Impacts by Elizabeth Schmidlin from Sony.
Elizabeth shared the results of a large scale survey she had carried out among undergrads at a Texas University looking at reasons behind rage quitting. Some of the headline findings were that men were more likely, even when controlling for exposure, to report raging in FPS games, whereas women were more likely (again when controlling for exposure) to report raging in racing games. The biggest determination of rage quitting due to in-game features was reported as players feeling like they were not getting enough out of game compared to the effect they were putting into it. Followed up by getting stuck on puzzles. Whereas, in terms of more technical features a game feeling like it wasn’t responding as expected was the other most commonly cited reason for raging. Finally, social features could also lead to raging with the three most important features mentioned (tied in terms of importance) were another player being overly aggressive, another player cheating, or another player being perceived as just too good.
In this talk Bill covered his time at Irrational Games and how Irrational handled user experience (UX). Going from System Shock (before he joined) all the way to Bioshock Infinite. Interesting titbits were that 8 months out from shipping the game an external playtest was run, only to find that players were scoring the game three out of ten and reporting that they had no idea who they were, who the robots were (they had no good understanding of the Big Daddies), and why they were in the ocean. Players, apparently even referred to the game as a cheap Half-Life 2 rip-off. Based on this feedback the plane crash was added and the introduction to the Big Daddies that is in the game now was also added.
Similarly, while there was more focus on testing earlier in Bioshock Infinite the introduction was initially not hitting with the players who found there was no sense of urgency. Based on this the note and corpse were added to the Lighthouse, therefore increasing the feeling of tension at the start of the game. Interesting stuff. He also discussed internal arguments over keeping the controls from Bioshock or moving to an Aim Down Sights model. These control methods were apparently playtested and it was found that while the aim down sights model didn’t impact on vigour use (a predicted outcome) it did lead to people standing still and aiming more often, an outcome the developers were not happy with. Finally, Bill discussed the Skylines in Bioshock Infinite as being an exciting feature, but one that players struggled with due to the high cognitive demands of maintaining control, shooting, and navigating the skylines at the same time. As such, a Skyline Playground zone was built to test with and allow fine tuning of the mechanic. In the end he wrapped up by stressing the importance of user research at developers, recommending starting with expert reviews and internal playtests, and then moving to external playtests (but not waiting too long for this).
Recruiting Tactics for a Modern Audience and Lab Space Design Panels.
These panel sessions were a joy for a methodology nerd such as myself. In them the different recruiting methods for getting participants for user research were discussed and the hassles and victories that can come with setting state of the art user research facilities were examined. Hint, phone screen players to make sure they have the knowledge and experience that you need, and consider the noise that can come over cable trunking. Oh and OBS was rightfully unanimously praised as being awesome.
The message from Games User Researchers is, rightly, test early and test often. Do it, all the time. Playtest. Please. As soon as possible. Like when you have paper and/or digital prototypes. But that doesn’t mean that there is no value in testing later in production. In this talk Veronica focused on the user research carried out in the last 6 months before the launch of PvZ (with playtests being run as late as two weeks before launch). This testing was mainly focused on map and class balance and Veronica stressed the importance of using large sample sizes, with longer play sessions and a mixture of repeated quantitative and qualitative measures across a long play session. Asking players as a day of play went on how motivated they were to continue playing, how engaging the game was, and how much enjoyment they were getting as well as looking at their behaviour. She also stressed that user research at this time should be about fine turning and needs to be focused to address issues that can be fixed relatively quickly.
In this talk Jennifer discussed the work she had carried out on Destiny focused on the combatants in the game. She outlined the early work where there was a focus on players evaluating concept art and looking at images that were presented to them rapidly (for about 5 seconds) to see if enemies could be spotted amongst various backgrounds and if their design gave appropriate indications of their threat and faction/race affiliations. There was also a focus on how enemy weapons looked, the threat they communicated, and on what abilities and weakness players expected different enemy types to have. Jennifer then discussed some of the subjective measures used to gain deeper understanding of player perceptions. Many of which she said were tracked over year long periods, allowing comparisons to be made across tests as the game developed. These including general questions about how combatants behaved and interacted with the player, indications of how dangerous enemies were, and how players approached bosses and how satisfactory they were to fight.
She also discussed the objective data that was collected, such as event based data on when players were killed, how, by what, etc. One interesting case study from this was the “Shank Flank” where it was found that the low powered Shank enemies in the game were getting too many kills. This was, they found, because Shanks would spawn behind the player and stay there. Then due to their low damage players weren’t noticing and Shanks were finishing the players off when they were engaged with more powerful enemies that they could see. This was fixed by changing the Shank spawn points (World Design) and changing their behaviour to attack from the front (AI).
Destiny User Research Q&A featuring Jennifer Ash, John Hopson, and Nick Hillyer from Bungie.
In this panel three user researchers from Bungie faced a barrage of questions (I may have snuck in two…) about their processes and various factors in Destiny including, but not limited to, the Cryptarch, loot drop balancing, class balancing, colour blindless, simulation sickness, Phogoth’s tummy, the move to aim down sights, the surprise of how the community reacted to the Iron Banner, and how they work in terms of carrying out user research within Bungie.
Anders gave a comprehensive presentation on how to start to deal with the masses of player data that is often being collected. Specifically, he outlined four steps: 1. Knowledge discovery, 2. Selection, 3. Interpretation, and 4. Application. He stressed that you must go through these steps and that it was cyclic process as player behaviour changes over time, as do games, which echoes into the data. He then discussed player profiling, mentioning bottom up profiling (or “explorative profiling”) where you work from data coming in, which can lead you open to biases. The alternative is top down profiling, where you go out with specific hypothesised profiles and see if they work with the data. Anders advocated for the idea of building “protean” profiles from day 1 (e.g. persona like profiles) and then following players through the whole life cycle of play (from start, until they abandon the game).
Anders also made sure to point out that data profiling is not an objective process. This is because as a profiler you have so many different choices to make about processing data and what you chose will change the outcome. He also stressed that you can get pretty far even with just descriptive statistics and segmentation of player groups. Anders then went on to cover some other methods such as clustering. All and all his talk was jam packed with details.
At this point, my iPad, on which I was taking all my notes crashed. As such, from this point on (and for the rest of GDC) I am pretty much relying on my memory. So, if there are less details it is not that the talk wasn’t interesting, it is just that memory is not as good as note taking (not even close).
For this talk James took us through some of the challenges and situations that came up when carrying out user experience research on Dragon Age: Inquisition. These included working on a good plan to make sure a diverse and LGBT recruit would be carried out, and then finding that due to Bioware’s relatively inclusive history in this area that a normal recruit could just be run and would result in a diverse crowd coming in to test the game anyway. James also stressed the difficulty of carrying out user research in such a big, open game – which was also still under development so constantly changing. He cited one specific example where a player spent three hours just wandering around in featureless, content-less mountains and generally having a great time. Great for that type of player, but not so useful if the playtest session was to test out class balance or some other mechanical aspect of the game. Much like Bungie with Destiny, concept art was also tested in Dragon Age. In this case they looked at how recongisable the characters where, romantic interest, enemy threat, and so on.
James also covered how DA:I originally had a Skyrim like compass navigation system, but users just couldn’t use it, leading to the (much, much better in my opinion) system that is in the game at the moment with the bottom mounted radar. Other interesting nuggets of information were that about 25% of players use tactical mode all the time, with most others only using it where necessary, that bears were the bane (and ultimate opponent) of many playtesters, and for a long time Solas had text to speech for his lines, which some players thought suited him.
After a little bit of housekeeping from David Tisserand (current head of the IGDA Games User Research SIG and researcher at Ubisoft), it was time for the keynote UX Invaders: We come in peace! How we can collaborate with developers to frame “fun” by Celia Hodent from Epic Games.
She had the best t-shirt of the day and discussed different ways to approach user research and getting developers on-side. This started off by addressing common misconceptions, like user research is just “common sense” (no such thing, this is hindsight bias!) and that the claim that Dark Souls shows that user research isn’t needed misunderstands the process. User research isn’t about making games easy, it is about making the best games possible for a certain target audience. As such carrying out user research on Dark Souls would be entirely possible, you just have to consider the audience, and in this case make sure the game was punishing enough for them. Ultimately, Celia was stressing that user experience is not just the concern of user researchers and user interface designers, but that it is about the whole studio achieving their goal. A sentiment that you just can’t disagree with.
She then went into her usability plus game flow = good user experience model. To do so she went through the heuristics she uses for usability (Signs & Feedback, Clarity, Form Follows Function, Consistency, Minimum Workload, Error Prevention & Recovery, and Flexibility) and for game flow (Pacing, Motivation, and Emotion). Celia provided clear examples for all of these, calling mostly on the work that she has been carrying out at Epic on Fortnite. The talk in general reminded me of listening to a lecture in Human Factors class, and I mean that in a good way (I know some people had bad experiences at University, but I loved it and had great lecturers). Celia finished up her talk by outlining four steps for collaborating with developers in user research: 1. Debunk UX misconceptions, 2. Insist on testing early, even with paper prototypes to demonstrate early, easy, wins, 3. Come up with a “model” (her use of quotes, not mine) that can be communicated and understood by developers in order to get buy in, and 4. Put user experience at the centre of the studio, as mentioned earlier it is up to everyone, not just the user research team.
After that it was time to get back on the buses, head back to SF, and to party until late with the wonderful user research community. Good times were had.
Day Three – Wednesday – Get to talks 30 minutes early or you miss out
I really wish my iPad had been playing nice on this day, because on day three I made it into several extremely packed talks (I turned out 30 minutes early, but many others were turned away). These talks were packed because they were interesting, and thus it sucks I don’t have any notes. Anyway, again, these descriptions and impressions will again be mostly based on my memory.
Derek delivered a well-paced, laid back, interesting, and humorous (some really great groan worthy puns were made) talk about how the user interface for Hearthstone was created (an on-going process that was carried out over the six years of the games development). This can apparently already be watched online as part of Gamespot's coverage. He shared some really early (and very different looking) screens of the early user interface for the game. However, the main thrust of the talk was about finding the “seed” for the interface. The idea that everything flowed from. In the case of Hearthstone this seed was the idea that the game was played and held inside the “box”. This seed took about a year to come up with and many other decisions flowed on from this. For example, the question “well, where is the box then?” lead to the idea of the game being played in an Inn, in turn leading to the introduction UI for the game and other associated “Inn like” touches.
Furthermore, it lead to a commitment to a “physical” design for the game and the UI. Interestingly, this was not just aesthetic, it also impacted on the mechanics of the game. For instance, the fact that there is a played card limit in the game arose from not wanting to shrink down the cards to make room for more – yet that limit also forces players to consider their moves and restricts strategy in interesting ways. Similarly, the deck size used to be larger (60), but that wasn't very workable in the UI. So it was reduced.
That is not to say that aesthetics aren't also important for the experience. In fact, Derek shared that his child’s idea of “playing” Hearthstone is to get him to load up the Gnome Workshop level and launch the interactive rocket – a “physical” extra element added to the game to serve this UI principle, but that has no mechanic impact on the game itself. It is there simply to serve the player experience and the “seed”. All of this said, Derek did admit at times their desire to stick to a physical design had caused some usability issues with multiple steps being needed to perform some common actions that could be done faster. However, he also stated that this would be addressed while staying true to the “seed”.
Clash of Clans: Designing Games That People Will Play For Years by Jonas Collaros from Supercell.
Another packed out session had Jonas, a programmer on Clash of Clans laying out the reason for their success. To start out Jonas clearly stated that they had been lucky. He then moved on to discuss the other factor that Supercell believe is important, their commitment to focusing on the player experience and designing for long-term play. To do so he outlined the history of Clash of Clans development, from the 5 person initial team to the 13 (yes, 13!) people that currently maintain the game. Over, and over, Jonas clearly st