This article is cross posted on my personal blog here.

Feelings cease to be feelings when subjectivity is quantified.

Are games the only medium in which reception can be judged objectively? In this article I take a look at whether player data is an effective way to analyse a gaming experience. As player data becomes more widely available than ever before and metrics driven design dominates more and more of the industry’s mind-share, have we thought deeply enough about what design using player data really means?

Player Data: Unique media of games

Before we can dive into the meat of the questions laid out above, let’s take a little detour into media theory, anthropology and ethnography to help us understand games’ interpretive ancestor’s a little better and answer a more pressing observation about games that is often neglected by the industry; “Why is there data, instead of no data?”

Much has been said about interpreting different forms of media and even more has been said about interpreting the feelings of people who consume that media. The former is, anyway, a subset of the latter; the reviewer (or critic) is always simply interpreting their own feelings towards a piece of work. The anthropological and ethnographic studies that have explored the huge breadth of media throughout the twentieth century have generally agreed that subjectivity is at the heart of individual interpretation. Slowly the nineteenth century idea of the anthropologist and ethnographer’s role as a mute, passive and neutral observer whose probing “intellectually astute” and “keenly sensitive” questions about the nature of the consumer’s state of mind and feelings toward (the consumption) of a text[1] has disappeared. It has been replaced by a reflexive view of the observer. That the observer is not granted any privileged information. The media analyst is themselves a party in the study, whose ideas and biases are an essential part of their exploratory process. More importantly the subject’s perception of the observer fundamentally changes how they answer any question.

All this leads to the conclusion that human’s are staggeringly unreliable when speaking about their own feelings. They will censor and alter how they articulate their feelings based on their own deeper prejudices and ideals. And what’s more, the person analysing their answers will themselves apply their own mental filters to this information causing it to go through the mill of human unreliability twice. Human’s seem incapable of empirically and accurately articulating their own patterns of thought.

From these observations in media theory we can now see why the fact that games can collect objective data about a player’s actions is pretty amazing. Films, Books, Television, Sports, Visual Art, Religion, Sex etc. are all what I have elected to call Vicarious Mediums. The only[2] way to investigate these mediums is by interviewing people about their experiences, or going directly to the source of the medium and asking yourself what you think of it.

Games are part of a much smaller subset of mediums that I’ve decided to call Specific Mediums. These are mediums that can facilitate the collection of objective data about the behaviour of an audience’s interaction with that medium. This functionality is sublimated into the flexible systems that make up the nature of games. In other mediums it might be possible to collect a small amount of data by augmenting said medium but this data is almost always tangential and of limited use[3]. A casual observer might then come to the conclusion then (as I fear much of the industry has) that games are unambiguously superior to other media in their ability to collect objective data about a player’s behaviour. Every moment of a gaming experience can be monitored and tabulated. This information can then be used by the designer to improve the experience, thus leading to better games and a growing of the medium as a whole, simply by virtue of its ability to culture this information.

Data is Death: Why I was wrong

Up until this point I have presented the collection of data in games very positively. What I have said is true to a degree, and it is what I initially thought, but as I started to research the blogs of many designers who use player data as a day to day tool in their design I started to notice some very worrying trends.

Most designers, whose blogs I read, were just as effusive about the capabilities of player data as I have been in the first part of this article. I often saw designers make statements like “Now that we use data from play-testers we no longer need them to fill out forms about how they felt about the experience, or even ask them their opinions”. The general feeling seems to be that data is a replacement for play-testers describing their experience. From this I have identified two assumptions designers have made that are leading to this lamentable view of player data:

2)   The goal of design is to create a game that is “fun”.

These two ideas are toxically reciprocal, each reinforcing the other. The first assumption, stated in the form above, may seem somewhat absurd – there is no way behaviour can ever signify one-to-one the thoughts of a subject. This would require the observer to have access to the subject’s mind to verify this assumption as even remotely true, however, this view of data seems to be the general consensus among designers.

How did this assumption arise? “Fun”. The idea of “fun” has been valorised almost to the point of absurdity in contemporary game design theory. What designers seem to have collectively forgotten is that “fun” is an incredibly ill-defined emotion. The constant flow of books and articles about what “fun” is, has lead to the illusion that the industry as a whole can quantify this extremely abstract idea. The target emotion of games has therefore been painted in objective terms leading to assumption that we can also analyse our player’s feelings in objective terms. If we can analyse our player’s feelings in objective terms then fun must be an objective standard. Phrased like this, the logical circularity of these assumptions becomes obvious.

Often these assumptions would manifest in posts[4] like this:

“Looking at the data for player deaths by level we can see the difficulty curve for our game…”

“This means our game isn't fun. Because I’ve read lots of articles and books on how to design fun games and so I know that only a smooth gradually climbing difficulty curve is fun.”

This sort of logic is then applied to absolutely any piece of data designers receive. “A player spent too long in this location. I think its because they really liked the scenery here, therefore we should put more of that scenery throughout the level.” Or, “A player is dying to many times at this particular point, a stuck player is not having fun, remove this section of the level immediately.” These arbitrary conclusions are often accompanied by sets of bland[5] bar charts, each piece of data taken completely in isolation with little to know cross referencing between data sets. This data in applied indiscriminately to received wisdom about what makes a “fun” game. Many designers seem to fail to realise that there are multiple implications and meanings of the data they see. Often this is forgotten in favour of some immediate arbitrary conclusion that comes to mind when designers look at their data.

“Fun” is not a constructive criterion with which to judge the success of a game. While any emotion is difficult to articulate, “fun” is particularly problematic. Designers should stop asking themselves “Is my game fun?” and start asking “Is my game affective?” i.e. Is my game successfully communicating the emotion I want? As our medium grows, designers are theorising games that explore a range of emotions; longing, sadness, mystery, bliss, peace, wistfulness etc. etc.

Data is completely incapable of articulating these complex emotions. This may seem contradictory to what I have said in the first section of this article, but player testimony is the only way we can truly assess whether these ideas and emotions have been reached. As flawed as player testimony is, behaviour can only ever imply thought and feeling, it never signifies it. The words players give us about how our designs feel, while subjective and necessarily complex, give a much better view of how our games are really experienced. The desire for simplistic, unambiguous design goals needs to be abandoned.

This is not to say that data is useless. Far from it. If we as an industry start to escape the bad habits we have acquired in analysing our player data, it can still be of incredible use and utility and give us access to design patterns and creations that other mediums only dream of. But this data has to be taken holistically, to aid in the improvement of specific moments that are understood in the context of the whole experience. This analysis needs to be triggered by the testimony of a player. The data can suggest interesting questions to ask our players, to tease about pieces of information they would never normally volunteer. Coupled with player testimony data has a very strong position in enhancing design. Data taken on its own quickly becomes meaningless.

There is of course a double meaning in the title of this article “Data is not fun”. Literally; using only data to design games completely robs us of the joy of the design experience. Designing using only data sequesters the industry in an arrogant objectivity. It turns the designer into nothing more than a data analysis machine whose only job is to create more and more compulsive and exploitative systems in aid of producing the “correct” results in player data sets. In this view player’s cease to be alive as do the designers. A design using player data is a design for the dead.

@infiniteoctagon

[1] Here I use the modern concept of a “text” as in Jameson etc. whereby any form of media that is read and interpreted can be constituted as (cultural) text.

[2] This certainly hasn’t stopped large industries (particularly the film industry) from trying to data-ify their audiences feelings. The film industry spends millions on collecting information of the age, gender, interests and socio-economic positions of the people that go to see their films.

[3] For example; it is possible that in a medium like television we could monitor that frequency with which a viewer might look away from the screen with a camera mounted on top of the screen as a way of collecting data on viewing habits, but the limited use of this data quickly becomes apparent.

[4] This is my own re-creation of what these posts usually look like. I’m not naming and shaming.

[5] As an aside, for an industry that specializes in essentially presenting data structures in interesting ways, we could really work on how we do graphs, almost every design blog I read used bar graphs and only bar graphs and only ever presented one set of statistics at a time. It was as if they were allergic to the concept of interestingly presented data.