[Blitz Games Studios studio design director John Nash takes you through the process his team devised for coming up with Kinect game interfaces during the development of titles such as Yoostar 2 and Fantastic Pets 2.]

Though we could spend many moons debating what a designer actually does and perhaps even longer debating how we teach designers, one thing is certain: the designer's job is getting harder by the day.

Of course the constantly evolving face of the game development landscape is an issue, as is the polarization of the gaming marketplace, but these problems boil down to a few basic issues. The one I will be focusing on here is the recent proliferation of gaming hardware interfaces and the subsequent challenges that the different pieces of hardware pose for designers.

For 30 years the triumvirate of the mouse, button, and joystick has ruled the roost as the preferred gaming interfaces between us and the video game.

Recent proliferation in the accessibility to gaming platforms, general global gaming demand and first party brinkmanship has meant that new ways to interact with digital entertainment have been developed. The goal of these new interfaces is simple: to reduce the entry barrier for everyone whilst still including traditional gamers.

This contemporary profusion of new interfaces has had a profound effect on gaming. The new interfaces have created a new breed of mechanically simple, often aggressively-priced games that have in turn attracted new groups of players to the world of gaming. This technological chicken-and egg-cycle means that we are now able to interface, control, and design games in a number of novel ways.

The impact on designers from this interface explosion has been huge. Not only do they have to learn to adapt to these new interfaces, but they must also adopt a more pragmatic approach to the entire design process. The market is changing, the gamers are changing, and so must our mode of development.

The good news is that although the challenge is great, there are a few easy-to-adopt approaches to make the creation process simpler and more rewarding.

To illustrate my point, consider the following diagram:

(click for full size)

With the help of my good friend Wikipedia I have selected a few key events in the chronology of gaming interfaces and dropped them on a timeline. Even though the choices I have made are relatively arbitrary, two things are glaringly apparent.

Firstly some interfaces were developed surprisingly early on and secondly, in 2011 there are an awful lot more of them to deal with. Other more esoteric or one-off interfaces have come and gone, and contemporary interfaces like Kinect blend several elements to create a new interface.

Motion Control and Natural User Interfaces

In a bid to avoid a "geek-speak" fueled war of semantics, I thought I'd define a few interface types to make sure we're all on the same page (at least for the duration of this article!) Broadly speaking, it's useful to think of the new interfaces (in the sense that they have become part of the core gaming landscape, not necessarily brand new technologies) in three categories:

Why This Subject and Why Now?

The main trigger for this article and its rather sensationalist title is the Microsoft Kinect sensor. With its innovative combination of hardware sensors and software processing, Kinect is the first truly mass-market NUI for the global video game market. Blitz Games Studios are one of the world's leading Kinect developers, having been developing games and technology demonstrations on this platform for nearly two years.

Throughout that time we have developed new ways of thinking in terms of game design, a healthy respect for the pitfalls and a love for the experiential space of possibility that the Kinect offers.

Lost in Interpretation

The challenges of developing a game based on a NUI stem from one simple concept -- the player approaches the game with an uncommon interface, their body, to interact with a game where the designer has to simultaneously channel the player's expectations and actions whilst accommodating 80 percent of their collective self-expressive interactions.

Put simply, people approach a NUI game with a wide range of expectations and abilities and the designer has to deliver a compelling game experience without the certainty of intent and leveling of a conventional joypad.

A Matter of Confidence

Confidence is a great word where NUIs are concerned and it has two very distinct meanings that occupy the efforts of the designer.

The first is the designer's confidence in predicting what 80 percent of players will do at any given point in the game. By using controls that are intuitive, working with the expectations of the average player and channeling the player towards specific interactions with good instructions and subtle multi-sensory manipulation, the risk of player frustration is significantly reduced.

The second meaning is the confidence in the engine (and tangentially the designer) that the interactions enacted by the player are being unequivocally understood.

The success of a NUI lies in delivering confidence in both these instances, whilst also delivering the intended game experience without interruption. If a game expects something unusual out of the rules or fiction of the game, or indeed fails to detect a valid input, the illusion fails instantaneously.

The Kinect Effect

Kinect is the first truly mass market NUI for video games. It provides six major data streams to the designer, which provides a massive space of possibility for creating new game experiences.

If you also consider that these streams are programmable, then potential upgrades via clever team-side custom code and official software updates will ensure an exciting future.

Consider Data Streams

In simple terms, the Kinect delivers a number of streams of data, and the combination of these streams defines the type of experience that the player will have. The range of possibilities is huge, but so is the amount of potential misinterpretation at every stage. There are six primary streams:

Throwing Example

Let's jump right in with a deceptively simple example: throwing a ball for a dog using Microsoft's Kinect. We experienced these issues during the development of Fantastic Pets 2. There are three main challenges here: the player understanding that they have to throw the ball, the detection of the throwing style of the player, and the release point of the ball.

Assuming that the game is named appropriately, in this case something catchy like 'Fetch', that the dog is eagerly waiting in front of you, and that an avatar has demonstrated the throw on-screen to the player, we can safely assume the player is primed for some top quality throwing action.

The biggest paradigm shift for design is centered on point two -- detecting the throw. Essentially the designer has to construct a set of rules that will detect most peoples' throwing style with a high degree of confidence. By defining a rule set based on combinations of joint positions, angles, vectors and velocities, it is possible to detect the throwing of a ball irrespective of the actual throwing technique.

Finally, the release point of the ball has to occur without the use of a traditional button press. In these situations, the use of predictive confidence (discussed below) is the way forward.

Iconography

The communication of the game, and the subsequent interpretation of the player of that communication, both rely heavily on the effectiveness of iconography. Traditionally we are used to selecting graphical representations for a game and using them until release.

The effectiveness of the iconographical mechanisms can and should be tested exhaustively with a usability program. It is no exaggeration to say that the use of one graphic over another can save untold numbers of problems. Truly, a picture can be worth a thousand words!

Iconography Example 1

For a demo featuring a driving mechanic, we tried a number of techniques to instruct the player how to steer a car. Everything from onscreen animated arrows to speech was attempted. Trying to get the player to understand how they should extend their arms and steer in mid-air seemed to be a tricky problem.

One of the artists had an idea and simply placed a graphic of a steering wheel onscreen. In tests, this worked perfectly; clearly, using the right image to trigger real-world actions is the most compelling way to interact with a NUI.

Iconography Example 2

During the development of Yoostar 2 the usability testing spawned a lot of very useful information. On the first screen we had a slider button to start the whole experience. The design went through several iterations until we settled on the single 'hover' button solution.

Evolution of the START button

The development of this button was driven by user feedback from the usability testing. To illustrate the kind of feedback we were working from, here are a couple of excerpts from the usability reports:

“How to interact with the 'handles and rails' UI component is not intuitive. Visually, it resembles a VCR-style play button with corresponding descriptive text. As a result, most users tried to push or grab the button as there is no indication that it should slide.”

“Many users hovered over the handle, then began the swipe action to select their choice.

However if they didn't move in a horizontal line, it was common for the action to be cancelled, as the cursor had moved too far in the vertical direction. Typically, users gesture in a diagonal motion, not horizontal.”

“Some users had difficulty staying within a handle. The 'gravity well' effect worked well in snapping them to the handle, but it was sometimes not strong enough to keep them contained within it.”

Speech Control

One of the jewels in the crown of NUI systems is the ability to use voice control. Whether it's phonetics-based or waveform analysis, there is no doubt that speech recognition is the area that requires the most investment in terms of design and technology as compared with the other sensors in Kinect.

The key to success here is twofold: firstly make sure that all active words or phrases are very different to one another, and then ensure that the system is only trying to identify a limited number of words or phrases. Too many, or non-dissimilar, words and phrases always results in much lower recognition confidence that will inevitably lead to recognition errors. In terms of the player's experience, remembering the phrases, saying them at the right time and accent issues will all complicate matters further.

Gestures

Gestures are a cornerstone of NUI interaction. Again, restricting the number of gestures that can be recognized at any one time is the best approach. From user testing it is also apparent that players can only remember a certain number of gesture shapes.

These shapes need to be regularly repeated by the player, or at least there must be a non-invasive memory jogger system such as environment shapes or tracing shapes. Additionally, gestures often work best in games where there is a good context for them, for example a wizard game where you are casting gesture "spells".

Front-end navigation is also a very obvious and viable use of gestural control. As more titles are released, some control systems are clearly more "usable" by a greater part of the gaming population than others. Development guidance started with handles and rails but seems to be polarising between the pointers and 'hover' buttons system, and the arm sweep menus and swipe system. Interestingly, approximately 20 percent of players seem to have difficulty with the best gesture systems indicating that there is still much to learn in this area.

Second Order Design

So we know about the variety of sensor streams and the fact that the players can do pretty much whatever they want in front of the sensors. What exactly then does NUI design need over and above regular joypad game design?

To be honest, I'm struggling to find a term. In essence it's a true second order design process much like emergent design. There are relatively few basic components but the range of possibilities between player interpretation and system confidence means that the designer must employ some sophisticated techniques to reduce error and confusion.

By combining clear instructions with solid iconography and continual positive feedback, the designer should, and must, be able to channel the player towards an expected interaction that produces a satisfying experience. As the days of pressing X to jump draw slowly to a close, so the days of perceptive design beckon us into a new era.

Predictive Confidence

For situations that require the game to react without user confirmation, it is necessary for the game to predict that the player has completed the action. With the ball-throwing example, there is currently no real way to determine the point of the ball's release.

In this case, if the game has detected a throw, it is reasonable that the player will complete the throw; therefore the game should release the ball at the expected point without player confirmation of the release.

This system is also useful when driving pre-set animations, as opposed to using avateering. If the game has given good instructions as to what it expects the player to do and is waiting for a small set of actions, it is possible for the game to partially confirm player intent before triggering a pre-canned animation.

The key to making sure that this system is transparent to the player lies in listening to the feedback from the players during development; a heavy-handed system or one that suffers marked latency will shatter the illusion very easily.

Focus Testing vs. Usability Testing vs. Analytics

Internally at Blitz Games Studios we use a number of systems to make sure our design and production processes are player focused. These systems have distinct functions as follows:

Whatever these rather evocative phrases conjure up in your head, they mean only one thing: listen to your players. If you're wondering exactly when you should be listening to your players, the simple answer is always. Focus test a feature set with players before you implement those features, then prototype them and let the players give feedback on those features in usability testing.

If you're really smart, you'll have an analytics or metrics engine running in the background feeding your dev team with constructive feedback 24/7. Seek out feedback, listen carefully to it and plug the outcomes into the next iteration.

Takeaways

The Player Becomes the Producer

A contentious title for sure, but why does it resonate so well now? NUI systems like the Microsoft Kinect have opened up a cyclone of creativity that will continue to sweep through the landscape of game development. The demand for these new gaming experiences will drive developers to seek new genres and in doing so they will have to rewrite some of the rules of game design.

NUI games are a positive because they remove more barriers preventing players from engaging in our experiences. The flipside is that as developers we find ourselves facing ever greater challenges of coping with the random inputs and expectations of a very large and varied game-playing population.

It is clear that as these interfaces become ever more complex and intuitive, so our game designs and processes must become more flexible and able to adapt. Iterative design based on a backbone of usability testing is the only sure way forward.

The age of testing games just before release is gone; in fact the idea of only exposing our creations to the players at the end of the development process now seems positively ridiculous. Social games and NUI games have proven the validity of regular player feedback and in this way the player has indeed become the producer.

As a boy I dreamed of owning a jetpack but sadly, like many other dreams, that dream was dashed on the rocks of slow mechanical technological advance. So I'm now focused on a new dream; now I want my own holodeck. For me, NUIs like Kinect are the next logical step on a line between the potentiometer and the holodeck itself. Being an impatient designer, I want to do as much as I can to bring that day closer.