We live in an age where gaming in the living room is no longer restricted to the joysticks and gamepads of yesteryear. Instead, players can control games by waving a motion-sensitive remote, or by jumping up and down while having their body position tracked by a camera. Perhaps most intriguingly, users can now play games that take advantage of ubiquitous handheld mobile devices such as phones and tablets, which act as second screens in concert with a Smart TV. The ability to simply tap on the screen of a mobile device and switch to a new, fully customized control for individual games played on a TV opens up an immense world of creative possibilities. Using the power of all the sophisticated sensors packed into a modern phone combined with immediate visual feedback from a high-resolution touch sensitive display will allow developers to create entirely new genres of games and bring Smart TV to the forefront of home entertainment.

The TransGaming Perspective

Throughout the history of video games, the mechanism through which players interact with a game has played a crucial role. From arcade systems with basic joysticks or steering wheels, through to today’s ecosystem of motion control, user-tracking cameras, and multi-touch screens, the way that the player physically interacts with the game imparts a visceral connection between the player and the machine. This connection is fundamental in creating emotional response and engagement in the player – a poorly honed control system will leave players frustrated and angry, while a well-built system immerses the player into the game’s environment to the point where the user no longer consciously perceives the physical controller.

At TransGaming, we began thinking seriously about input in games as we were building GameTree™ TV, a platform aimed at providing games through next-generation connected TV and set-top systems. As our development began, we faced fundamental questions about how users would be interacting with these new systems. Original equipment manufacturers (OEM) and cable operators have their own ideas on this front, with practically every form of input hardware considered.  Different vendors are looking at everything from the venerable infrared (IR) remote through to radio frequency (RF) and Bluetooth devices with motion tracking, and multi-screen input through users’ smartphones. Some vendors have even built miniature keyboards into remote devices to enable users to search for content by name.

We quickly realized that we needed to build a mechanism to allow games deployed through GameTree TV to be device agnostic – capable of being controlled in a variety of different ways depending on the available hardware, while still allowing for careful tuning to ensure that input did not become a barrier to the player’s connection to the game.

StreamInput is born

In mid-2010, we decided that it was high time for the development of a truly universal and extensible application programming interface (API) for input – one that would allow games to be developed to support and be easily tuned for multiple different input devices, across multiple platforms.

Up until then, there had been few efforts to try to standardize the ways in which developers handle input in games. Every operating system, from those for game consoles through to PCs, tablets, and mobile devices handled input differently. When new types of input hardware became available – the depth cameras made famous by Microsoft’s Kinect, for example – manufacturers provided their own custom APIs, which seldom had any usefulness outside their specific domain.

From the beginning, we recognized that in order to have a significant impact in the industry, any new input API would need to have widespread support among vendors as well as developers.

The Khronos Group

The Khronos Group (https://www.khronos.org/) was founded in January 2000 as an industry consortium dedicated to creating open standard APIs to enable the authoring and playback of rich media on a wide variety of platforms and devices. Among the critical APIs that Khronos creates and maintains is OpenGL ES, which is used to power 2D and 3D graphics for almost every smart-phone and tablet in the market today, enabling software developers to create high-performance, visually compelling graphics for a wide variety of devices 

In addition to TransGaming, Khronos counts as member companies such leading organizations as Apple, Intel, Nokia, Samsung, Sony, AMD and NVidia, as well as over 100 others. Given that kind of membership list, and a singular focus on creating open standard APIs, we felt that Khronos was absolutely the right umbrella under which to develop a new industry standard for input.

In 2011 the Khronos StreamInput working group was officially launched, with over a dozen other companies interested in fostering the development of a new approach to input quickly joining TransGaming in this effort, including Intel, EA, NVidia, ST Microelectronics, Qualcomm, SoftKinetic, PrimeSense, Freescale, Aptina Imaging, Sensor Platforms, Broadcom, and many others. Many of these companies are at the forefront of development of new sensor technologies, ranging from advanced motion-sensing hardware and software to depth camera systems with algorithms that can track a user’s individual finger movements.

Collectively, our vision with StreamInput is to create a well-defined standard that brings together device manufacturers, platform vendors, and software developers, to create new opportunities for innovation while simplifying the process of creating or customizing software to support new devices.

Semantic Input

The key to StreamInput is the concept of ‘Semantic Input’ – input with meaning. Instead of writing specific code to support this controller or that device, developers instead define the semantics that are meaningful to their application. For example, in a racing game, that semantic might include the ideas of acceleration, braking, and turning the wheel. In a golf game, it might mean a player’s positioning, swing speed, etc. For a word processor, the critical semantic is simply text entry, regardless of whether that text came from a keyboard, voice recognition, handwriting recognition, or from a plug jacked directly into the writer’s brain.

Once the high level semantics are well defined, different ways to generate this input are possible on different devices. For example, in a racing game, there are many ways to represent a turn to the left or right: using a joystick or game controller, tracking the motion of a gyroscope mounted inside a remote control, or even using a camera to watch a user turning an imaginary wheel.

This approach doesn’t mean ‘one size fits all’ – instead, by simplifying developers’ focus on the high level, StreamInput will give developers more time to concentrate on tuning their input systems to individual devices. Furthermore, by setting a reliable standard for how applications can integrate with input devices, StreamInput simplifies the way new input innovations can be brought to market.

Multi-screen gaming

By separating out the high level control in this way, we open the door to the opportunity to disconnect the physical input device from the system completely, and instead, use devices connected over the network, such as smartphones or tablets. In the simplest case, a phone could simply be used as a basic control mechanism for a game, similar to how a gamepad or joystick might work. Unlike a gamepad however, modern touch sensitive screens can enable a phone to serve effectively as a remote track pad device for a pointer, for example.

A further step up from this would be to provide a fully custom application on the mobile phone that takes advantage of not only the phone’s touch screen or local motion control sensors, but combines that input with output from the phone’s own high resolution display and audio output. Using the phone’s own screen directly allows sophisticated secondary feedback to the player to become possible.

Imagine playing a poker game where the dealer’s cards are shown on a large Smart TV screen in the living room.  Your friends’ are holding “cards” that render on their individual handheld devices, as are you.  You peek at your cards and make your bets from your mobile device, while the overall action is tracked on the shared screen of a Smart TV.

This kind of multi-screen approach offers huge potential for creative new ideas that provide engaging and immersive game play combined with the social connection to friends, whether they are local or playing from their own homes.

And beyond

Ultimately, the input-agnostic approach will usher in a distinct new genre of video games and accessories, and bring out new and creative ideas from game developers. With StreamInput, we are defining a new standard that will give new tools to developers that allow them to deliver a richer, more engaging experience for players.

When the time does eventually come that human beings are able to connect their brains directly into a computer to work or play video games, it may well be in part through the efforts now underway to standardize the StreamInput API and the high level semantics for input agnostic computing.

Byline: Gavriel State is the Founder and Chief Technology Officer of TransGaming Inc., and chairs the Khronos StreamInput working group.