VRDC Speaker Q&A: John Austin on building MR experiences in the real world

Got questions about what mixed-reality experienes are, and how you can design them to complement the real world?

You should come to VRDC Fall 2017 next month, because John Austin (a designer and programmer at A Stranger Gravity) will be there to present his talk Grounded Mixed Reality: Avoiding the Sticker Effect, which will discuss the creation of "grounded" mixed reality experience.

Here, Austin gives us some information about himself and his work.

Attend VRDC Fall 2017 to learn about immersive games & entertainment, brand experiences, and innovative use cases across industries.

Tell us about yourself and your work in VR/MR

I currently lead the San Francisco-based studio, A Stranger Gravity, making independent games and interactive experiences (previously Gathering Sky). We’ve been working deep into MR/VR over the past year, for quite a variety of platforms, including Google and Occipital. My background is largely technical, though I’ve been building/designing games for quite a while. Previously I worked on a Computer Vision team at Google build machine learning-driven AR experiences for mobile, and my experience there made it a natural fit for our studio to focus on this area.

Without spoiling it too much, tell us what you'll be talking about at VRDC

The talk will be half technical, half philosophical. The standard approach to MR is to try to embed virtual objects into the real world, often with total ignorance to the context of the world the objects are being placed in. My argument is that MR can’t succeed along this route. MR is fundamentally an “extra effort” medium, and if it doesn’t deliver something unique, it won’t last beyond the gimmick. When we look towards the experiences that only MR can deliver, the environmental context is key.

In games, this might mean using the geometry of your room as a puzzle mechanic, making each room you walk into a new experience. For consumer apps, this means delivering information based on the world around you as a query itself. Notably, this means we should not be using MR if it’s simpler and faster to just use Google. Rather, we should make sure what we’re building has a specific reason to be in MR.

What excites you most about VR/MR?

We spend our entire lives dealing with a barrier between the digital world and real life. MR is the realization of the dream of removing that barrier and letting the two cross-pollinate. I think this can only lead to better, more comfortable, and more intuitive ways of interacting with and bringing technology into our lives.

What do you think is the biggest challenge to realizing VR/MR’s potential?

The fundamental long-term challenge of AR right now is technical. Understanding the context and the environment of the world around you is an extraordinarily difficult task. Additionally, the form-factor constraints of MR require this understanding to be done on a computer that fits in your pocket, all without completely draining your battery. We’re only just now starting to solve these challenges with technologies like deep learning, and it’s moving at an incredible pace.

The biggest challenge in the short-term is simply whether MR will stick around until we get there. We need to spend less time pitching the dream, and more time thinking about how to build useful and beautiful experiences with what we have now. Otherwise people will start to see MR simply as a gimmick.

For those of us that are unfamiliar, what is the "sticker effect" when creating an experience for MR?

At its simplest, this is an effect in which virtual objects, rather than looking like an extension of the real world, seem to sit on top of the world, like a sticker. This happens from both technical issues (lack of ambient occlusion, lack of global illumination, etc) and creative causes (using game assets not built to look great in the varied color environments of color feeds).

More fundamentally, the sticker effect is a symptom of designing first for virtual and then attempting to copy this into the real world. In order to combat this effect, we must work in the other direction: starting from reality and extending it piece by piece, into the virtual. This strategy keeps things “grounded”, and keeps the feeling that MR is a true extension of the world.

What would you say is the hardest part of designing for MR/AR? Is there any advice you would give to other studios that strive to create "grounded" experience in MR?

The most difficult part of designing of MR/AR (and consequently the most fun) is designing experiences to take place inside of arbitrary environments. In some ways, the architecture in which the game is played is a co-designer. Building generalized mechanics without having the control we’re used to as game developers has been refreshing and at times extraordinarily frustrating. For instance, we’ve been designing a game in which every unique room is a puzzle, simply based on the geometry of the room. As you utilize more of the environment in the puzzle, you give up control, and consequently balancing becomes tricky. In many cases, we’ve had to push for more player-driven, creative experiences, as these don’t have to be quite as strict.

When I initially wrote this talk, the intention behind a “grounded” experience came largely from the visual and technical side: creating experiences that felt solidly embedded in the real world. Recently, though, I’ve thought of a second definition: “grounded” experiences in the sense of creating something “down to earth”. With any new technology there’s always a rush to create wild and interesting experiences, and I do think this is extraordinarily important to innovation. However, I think far more important is for MR developers to build things that provide value. It’s neat to show recipe ingredients hovering on your counter, but realistically people will (and should) just use their phone. When we find the things that can only be done in MR, and do them well, then MR will really take hold.

Register for VRDC Fall 2017 to hear more about designing MR experiences from John and join other creators of amazing, immersive experiences at the premier industry event.

Gamasutra, VRDC, and GDC are sibling organizations under parent UBM Americas