The Holodeck may be possible, but it's going to need game designers

Science fiction for many is mere fantasy and escapism, but Lee Sheldon—a writer and producer on Star Trek: The Next Generation and now an associate professor at Rensselaer Polytechnic Institute—sees the fantasy of Star Trek’s Holodeck as entirely possible.

With a seed grant, he and his team are working towards creating the Emergent Reality Lab, using modern technologies to create a “first-generation Holodeck.”

Alternate reality simulations have long been created for military and medical uses, Sheldon said at a talk at the Montreal International Games Summit, admitting that his team are “not the first ones to do this.”

He was, however, particularly dismissive of a recent Microsoft patent with similar aims, mocking the patent image’s unrealistic living space—“If Microsoft wants the patent on that couch, they’re welcome to it.”

For attendees who were skeptical about the possibilities of a Holodeck using modern technologies, Sheldon summarised recent individual progressions in virtual reality, such as the “Cyberwalk” treadmill that lets the friction of the users feet move it allowing them to “walk” while stationary (“I love this—one of the problems with it is however that it costs ten million dollars”) and the Montreal-local McGill University’s haptic floor tiles.

Such technologies when combined, could work towards a Holodeck experience, Sheldon argued.

“You have 360 panoramic screens, 3D stereoscopic projection, surround sound, motion tracking interfaces that include everything from Kinect at the low end, intelligent virtual agents, omi-directional treadmill floors, compliant surfaces, adjustable air flow, temperature gradients, and Smell-O-Vision... why can’t we bring back Smell-O-Vision?”

That’s just technology, however. Sheldon appealed to the fact that game designers would add the important factors that would tip the combination into virtual reality: sustained narrative and contextual play.

“The Holodeck was not about the technology,” he said, "but the narrative immersion.”

Although the first Emergent Reality Lab has not yet been completed, Sheldon took the attendees through the lab’s “dress rehearsal,” an eight week pilot with a class of beginner Mandarin students that involved sets, actors and Kinect-based play all within a single classroom.

Four weeks into the class, the classroom was transformed into a Beijing airport, where they were told to expect an RPI representative to greet them and help them through customs, getting a taxi cab and checking into a hotel and other tasks. Instead they found only a crumpled piece of paper implying the representative had disappeared and were left facing native Mandarin speakers playing roles such as a custom agent who would not respond in English.

“This was designed for an hour and fifty minutes of play, and the students completed it in half an hour. From that point on there was emergent play, they started to ask questions to the Mandarin actors relating to the RPI rep’s disappearance,” Sheldon explained.

Other days included a visit to a tea room, which included interaction with a virtual actor via Kinect.

“We served real tea,” Sheldon said. “you have to be careful mixing this space with real furniture and projected furniture, make sure people aren’t putting a cup down on a table that doesn’t exist.”

Sheldon described that his earlier experience designing ARGs helped him in preparation for designing a Holodeck-like experience.

“The real world is totally unforgiving. There’s no patching reality. In an ARG things change from moment to moment; you can’t think of everything and you have to be able to adjust to how your players play the game.”

“Your players are smarter than you,” He continued. “They’ll figure things out faster than you, like the emergent play in our airport. You have to be very careful, you have to be very flexible.”

“Forget linear storytelling, we have to come up with new paradigms for this. We cannot tell stories in the same way to make this thing work; in fact we have to give up some authorial control. But not all of it, give up just enough that the players /think/ they are in control. That’s most important.”