For many, the creation of games can be a way to share important personal experiences with others. That's part of Robin Arnott's goal with SoundSelf: The project aims to create a sonic and visual experience using the player's own voice, leading to a mind-body experience where the senses don't feel separate. Hum or chant to SoundSelf and it responds, a fascinating experiment in creating not an intellectual gaming experience, but a physical one. Arnott, a thoughtful, warmly effusive person, has come to games through a personal journey, and his work in the games space expresses that, leading up to the complex, Kickstarter-funded SoundSelf, for which a public prototype is currently available. He previously did the sound design for Antichamber, and gained acclaim for the unnerving Deep Sea, an auditory, sensory-deprivation experience where the player's own breath control is paramount to the experience. He began as a film student at NYU, and found himself quickly growing disenfranchised with what he describes as an "economy of favors" in that space where ego, self-worth and social capital felt like poisonous currency. "At the time I may have fit in [there] well, but as I became more empathetic... and somebody who wanted to do more good, I think I became conscious of it."

From avant-garde theater to experimental games

Over the course of traveling, taking a break from school and making new friends every day, Arnott says his relationship to film changed, and led him to explore other things to do with his background in sound. One of his first classes on his return to school covered avant-garde theater, and while he says he didn't do well in the class at the time, he was struck by the way progressive art engaged with the audience -- "stuff that was being practiced in an even more interesting way, in my opinion, in video games. Though at the time he hardly played video games, he loved the way games prompted players to act, and how choosing not to act was in itself a decision. He set to work playing all kinds of games, from BioShock to Shadow of the Colossus, and took a game design class at NYU's Game Center during the program's very first year. At first, he wanted to do sound design for AAA games, admiring the invisible role that sound played in creating immersion. But the experimental prototype he designed during his game course would become Deep Sea, which was featured in the university's first No Quarter exhibition.

Learning from Deep Sea

The reaction surprised him, although he says it took years of tweaking and watching people play the game, which requires a wearable mask, to figure out what worked about it. "I think what's really cool about Deep Sea is the way it mixes up your sensory signals; the way it's actually depriving you of oxygen has a really dramatic impact on the gameplay experience," he says. "...as a story or even as a game it 'fails', it's not an interesting mechanical experience, it's not even that mysterious a storytelling experience, but it's a powerful experience, because of the way it blinds you so that it forces you to rely on a sense you're not used to relying on." "I became really interested in mechanisms for effecting experience outside of the constructed rules of what a game experience can be," he reflects.

Journey into the desert

His journey away from what felt like a narrow and complicated film culture and further into the game culture continued in tandem with personal explorations into meditation, philosophy and the idea that the mind and the body aren't separate. "I became really interested in reaching mental experience through body experience, and what kind of mental experience you can have from a body experience, and how that differs from a video game or a movie that is trying to approach your mental experience more directly." In 2011, Arnott created a Kinect project called Synapse for the Burning Man festival; the game let players control a music mix through body movements, and he found attendees had incredibly moving, experiences with it ("I planned to have vibrators installed on the floor so people could have a sexual physical experience along with the whole ecstatic physical experience, but it never worked out," he reflects). SoundSelf follows on from that, he says, emerging from Arnott's growing connection to a community of Burning Man enthusiasts in his current home of Austin, people interested in exploring mindful connection to themselves and others. One practice he began participating with is a group Om, where hundreds of people sit and chant together. "What's amazing about it is how the group sound you create together feels when you're chanting... you don't feel separate from that sound, and in many ways, you're not," he says. That kind of intimate experience, combined with explorations assisted by hallucinogens, can meaningfully shift one's experience of the universe, Arnott says. As a result, he's more interested in "creating game experiences that operate in the space between where physical experience is clearly physical, and mental experience is clearly mental," and sharing that pursuit of experimentation and synesthesia with others is a key goal for SoundSelf.

Experiments in sound

The development of the game itself has been an experiment, with some theoretical elements, like binaural beats. But Arnott's study of our natural physical response to certain sounds began with IGF Technical Excellence award-winning Antichamber: Arnott says he chose to populate Antichamber plentifully with bird sounds, because the song of birds creates a primal physical safety response in humans -- it means there's no predator nearby. Deep Sea features bass rumbles: "I don't know the [evolutionary] reason, but when you hear a heavy bass rumble, you become uneasy and you want to leave that space," Arnott says. "There are lots of these kinds of shortcuts into your mental experience... in media we use them sometimes, but I think in the service of something else, and I'm interested in using them in service of themselves, of catalyzing a mental experience that is out of the ordinary." With SoundSelf, Arnott took inspiration experimental musician Sxip Shirey, whose work he long admired before Shirey was struck by Deep Sea. The first seed of the idea was matching the sound to the player's own voice, and then synchronize with the visual experience, so that ideally the visual and sonic experiences aren't able to be separated. The logistics of that goal have been the domain of programmer Evan Balster, who's already created "incredible" tools early on that should continue to lead the prototype forward to something that works exactly how Arnott hopes. Currently the game can detect tone and duration of the player's sounds, but the goal is to have SoundSelf respond to details like vocal texture and breath rhythm so that the experience is unique to each player. There's already a system underway to address that, he says. Further, "we need to dramatically change the structure of the experience so that it takes you on a real adventure -- not in the way Journey is an adventure, or any video game with avatars, but that feels like it has kind of an arc to it," says Arnott. "That's going to be a tremendous amount of work.

Adding 3D with Oculus Rift

The prototype requires only a PC, headphones and a mic. Originally Arnott planned on making a tablet or mobile experience -- although the vocal input required by the player might make it less than ideal for on-the-go gaming. But then came the Oculus Rift just half a year ago: "We started talking to them, and we're going to be developing it to support full 3D," Arnott reveals. "It already works incredibly well on just a 2D screen with headphones or a microphone and speakers, but I think combining the Rift, which is a new way of fully enveloping a person, with this experience... is going to be something out of this world. I'm really looking forward to it."

Sharing oneself with others through games

Amid all the excitement, Arnott feels the tension between aiming to share something of personal, almost spiritual value and worrying about whether people "get it." "I think that's something a lot of game designers might have challenges with, conveying such personal experiences to other people in a way that's inteesting," he reflects. "It's really scary to put something out there that feels like part of me. Not only that, but mind-body experience stuff and spirituality... is something I feel like I'm growing a lot in, but I don't know exactly where I stand." "I'm afraid of coming off as kind of 'woo-woo'," he admits. In a way, pushing forward to develop SoundSelf into a fully-fledged interactive experience that could innovate meaningfully on physical input and emotional response is another important step in Arnott's development journey. But he feels very optimistic: The first time anyone outside of his close friends saw SoundSelf was at the most recent IndieCade annex: "I was terrified," he admits. "I was keeping an eye on the space, trying not to be invasive, but watching people." "And they loved it," he says. "People loved it."