Producing Interactive Audio: Thoughts, Tools, and Techniques

Times have changed in the game development industry. All of the stupid money has fled to Internet commerce applications, and companies in the game world are left having to make their way as viable businesses. There is no market tolerance left for "junk," or as it used to be referred to by a producer at Sega some years ago, "library titles." The "B" and "C" titles that filled out your publishing profile aren't viable any more - in fact, they can be downright deadly. Every game made from now on has to have a legitimate shot at becoming a hit; otherwise, it's just not worth making. This philosophy has inevitably trickled down to audio departments, where the focus now is squarely on quality.

A Rant on Interactivity

What exactly does it mean to produce interactive audio? For some time, interactive audio suffered from an identity crisis. The term has come to mean less and less. In my own jaded way, I always imagined the adjective "interactive" modifying a noun such as "media" or "audio." When a person says that they are "doing interactive" in reference to audio, it usually means the individual has stumbled into a job working on a CD-ROM or web site and is desperately trying to figure out how make 8-bit, 22Khz Red Book audio not sound as if it's being played back across two tin cans connected by string. To this person, "interactive audio" simply means sound for nontraditional media: CD-ROMs, console games, kiosks, and web sites. To me, the term means something entirely different.

If you're describing audio as "interactive," you're implying more than just linear playback. Interactive audio should be constructed in such a way that that the user can affect its performance in real time during playback. I'm talking about reactive, responsive audio, coming from audio drivers that are "aware" of what's happening and can respond by changing the music appropriately. Spooling a two-minute pop song in an unchanging, endless loop during a real-time strategy game is not interactive audio. Perhaps the term "audio for interactive media" would be appropriate instead.

Imagine instead audio that is an interwoven part of a 3D, free-roaming world. As you explore the world, the sound smoothly accompanies you, rising as you encounter danger, falling away as you explore in peace. It sounds triumphant when you succeed, and distant and mournful when you fail. And all of this happens with the precision and emotional impact of a great film score. In a user-driven world such as a game, you have no linear timeline by which to synchronize the changes in the music, as you do in a movie. The audio entirely depends upon the unpredictable input of the user. How can you make this work?

The answer lies in the nature of an interactive 3D world and is made possible by new tools and technologies. The 3D game world is open-ended, a database of terrain, objects, animations, behaviors, and their various relationships. Therefore, the music must also become a database of musical ideas, sounds, instruments, and relationships, imbued with awareness of the other objects in the world and programmed with responsive behaviors of its own.

The Rules of Interactive Sound Design

Over the years, I've worked on about 100 titles, 60 or so in a substantive way. I can distill much of what I have learned from this in a short set of rules.

Creating an Adaptive Audio Engine

Armed with the desire for truly interactive audio, we at Crystal Dynamics set out to create our own sound driver for the Sony PlayStation and, perhaps later, for Windows 95. From my notions of interactivity and set of rules for interactive audio, we derived a number of design goals for our driver.

EMPHASIZE MIDI OVER DIGITAL AUDIO. For most of our products, MIDI and custom DLS-like instruments are a better way to go than Red Book or streaming digital audio. Rules 1 and 5 have some implications for Red Book audio. Red Book audio sounds great, but fortunately for Crystal Dynamics, our programmers know how to get the most out of the CD-ROM drive for game play. Therefore, it's not always available for audio. Furthermore, creating interactive sound designs using streamed, branching digital audio segments is limited in many ways, primarily by disk space, seek times, bus bandwidth, and buffer sizes. Red Book, or any kind of linear, streamed digital audio requires a lot of storage space in any situation. But it becomes even more problematic in an adaptive setting where each variation of a given section of music has to be remixed and stored uncompressed. Finally, most consoles (including the PlayStation) save money by using inexpensive (read, slow and not very reliable) CD-ROM drives. Thus, the constant seeking and playing of digital audio tracks is likely to tax the CD-ROM drive to the point of failure. Red Book audio should therefore be reserved for noninteractive sections of the game, such as title screens.

On the other hand, MIDI is small, compact, and easily modifiable on the fly. The PlayStation has its own dedicated sound RAM, and all of the data needed for a level can be loaded in less than a second. Once loaded, the data is out of the way and the CD-ROM returns to its other duties. Furthermore, the PlayStation contains a pretty good sampler. Respectable music and sound effects were created for the Super Nintendo as well, but that platform suffered from limited sound RAM. Fortunately, the PlayStation has almost ten times as much sound RAM, much better sound interpolation (an on-the-fly sample rate conversion technique used to stretch a sample up or down the keyboard from its native pitch), and superior DSP (used for reverb and the like). In my opinion as a confirmed curmudgeon, anyone who says that they can't make high-quality MIDI music on the PlayStation under these conditions is just whining about the amount of work involved in such an endeavor.

KEEP SOUND DRIVERS EFFICIENT. When we considered replacing the existing sound driver with our own technology, we decided that our code would need to be faster, smaller, easier to implement, and more capable than the code we would be replacing. Otherwise, the project was not worth undertaking. Rule 1 dictates that sound drivers must be small and fast, since both system RAM (where the driver resides) and CPU time are scarce commodities in a fully rendered 3D world. Making sound drivers easy to use is important; programmer and game designer time are limited commodities, so making basic implementation easier leaves more time for these folks to work on making the world ready for your interactive audio.

There should also be a simple, consistent means for your game to communicate relevant information about itself to the sound driver. Adding interactive sound capabilities requires programmers and designers to spend more time communicating information to the sound driver about the state of the world and the characters within it. At Crystal Dynamics, we tried to remedy this situation by communicating the state of the world to the sound driver in the form of a set of simple, numerically registered variables. Most often, we use values from 0 to 127 so that they could be set from standard 7-bit MIDI controllers. Thus, the number of enemies alive on the screen might be represented as one 7-bit variable. Your distance from the level's exit might be stored in another. We have tried to use these same variables throughout the game so that they only need to be coded once.

CODE OR DIE. It's important to put the logic programming in the hands of the sound designer, not the game programmer. Rule 8 clearly shows the logic behind this. It's hard enough to explain which aspects of the world you need to track. It's almost impossible (and I think unreasonable) to expect the game programmers to write code to mute and unmute specific MIDI channels when various condition arise. To solve this problem, we created (with some help from Jim Wright at IBM's Watson research lab) a programming language that allows us to author logical commands within a stock MIDI sequencing environment and store them within a standard MIDI file. The language contains a set of fairly simple Boolean functions (if then, else, endif), navigational commands (goto, label, loop, loop end), a set of data manipulation commands (get, set, sweep), and parameter controls (channel volume, pan, transpose, and so on). Next, we created an auditioning tool that allowed us to simulate the run-time game environment, kick out logic-enhanced sequences, manipulate the game state variables, send commands, and see what happens.

Case Study: The GEX Project

Most of my work this year has centered around the interactive sound design for our upcoming PC and PlayStation title, Gex: Enter the Gecko. (The PlayStation version will be the first title to use our new sound driver.) This game is the second installment of the Gex franchise and has undergone a complete technological overhaul by some of the best programmers and designers in the business. The game will include at least eight worlds thematically based upon TV and movie parodies, as well as secret, bonus, and boss levels - fertile ground for a sound designer indeed.

I'll talk briefly about what we're doing currently in the game's audio. For clarity's sake, I've focused on the treatment of a variable called numberofenemies and a small number of related variables. These few examples are by no means all that we plan to do in the game, but they illustrate my main points.

One aspect of the new Gex title is the game's fast and efficient engine. It allows us to have more enemies on screen moving at a faster rate. Since this is one of the features that really makes the product stand out, we on the audio team designed the interactive audio to work tightly with the new engine functionality. We set up a 7-bit variable called numberofenemies that reflects the number of enemies on the screen and is updated by the game continually. This variable is read and used by the sound driver to adjust the game audio.

Here's a breakdown of the Gex audio, by game level:

THE PRIMAL ZONE. The control track of the MIDI sequence has logic built into it. When the numberofenemies register is 0 and kills is 1, the sequence is paused, and playback of all tracks is branched to a short stinger of the Gex theme song orchestrated with the instrument palette of the Primal Zone level. When the stinger is over, the main sequence is resumed.

We keep a location register that allows the sound driver to see if Gex has entered a new area of the map that has different music. The control track checks every four beats to see if the value of location has changed. If it has, a short transition drum fill plays, and then the MIDI sequence that is matched with value of location is started on the beat.

SCREAM TV. Scream TV is the "horror" level of Enter the Gecko. For this level, I used eerie chamber music laid over some ambient loops of slow ragged breathing and strange heartbeats. The same numberofenemies register is used in this level, but instead of muting and unmuting, the value of the register affects the harmonic center of the piece. For this type of transformation, we use a technique called "pitch mapping." A pitch map in our system is a mechanism to remap the pitches of individual notes as they come from the MIDI sequence on their way to the physical voice. Our pitch maps are built so that the transformation can be kept within the prevailing harmony of the piece. In other words, for a scary piece such as this, the pitch table will confine all of the note remappings to a diminished scale. The numberofenemies value is used continuously to set the number of scale degrees upwards to transpose the harmonic instruments. This creates a very smooth, subtle, intensifying effect as more and more enemies are on the screen and danger is increasing.

CIRCUIT CENTRAL. The Circuit Central level looks like the inside of a CPU. The opening music is weird, ambient analog electronica with a slow, trip-hop beat. After some exploring, it becomes clear to the player that there are jumps and expanses that cannot be passed in Gex's normal state. Within this time, the player should also discover a number of chargers set out in various locations. When Gex steps into a charger, he starts to glow in a green light with orbiting electrons. This state lasts for 15 seconds and gives Gex the ability to use embedded "chips" in the floor to do super jumps and turn on "data bridges" that span the formerly uncrossable chasms.

The state of being charged up is stored as a 1 in a chargedup register, and location keeps track of the seven different areas within the main level. When the control track in the main sequence sees a 1 in chargedup, it checks location and matches the area to one of seven different 15-second-long high-energy pieces of audio.

While most often the game affects the sound design, this relationship can also work the other way. In Circuit Central, we set a register at every beat of the music. Since each measure has four beats, I set the beats variable with a 1 on the down beat followed by 2, 3, and 4 on the remaining three beats and then cycle back to 1. This timing information will synchronize the lighting effects in this world.

Only the Beginning

Adaptive audio is a very new field. Many challenges lie ahead, but I firmly believe that it represents the future of sound in interactive media. To move forward, we need a major paradigm shift in how we think about music, great tools, new technology, and a healthy dose of realism. I hope that my anecdotes, rants, and factoids have shed some light on and sparked more interest in creating interactive audio. So long for now from audio central at Crystal D.