The Miracle of Performance Modeling

What is…

Performance modeling is the process of synthesizing the stylistic elements that make up important human nuances in music for application in electronic music. Without these elements, an electronic music performance can sound contrived and even unpleasant. Performance modeling works by artificially rebuilding performance characteristics and reintroducing them into electronic music performances. The goal is to craft a realistic and believable sounding piece of music.

Electronic musicians employ software tools, recorded MIDI data, various synthesis techniques and an understanding of orchestration to manipulate various performance parameters. By modifying the various parameters associated with a musical performance an electronic music producer can greatly enhance the quality of their work. It isn't an easy task but thanks to advances in performance modeling tools technologies things are getting easier.

Performance parameters are the elements of an instrumental or vocal performance that musicians and engineers manipulate to introduce expressiveness and variation into their work. Some general performance parameters common to all instruments and performances include pitch (note number), timing (note time), overall volume (controller #7 - volume), instrument volume (controller #10 - expression), velocity and panning.

The more subtle and more often ignored performance parameters specific to particular instruments or family of instruments include chord note arpeggiation on guitar, breath pressure on a brass or woodwind instrument, or use of various pedals on keyboard instruments. There are also side effects that result from normal mechanical use of an instrument. For instance, using different microphone placement techniques, recordists can control the amount of pick and fret noise captured during a guitar performance. These cues add greatly to a performance's perceived realism.

Consider a strummed string instrument performance. Note articulation for chords played on a strummed string instrument differ from those performed on a keyboard instrument. In most cases a synthesized strummed guitar part played on a keyboard will not sound authentic when compared to a strummed guitar part that is played on a guitar style controller. It's true that you could invest in a MIDI guitar to assist you in developing truly convincing synthesized guitar performances. But what if you can't afford a MIDI at the moment? What if you don't play guitar? What happens if your primary instrument is percussion? What if you aren't an instrumentalist at all?

Now I know that all game audio pros are virtuosos on all instruments and have infinite budgets, but for argument sakes lets assume for a moment that MIDI guitar isn't an option for you. You can step record or worse draw the notes in by hand. You are inevitably going to spend hours tweaking note velocities and positions to get a semi realistic sound. This method isn't practical. It is time consuming, unintuitive and ultimately frustrating.

Hardware and software performance modeling tools offer electronic musicians an alternative to the tedium of manual editing. Their interfaces are more precise than random generation methods and more intuitive than slaving note after note with a mouse. In the body of this article we will view those various options. But first lets talk about the role of MIDI in all of this.

Why MIDI Still?

The demise of MIDI is one of those things is prophesized quite often. If we dig deep enough into Nostradamus's notes I am sure we will find something about MIDI and doomsday. I have news for the doomsayers. MIDI isn't going anywhere, in fact there have been some pretty big steps forward for MIDI recently with DLS-2 (see below) and XMF. MIDI is a cost efficient manner of delivering musical content in interactive media. MIDI is an indispensable tool for game developers because of its flexibility and small size. Digital audio is just too big for some projects and can get expensive. MIDI also works great as a basis for interactive music composition.

If MIDI has so much to offer then why are some people so anxious to see it go away? The answer is simple. In the wrong hands MIDI sounds really awful. Most lay people don't understand why MIDI can sound so bad. They hear a quantized sequence played through a 500K sound set and their immediate reaction is "Wow that MIDI thing sounds awful! It's not going in my game!" They don't realize the various factors that are involved in creating a realistic sounding piece of sequenced music.

Obviously part of the problem with bad MIDI is a poor quality sound source. This isn't as big of a roadblock as it was a decade ago thanks to drastic improvements in sample based synthesis and sound synthesis in general. This is especially true today with hardware sequences sporting 512Mb of sample RAM and software synthesizers allowing sample sets up to about 4Gb! As music memory budgets improve on consoles sequenced music, and more importantly interactive music, are going to level up.

The other deciding factor in the perceived realism of a sequence relates to the expressiveness of the performance itself. Step editing in a performance and running a humanizing function usually falls short of perfection in terms of creating a convincing performance. This method usually takes way too long to get the desired result, if that is possible at all. Professional electronic music composers need robust tools that allow them to model performance data in intuitive ways.

Hardware manufacturers have provided a number of useful and often times esoteric pieces of gear to help acoustic musicians translate their performances into digital data. Thankfully a number of software manufacturers have begun developing software that allows non-instrumentalists to model performance data using GUIs. Using these various tools electronic musicians gain new levels of control over the various performance parameters involved in music production.

Sound Synthesis

Sound generation methods for electronic music have come a long way since their inception. Both modeling and sample based sound generation give musicians flexible means for sound design and musical performance. The quality of a production's sound sources and the eloquence of their designs play a role of obvious importance in delivering professional level electronic music for interactive media.

There are plenty of professional grade multisamples and modeling synths on the market that can be manipulated to display truly expressive responses during musical performances. Hard disc streaming of samples allows for massive amounts of sample data and therefore plenty of opportunities to add additional performance modeling material. Physical modeling synths are great for reproducing the realistic aspects of an instrument's performance characteristics. These aspects are often capable of being modulated in real time as well making physical modeling a truly expressive and powerful sound generation method.

Unfortunately the benefits of large professional samples and powerful modeling synthesizers are beyond the scope of most interactive media productions. Audio being what it is in interactive media, we don't always have the resources or the tools to be picky with how our message based music gets reproduced. Modeling technologies are generally too tasking on CPUs to justify use in music production. Sample playback is often limited to sometimes obscene file sizes, and will be until a method that allows for larger samples to be used for playback (picture GigaSampler style disc streaming on the next-generation of consoles… or PCs for that matter!).

This is especially true for portable and mobile games. You can't fit a 6GB-woodwind bank into a cellular phone. And GBA doesn't have the power to dedicate to cycle-hungry physical modeling synths. Crude waveform generators and small samples are the sound sources we have to construct our masterpieces. That being said (and widely understood), interactive audio pros make due with what they have. Among the things we do have is DLS-2. DLS-2 (Downloadable Sounds Level Two) is an open standard sampling format based on Sound Font 2.0 that many game developers implement into their titles. DLS-2 provides synthesis tools like LFOs, LPFs, a six-stage envelope and the ability to layer voices.

I mention DLS-2 for two reasons. First off there are a number of sound library providers that develop for DLS-2. You can avoid the agonizing process of taking a masterfully recorded set of instruments and squeezing the life out of them so they can all fit into half of your 4Mb sound budget. Secondly, DLS-2 is supported natively in DirectX Audio. DirectMusic Producer has a built-in DLS-2 Sampler that you can use to edit or design your own banks. Remember DMP is free so check that off your budget list.

Without the use of some sampling standard like DLS-2 or proprietary synthesis tools, you won't have a method for molding your raw sample and synth data. For example, a modeled guitar performance triggering a singular recorded guitar note for the entire range of the instrument with no regard for velocity or other performance parameters won't sound very realistic. In fact it will sound quite bad. Compare that to a modeled guitar performance triggering a multisample that includes a layer of fret buzzing and a layer of string noise. This fret buzzing may be introduced when the sequence triggers predefined notes at certain velocity levels. The audio developer may script a behavior for introducing the string noise at random intervals.

These effects are subtle but add greatly to the perceived realism of the musical performance. Where is the power of music and sound design if not in its subtlety after all? You can't do all this without velocity switching, envelopes, algorithmic tools etc. Synthesis tools (filters, oscillators, envelopes, effects etc.) allow us to reproduce performance parameters integral to modeling realistic musical performances. LFOs, filters and the like can model performance nuances such as vibrato, tremolo, dynamics etc. Not only are they essential in producing appropriate samples for music data playback, they can be used for sound design as well (as in the case of cone filtering for achieving directionality effects in DS3D).

There is a greater potential for these and newer synthesis methods in interactive media as music and sound design become more interactive. As interactive music begins to resemble real-time performers, programmable/scriptable real-time synthesis controls will become quite valuable. This is something to consider when setting forth your tool requirements to your audio programmer.

MIDI Controllers

The most efficient and intuitive way to capture a musical performance for sequencing is to simply record a live musician performing on a MIDI controller. Thankfully hardware manufactures caught on to this early and seized the opportunity to provide electronic musicians with a number of useful solutions for recording MIDI data using electronic adaptations of various instruments.

The keyboard is a staple of electronic music production as well as music education and performance. Most synthesizers are built around keyboard controllers. You can buy a keyboard controller without any sound generation mechanism built in for use with external sound modules and samplers. Fatar offers a wide selection of keyboard controllers. Yamaha, Roland and Midiman offer keyboard controllers as well. Keyboards make truly versatile MIDI controllers, with many coming with their own share of additional controller and controller options, including sliders, pitchbend and modulation wheels, knobs and even ribbon controllers. They make for pretty good controllers all around but keep in that a generalized tool can limit your success in producing the most believable performance possible.

Guitarists have two options for capturing MIDI performances. A standard electric or acoustic guitar can be built with internal MIDI electronics or can be retrofitted with a special guitar synth pickup. RMC Pickups manufactures MIDI electronics for a wide range of guitars. The Parker MIDIFly and MIDIAxe from BMC Guitars are two examples of MIDI Guitars. Many guitar manufacturers will custom build a MIDI-ready guitar for you. Conklin Guitars for instance offers an interesting range of custom guitars with MIDI electronics. Yamaha and Roland provide MIDI guitar retrofitting. For more information on all things MIDI Guitar check out http://groups.yahoo.com/group/midiguitar.

MIDI-ready guitars, like this 8-stringer from Conklin Guitars, offer superior performance modeling opportunities for electronic musicians working with strummed guitar parts.

A second option for guitarists is a guitar style controller like the Ztar from Starrlabs. Not quite guitars, but they are in the same vein. They are definitely worth checking out. Guitarists aren't the only string players that can MIDI up their instruments. There are also MIDI basses and violins and even retrofittings for pedal steel guitars.

Wind and brass players have their own set of MIDI instruments available to them. They can invest in a wind controller like the WX5 from Yamaha or one of three models from Akai. Alternately brass players may be interested in hunting down a now discontinued trumpet-style EVI 1000 from Akai. These instruments give wind and brass players unprecedented control over their MIDI performances.

Another option is to use a breath controller in tandem with another MIDI controller. A breath controller like the BC-3A from Yamaha is used to control the virtual breath pressure of a wind or brass performance. The performer plays the notes on the secondary controller and controls the expressiveness of the performance with the breath controller. MIDI supports breath controller data natively with continuous controller #2. Wind players interested in learning more about wind synthesis and performance capture for MIDI should check out the International Wind Synthesis Homepage and the EWI/EVI website.

Triggering sounds and interpreting velocity information is fairly easy to do, so drummers and percussionists have a range of quality instruments available to them. Roland has really pushed the envelope for drum controllers with their V-Drums and V-Cymbal series. They offer a number of electronic drum kits over a range of price points. There are also percussion controllers available from Roland. Yamaha makes electronic drum kits as well. An alternative to the expensive MIDI drum kits are MIDI triggers. These are electronic sensors attached to drums that send MIDI messages when excited.

Aside from the run of the mill MIDI versions of traditional instruments there are alternative electronic instruments that give electronic musicians new methods to express themselves. Some of them are pretty wild, like brainwave to MIDI converters. Others are more practical like XYZ-pads, pedals and knobs. Vocalists are even able to capture aspects of their performances using pitch-to-MIDI converters. There are a lot of options for crafting your performance data before it even hits your sequencer. Check out Starrlabs's page, the Surface One from MidiMan and the ExpressionMate from Kurzweil and anything with a D-beam on it from Roland for some examples.

There are some negative aspects of working with MIDI controllers. First off they aren't for everyone. If you need to record a flute solo and you don't play a woodwind, a wind controller probably isn't'going to do you much good. Second, they can be expensive investments, much more so than software solutions. Third they are generally limited in their capabilities to deliver believable performances across a wide range of instrument families. Lastly, part of the power of MIDI lies in the fact that you can edit performance parameters after the fact. Don't discount the power of editing MIDI data just because you pulled off a pretty decent rendition of your performance.v

Performance Modeling Software

As we are sometimes faced with some serious obstacles to generating useful sound sources our focus should shift to the believability of our sequences. Even when listened through a really bad sample set sequences tweaked with performance modeling techniques can sound pretty good. When listened over a good sample set diligently modeled performances can sound very convincing. By taking the time to model the message-based portion of our electronic music production, we can dramatically increase the overall experience of the listener/player.

Performance modeling software tools allow for the generation and/or manipulation of large amounts of MIDI data in musically meaningful ways. These tools can do everything from act as a dynamic processor on MIDI note data to simulating the particularities of performing on specific instruments to generating musical material themselves. They are relatively inexpensive and can be easily worked into even the smallest audio budget.

Most sequencing packages come with a suite of built in and plug-in MIDI editing tools. These can be used to model performance data. For instance Sonar, a multitrack audio and sequencing software package, features various plug-ins used to edit MIDI data. Most of these can be used to transmute raw note data into the type of performance you desire. For instance the "Velocity" MIDI Effect has a randomization parameter I use extensively in my productions to eliminate the static sound sequenced music is capable of displaying.

These prepackaged tools are flexible in that they are not restricted to one or two particular types of instruments, making them cost effective to include in the sequencing software. On the other hand they are far from precise and are not intuitive when applied in specific situations. This is where specialized software tools come into play. With tools offering dedicated functions for modeling performances for one instrument type you are given more control over instrument-specific performance parameters.

Rhythm n' Chords from MusicLab is a performance modeling tool used for crafting rhythm guitar parts via MIDI. There are VST and Cakewalk versions of the plug-in available for purchase from MusicLab's site. You can get some really incredible results using this tool and in a significantly shortened amount of time when compared to hand editing every note in your guitar chords. I'll never forget the project where I spent over two weeks just trying to get the guitar strumming sounding right, just to find out for a $100 I could have completed it in hours with stellar results.

Take a listen to what I came up with in mere minutes using Rn'C. [MP3]

A new release from MusicLab is Slicy Drummer. It allows you to cook up drum loops and patterns from stock libraries of individual drum parts. The interface gives you lots of control over how the drum patterns are played. I haven't checked it out yet but if it does what Rhythm n' Chords does for guitar parts, it is undoubtedly invaluable. Definitely worth downloading the lite-version and seeing what it can do you for you in terms of drum/percussion performance modeling.

NTONYX specializes in developing performance modeling software. Their tools include plug-ins for Cakewalk sequencers as well as stand alone apps. Two tools of particular interest to interactive audio pros are Style Enhancer and Style Morpher. Style Enhancer allows sequence producers to add human elements to instrument parts by exposing and combining MIDI parameters in useful ways. Style Enhancer comes with a variety of predefined musical and instrument styles with additional downloadable styles available for free on the NTONYX homepage.

Style Enhancer from NTONYX provide electronic music producers with time saving methods for modeling music in various styles.

Style Morpher is a tool near and dear to interactive composer's hearts. Using intelligent music-based algorithms, Style Morpher produces a piece of music based on one of your own in a new style. So if you develop a theme for use in a game and you need to produce jazz, rock and orchestrated versions of it, SM can help you 'port' your music into these different styles quickly and efficiently.

KARMA (Kay Algorithmic Real-time Music Architecture) is a software tool developed by Steven Kay of Karma Lab. KARMA was developed in MAX, a powerful graphical programming language for musical applications. You might recognize the name as belonging to a popular keyboard instrument from Korg. That's because KARMA is in the KARMA so too speak. Korg licensed the KARMA software and stuffed it into a keyboard.

KARMA is… well… rather difficult to explain. It's like an arpeggiator on amphetamines. The system employs GEs or generated effects, which are real time controls used for manipulating performance parameters. Once you get your head around the expansive system you can pull off some pretty intense performance modeling. Heck you can even do things that no one is able to do on an acoustic instrument and all in real-time!

KARMA isn't currently available to the public as software just yet. There are plans to release both MAC and PC versions of the software later this year, though like many highly anticipated software releases the date has been pushed back before. So for those of you so enamored with this sweet technology that you just can't wait… buy the keyboard.

Twiddly Bits is a series of performance modeling tools from Keyfax. Unlike many of its counterparts in the performance modeling arena, the Twiddly Bits series isn't an algorithmic modeling tool. They are actually recorded performances of real musicians playing MIDI instruments. When you hear a Twiddly Bit drum pattern you are hearing the result of a professional drummer working the kit (an electric kit of course, but you still get all the feel that can be crammed into an 8-bit file). You can edit them and manipulate them appropriately for use in your project. This method of performance modeling has no learning curve and is very cost effective.

In Closing…

It's probably a safe bet that we will all be living on Mars feeding space pigeons in our well deserved retirements long before sequenced music will be truly indistinguishable from live human performances (if it ever happens at all). In the meantime, music manufacturers will do their best to provide electronic musicians with the best solutions they can to craft impressive synthesized music and we as composers and engineers should do our best to support and take advantage of their research.

Performance modeling is only one of many considerations that goes into creating great music for interactive media. But when you are dealing with a minute amount of sample resources it becomes the primary one. My suggestion is to look into acquiring some of the above mentioned tools and technologies for use in your next project. I am sure you will be pleasantly surprised by the results.

Resources

International Wind Synthesis Homepage: http://windsynth.org/home.html
Sonar from Cakewalk: http://www.cakewalk.com/
Rhythm n' Chords and Slicy Drummer from Music Lab: http://www.musiclab.com/
Style Enhancer and Style Morpher from NTONYX: http://www.ntonyx.com/
KARMA from Karma Lab: http://www.karmalab.com/
Twidly Bits from Keyfax: http://www.keyfax.com/