Sponsored By

Interactive music sequencer design for games is a real mouthful -- or rather an earful -- and a pretty large topic to boot. Scott Patterson addresses some of the design issues in making a computer music language, and the issues related to providing interactive music functions for game control.

Scott Patterson, Blogger

May 15, 2001

24 Min Read

Interactive music sequencer design for games is a real mouthful -- or rather an earful -- and a pretty large topic to boot. I'll begin with a summary of the motivations for making interactive music and then get in to the design discussion. To make the design discussion a bit more manageable, I will assume a lot of familiarity with music and synthesizer details and put greater focus on design options and programming methods. First, I will talk about the design issues in making a computer music language. Second, I will consider the controls available with audio synthesis. Third, I will add the issues related to providing interactive music functions for game control. Throughout, I will focus on concepts that directly influence interactivity, and point out approaches that make the complexity more manageable and help to make implementation practical.

Motivations for Interactive Music

Most forms of electronic entertainment include music. Games, movies, and TV shows are seen as incomplete without music. We hear music at the introduction to news programs, on our mobile phone rings, in department stores, and at coffee shops.

Music is its own form of entertainment. We listen for styles, attitudes, technology, improvisation, composition, and skilled performances. Our memories associate music with past situations, friends, places. We associate music with love and hate.

Games include music for many reasons: To identify with a particular audience. To establish attitude, tension, and mood. To let them hear the pride and glory of success or shame and ridicule of defeat. To march them off to the drums of war. To take them to a magical place. To set them in the past. To set them in the future. To take them to alternate worlds. To bring them back to reality. Music instantly adds definitions and associations beyond what the visuals can do.

Games are interactive. This means a player has control over the game in some way and the game asks the player to interact in some way. This control and interaction are the basis for how a game becomes immersive and entertaining. The quality of control and interaction are the basis of successful games.

Therefore, it is natural to want to mix the immersive quality of control and interaction in computer games with the immersive qualities of music. How do we control music? How can we create musical interaction? This is the motivation for this article.

The reasons for developing your own interactive music sequencer code are the same as the reasons for any code development. You may want standard features across many platforms. There may not be systems available that meet your needs. You may want an implementation that you can optimize for delivering the particular features you need. You may want control over your own code to provide improvements, enhancements, and reliability in line with internal scheduling requirements.

Making a Computer Music Language

Event Blocks
Music can be described as events or commands occurring over time. We can build our music command language with "event blocks" composed of three elements: time, event type, and event details. The time is stored relative to previous events and is called a delta-time, the event type is identified with a number, and the event details are zero or more parameters that are defined by the event type.

Figure 1 - Event Block

Delta-Time

Event Type

Event Parameters

With this basic design we can describe any groupings of events and time intervals. Building our computer music language is now a task of choosing how to store the delta-time, what event types, and what details are needed for each event type.

MIDI (Musical Instrument Digital Interface) Music
The MIDI specification has been around for a long time. Most composition and sequencing software provides compatibility with MIDI input and output ports, and these products also provide options to save and load MIDI files.

Since the MIDI specification defines a delta-time storage format and a useful set of event types and parameters, we should start with this model as a reference. This will make it easier to convert the MIDI language to our custom music language.

Keep in mind that our goal is to create music directly on a machine with audio capabilities, so we want to take what is useful from the MIDI design, but not restrict ourselves to its features. We are also likely to get music from composers in the MIDI file format, so we will want to convert the MIDI file music language to our own custom language.

MIDI Events
The MIDI specification defines several events called the channel voice messages, and there are also meta-events defined in the MIDI file format 1.0 specification. Some of the meta-events simply contain text string data. I summarize the MIDI events and meta-events we are interested in Table 1, Table 2, and Table 3.

Table 1 - MIDI Channel Voice Message

Event Type

Event Parameters

Note Off

Note Number

Release Velocity

Note On

Note Number

Attack Velocity

Pitch Wheel

Pitch Bend LSB

Pitch Bend MSB

Control Change

Controller ID

Controller Value

Program Change

Program Number

Poly Key Pressure

Note Number

Pressure Value

Channel Pressure

Pressure Value

Table 2 - Basic MIDI Event Types

Meta Event Type

Meta Event Parameters

End of Track

 

Set Tempo

Tempo Number

Time Signature

Numerator

Denominator

MIDI Clocks per Metronome Click

32nd Notes in a MIDI Quarter Note

Key Signature

Sharps/Flats Indicator

Major or Minor Indicator

Table 3 - Text MIDI Event Types

Meta Event Type

Text Event

Sequence/Track Name

Instrument Name

Lyric

Marker

Cue Point

Of the MIDI channel voice messages, the control change event is unique in that it specifies an additional event identifier, the controller ID. Examples of common controller ID numbers are 7, which refers to volume and 10, which refers to pan position. Many controller ID numbers are not commonly used and we can have musicians insert control change messages with controller IDs to mark the music in special ways.

The basic meta-events listed give us ways to get time organization details and tempo changes. The text meta-events give us ways to get custom data embedded in strings. Composers can put special track settings in the track name strings or special kinds of playback commands in lyric strings. These methods will save us from having to write custom tools for composers when deadlines are looming.

MIDI Channel and Tracks
The "channel" in MIDI channel voice messages refers to the fact that these messages are sent with a channel number from 1 to 16 (0 to 15). The channel number is used by MIDI devices to route commands. We are not routing commands to MIDI gear, so we won't talk about channels anymore. Type 1 MIDI files have any number of tracks where each track contains a series of event blocks. It is the track concept that is useful for our data organization.

Sequences, Tracks, Events, Instruments, and Voices
This is where we leave the world of MIDI and talk about our custom music sequencer language. A quick summary of the terminology used in the rest of this paper is in order: A Sequence is a collection of Tracks that run simultaneously. Each Track is a sequence of Events that control the current Instrument. Certain types of Track Events will turn on and off Voices of the current Instrument. Some types of Track Events will modify all active Track Voices. You can think of an Instrument as the default settings given to a Voice that is turned on with a NoteOn Event. I will present a list of class definitions that display this structure in the implementation section.

Event Type Possibilities
Table 4 lists many event type ideas so you can get a taste of the possibilities. There are a lot of details that could be discussed ast to how these event types might be implemented, but I will just mention the options here and move on to the other parts of this discussion.

Table 4 - Event Type Ideas

Event Type

Notes

Basic

These are like traditional MIDI events.

Note Off

Note Number

Release Velocity Optional

Note On

Note Number

Attack Velocity

SetTrackVolume

Value

SetTrackPitchBend

Value

SetTrackPan

Value

SetTrackEffect

Any Effect Type

Value

SetInstrument

Program Change

Track End

Loop to Start if the Sequence is Flagged

 

 

Test Values

We can have an array of test values at the sequence level or track level (the value could have a name and have the string mapped to a lookup number)

Set Test Value

Sequence or Track Test Value

Test Value ID

Value

Inc Test Value

Sequence or Track Test Value

Dec Test Value

Sequence or Track Test Value

DecZ Test Value

Do Not Decrement Below Zero

 

 

Advanced

 

SetSequenceVolume

Value

SetSequenceEffect

Any Effect Type

Value

SetListenerWorldPosition

Sets 3D Pan and Volume

SetTrackWorldPosition

Sets 3D Pan and Volume

SetInstrument If

Sequence or Track Test Value

Position Marker

Marker Name (String Mapped to Number)

Jump to Position Marker

Marker Name (String Mapped to Number)

Jump to Position Marker If

Sequence or Track Test Value

Marker Name (String Mapped to Number)

 

 

State Change

Fades, Ducking, Mute Groups

Change Sequence State

Set immediate state or set a target state and interpolation time

Change Sequence State If

Sequence or Track Test Value

Change Track State

Set immediate state or set a target state and interpolation time

Change Track State If

Sequence or Track Test Value

Change Voice State

Set immediate state or set a target state and interpolation time

Change Voice State If

Sequence or Track Test Value

 

 

Arrangement

 

Jump to Track

Jump to a New Track Data

Jump to Track If

Sequence or Track Test Value

Gosub to Track

Jump to New Track Data, Returns When Done

Gosub to Track If

Sequence or Track Test Value

 

 

Callback

 

Callback

Calls Game Code, Could Change Test Values

Callback If

Sequence or Track Test Value

 

Sequencer Data Structures
The main data structures that we could use for our music sequencer are listed in Code Example 1.

Code Example 1 - Music Sequencer Data Structures

typedef list< Sequence * >  SequencePtrList_t;
typedef list< Track * >        TrackPtrList_t;
typedef list< Voice * >        VoicePtrList_t;

class MusicSequencer_t {
     MusicSequencerState_t  State;
     SequencePtrList_t           ActiveSequencePtrList;
     SequencePtrList_t           FreeSequencePtrList;
     TrackPtrList_t                  ActiveTrackPtrList;
     TrackPtrList_t                  FreeTrackPtrList;
     VoicePtrList_t                  ActiveVoicePtrList;
     VoicePtrList_t                  FreeVoicePtrList;
};

class SequenceState_t {
     Tempo_t    Tempo;
     Volume_t   Volume;
};

class Sequence_t {
     SequenceState_t State;
     SequenceState_t BeginState;               // Interactive feature
     SequenceState_t EndState;                  // Interactive feature
     SequenceInterpolator_t Interpolator; // Interactive feature
     TimeUnit_t         TimeElapsed;
     TimeUnit_t         TimeStep;
     CallbackFunc_t *pCallback;                 // Interactive feature
     TrackPtrList_t    TrackPtrList;
};

class TrackState_t {
     Volume_t     Volume;
     PitchBend_t PitchBend;
     Pan_t         Pan;
     Effect_t      Effect;
};

class Track_t {
     TrackState_t      State;
     TrackState_t      BeginState;               // Interactive feature
     TrackState_t      EndState;                  // Interactive feature
     TrackInterpolator_t Interpolator;      // Interactive feature
     Sequence          *pOwner;
     char                    *pEvent;
     Instrument_t    *pInstrument;
     VoicePtrList_t   VoicePtrList;
};

class VoiceState_t {
      SynthVolume_t   Volume;
      SynthPitch_t        Pitch;
      SynthPan_t          Pan;
      SynthEffect_t      Effect;
};

class Voice_t {
     VoiceState_t                CurrentState;
     VoiceState_t                BeginState;     // Interactive feature
     VoiceState_t                EndState;        // Interactive feature
     VoiceInterpolator_t    Interpolator;   // Interactive feature
     Track_t                        *pOwner;
     char                               nKey;
};

Event Data Structures
To implement the event type commands we can have the event type command numbers correspond to an array lookup that holds the relevant function pointer and the byte length of the event type and parameters. Code Example 2 shows this code.

The function pointers give us a quick way to get to the code associated with each event type. The byte lengths give us a quick way to step to the next event block.

Code Example 2: Event Type Data Structures

// Example Note Off Event Block
typedef struct {
     char nEventType;
     char nKey;
     // no release velocity
}NoteOff_EventBlock_t;

void NoteOff_Function( Track_t *pTrack )
{
      // the pEvent is pointing at our event block
     NoteOff_EventBlock_t *pNoteOffEB = (NoteOff_EventBlock_t *)pEvent;

     // walk through this track's voices and turn off
     // any that have pVoice->nKey == pNoteOffEB->nKey
}

     // Example Note On Event Block
     typedef struct {
     char    nEventType;
     char    nKey;
     char    nVelocity;
}NoteOn_EventBlock_t;

void NoteOn_Function( Track_t *pTrack )
{
     // the pEvent is pointing at our event block
     NoteOn_EventBlock_t *pNoteOnEB = (NoteOn_EventBlock_t *)pEvent;

     // try to get a voice from the free list or
     // try to get a voice from the active list if possible
     // if we have a voice, turn it on with the pNoteOnEB->nKey
     // and pNoteOnEB->nVelocity and other state information
}

enum enumEventType
{
     EVENT_TYPE_NOTEOFF,
     EVENT_TYPE_NOTEON,
     .
     .
     .
     EVENT_TYPE_COUNT
};

typedef void (*EventFuncPtr_t)(Track_t *);

typedef struct {
     EventFuncPtr_t pFunc; // pointer to command function
     int nLength; // byte length of command
}EventTypes_t;

static EventTypes_t aET[EVENT_TYPE_COUNT] = {
     { NoteOff_Function, sizeof(NoteOff_EventBlock_t) },
     { NoteOn_Function, sizeof(NoteOn_EventBlock_t) },
     .
     .
     .
};

Timing
Different computer systems will have different ways of providing timing callbacks or threads that wake up at specific intervals. I will simply assume that we can have a function called at a specific interval that I will call the "audio frame callback". We can think of the time between the callbacks as the audio frame. During each callback we need to update our notion of how much time has passed and we need to send out all of the commands that have "timed out".

In the sequence data structure listed in Code Example 1 there is the TimeStep quantity that should be set based on the tempo, the delta-time parts per quarter note, and callback timing. We add the TimeStep to the TimeElapsed on each audio frame to keep track of the time. Since these time parameters are in the sequence structure, we can only change the tempo for the whole sequence. If we wanted to change tempo for each track individually we could put these parameters and the tempo setting in the track structure.

Audio Synthesis Control

Control
Certain interactive music effects are possible if sequencer commands can be tied to audio synthesis parameters. The SetTrackEffect event type could represent any effect parameter that the audio synthesis system provides. Some audio synthesis parameters are simply set before a note is played and may not respond to updates until the next note is played. Some audio synthesis parameters can be altered during notes.

The behavior of the audio synthesis system that we are using will determine what kind of control we can have, and music will have to be written with the control issues in mind.

By defining immediate or target state changes for sequences, tracks, or voices we can manage what could be many controls changes with simple ChangeState commands.

Connecting Synth to Sequencer
The key to connecting our audio synthesis to our music sequencer is the SetInstrument event type. This command looks up in to a table of instrument definitions and sets the pInstrument field of our Track data structure. When a NoteOn command occurs the parameters from the pInstrument are transferred to the voice to be started.

We will need to manage the available audio synthesis voices to provide good voice stealing logic. When we have used all of the available voices we may want to end an active voice because a new note on request is evaluated to have a higher priority. The priority system could use several kinds of weights. These weights could be based on the note ontime, time until note end, envelope stage, volume levels, count of this type of voice active, instrument priority number, track priority number, and sequence priority number.


Interactive Music

Interpolation
Game programmers are very familiar with interpolation. I mention it here because our ability to interpolate between sequence states, track states, and voice states provides us with interactive control of music.

Transitions
Transitions can be defined as one or more changes occurring over a time interval. A transition might mean an interpolation of some kind of state data done over a specific time interval. Or a transition might mean a new track section, musical cadence, key change or other compositional technique. A transition could be a combination of these things.

Transitions may be triggered by a combination of game logic and music language logic. It is useful to provide API functions for game code to directly set target states. These kinds of implicit and explicit controls over transitions are another key element of the interactive control of music.

Meanings
Some of the meanings that we might want to attach to the control of music are categorized in Table 5.

Table 5 - Music Meanings

Category/Type

Description

Self

What State is the Player In?

Health

Confidence in the Music

Power

Strength in the Music

Skill

Sharpness and agility in the Music

Mood

Heaviness or Lightness in the Music

Familiar

Music That is Familiar in the Game

Unknown

Music That is Unknown and of a Foreign Style

 

 

Others

What State are the NPC's In?

Friends

Pleasing Attitiude in the Music

Enemies

Harsh Attitude in the Music

Love

Sweetness in the Music

Hate

Violence in the Music

Familiar

Music that is Familiar in the Game

Unknown

Music That is Unknown and of a Foreign Style

 

 

Location

What is the Current Location Like?

Secrets

Occasional Secret Melodies or Instruments Play

Hints

Sudden Burst When Looking the Correct Way

Safety

Even, Predictable Music

Danger

Irregular, Ominous Music

Magic

Chimes, Echos, and Sprinkles in the Music

Familiar

Music that is Familiar in the Game

Unknown

Music That is Unknown and of a Foreign Style

 

 

Situation

What Kind of Situation Are We In?

Safety

Even, Predictable Music

Danger

Irregular, Ominous Music

Magic

Chimes, Echos, and Sprinkles in the Music

Preparation for Battle

Drums of War, Mechanized Beats

Tension

Sharp Tones and Dynamic Changes

Adrenaline

Tempo is Up, Mechanized Beats

Time is Running Out

Tempo is Up, Chaotic Passages

Reward

Triumphant Music

Failure

Whimpering Music

Familiar

Music that is Familiar in the Game

Unknown

Music That is Unknown and of a Foreign Style


Transition Types
Some of the many transition types are mentioned in Table 6.

Table 6 - Music Transition Types

Type

Description

Quick

Music that Stomps on the Previous Music

Slow

Subtle Alterations and State Changes

Fading

Fading Whole Sequences or Just Some Tracks

Intensity

Instrument Dynamics

Effects

Any Synthesis Parameter Changing

Key

Compositional Changes

Chord

Compositional Changes

Harmony

Compositional Changes

Melody

Compositional Changes

Accompaniment

Compositional Changes

Percussion

Compositional Changes

Transposing

Compositional Changes

Layering

Compositional Changes

Fills

Enter from any beat position, push music decoration events in to a queue.

Rythmic

Lagging, Ahead, Modified Swing

Randomness

Controlled Randomness of Varying Parameters

Instrument

Switching Instrument

Timing

Switching Tempo

 

Design Influences
There are four important factors in the discussion of interactive music: game design, game programming, music design, music programming. Music programming is influenced by the other factors in the following ways:

  • Game design will influence music design.

  • Music design will influence music programming.

  • Game design will influence game programming.

  • Game programming will influence music programming.

To point out these influences I will present some hypothetical examples.

Design Example #1

Game Design: Through player skill, a character can achieve a powered-up state. This state can last a very long time and a sound effect might get monotonous. We want to hear his energy in the music.

Music Design: Transition the melody and percussion track instruments to add DSP effects that add color and depth to the instruments.

Programming Design: Two sequence states are created and the game can choose when to set each target state.

Design Example #2

Game Design: When our player goes near a location in a level we want to hint that there is danger using the music.

Music Design: Fade down the main melody track and fade up the danger melody track.

Programming Design: Based on distance from the location, set the track target states for volume.

Design Example #3

Game Design: Lets say we have a game design where we change from day to night. Lets say that the player's role is more offensive in the day and more defensive at night. We want energy music during the day and tense and scary music at night.

Music Design: To keep it simple, we will describe three tracks of the music: melody, accompaniment, and percussion. We will define "energy", "mellow", and "creepy" versions of each of the three tracks. Again keeping it simple, we will define a "hard" and "soft" version of each of the instruments for each track.

Game Type

12 Noon

3 PM

6 PM

9 PM

12 Midnight

Control Value

0.0

1.0

2.0

3.0

4.0

Melody Track

Energy

Energy

Mellow

Mellow

Creepy

Melody Instrument

Hard

Soft

Soft

Soft

Hard

Accompaniment Track

Energy

Mellow

Mellow

Creepy

Creepy

Accompaniment Instrument

Hard

Hard

Soft

Soft

Soft

Percussion Track

Energy

Energy

Energy

Mellow

Creepy

Percussion Instrument

Hard

Soft

Soft

Soft

Hard

Programming Design: We generate our control value based on the game time. This control value is used to interpolate the instrument states and initiate transitions for the track states.
So when our game time reaches 3pm, the melody instrument transitions to the "soft" state. When our game time reaches 6pm, the "energy" melody track fades down and the "mellow" track fades up.

Conclusions

This article covered many motivations and implementation ideas for interactive music, and presented some core programming concepts that can provide the flexibility and control needed for interactive music while also pointing out the design influences, meanings, and transition types. The next thing to do with all of these ideas is implement your own interactive music sequencer and make sure interactive music is part of your game's design.

________________________________________________________

Read more about:

Features

About the Author(s)

Scott Patterson

Blogger

Scott Patterson got his start in the games industry ten years ago doing audio programming on many of Microprose's classic PC games. Since then, at Midway, he created audio systems for many games on many platforms and gained experience with 3D graphics programming, user interface programming, and even AI programming. And then, at Naughty Dog, created audio systems, managed audio resources, wrote audio control code, and contributed to graphics tools. The last two projects he worked on were Crash Team Racing for the Sony Playstation and Hydro Thunder for arcade. He is currently Head of 3D Technology for Next Generation Entertainment in beautiful Sydney, Australia. He can be reached at [email protected].

Daily news, dev blogs, and stories from Game Developer straight to your inbox

You May Also Like