informa
3 MIN READ
Features

Postcard From GDC 2005: GDC Mobile - Novel Ways to Use Audio in Games

As developers look for more ways to add value, Jukka Holm and other researchers at the Nokia Research Center in Finland are finding ways to have audio and music drive video game content in the relatively ignored mobile audio space, in this lecture report from GDC Mobile 2005.

Audio often gets the short end of the stick during the mobile game development cycle, but researchers at Nokia are looking at ways to start from music and audio to create compelling mobile game content. Jukka Holm, of the Music Technologies Group at the Multimedia Technologies Laboratory at Finland's Nokia Research Center, presented examples of how game designers could use music as the starting point for video game content creation.

Holm began the talk by discussing music-controlled games. Now that modern handsets have the power to spare, they can analyze music and audio data. Analysis yields control information to send to the game engine, which is then mapped to controls to game parameters. Even as handsets become more powerful, there must be a divide between MIDI and wave audio analysis, Holm explained, but the same paradigm roughly works in both realms.

He offered up three game ideas which, unfortunately, we were not able to see due to time constraints. The first was Bug Man, where the music creates the movement of the insects in the game. In CDEFGABC the player must collect falling blocks into a container. The positions the blocks fall toward are chosen by the musical notes present in the MIDI data that is sent. The last idea he presented garnered a number of laughs; the Striptease game features a dancer who moves between states of undress as the player chooses which music files for her to dance to.

He then presented a demo video of AudioAsteroids, which showed the game progressing through five different levels, each with its own different MIDI music. From Thus Spake Zarathustra to fast-paced drum and bass, the relationship between different enemy entrances and individual instrumental activity was very clear.


Nokia Research in Finland

This was all done using software from Nokia called Virtual Sequencer. Available for Symbian 6.1 and 7.0, Virtual Sequencer provides MIDI analysis for music in real-time, driving both game content control parameters and the music playback engine simultaneously. Games using digital audio fare less well than MIDI driven games, but some of the same content creation techniques still apply. In the wave audio version of AudioAsteroids, Holm explained that the control parameters the music created were generated non-real-time. He also explained that the pitch detection is much more difficult, especially considering the compression that is often used in today's pop music. It was also clear from the demo that the relationship between the music and the created game content was much less clear. "Eventually, we want heavy metal to be more difficult than 70s funk," Holm explained.

The last part of Holm's presentation presented the idea of Shared Sound Worlds. Rather than treat each player as part of a single sound stage or as two distinct sound stages, components are shared between the individual handsets in a single world of sound data. In an example showing a volleyball game involving four people with individual handsets, the volume on each handset reflects the individual's player from the ball while the pitch reflects the altitude of the ball. For the player who is closest to the ball, he must "hit" the ball back up when it is within a range of pitch.

Holm also showed off a tennis demo that used a cell phone with a built in accelerometer. As the tennis ball came closer, the pitch of the ball's tone became lower, and when it was low enough, the person in the video swung his phone like a tennis racket to hit the ball away.

There are still other uses for audio in mobile devices, Holm said after the talk, and developers will soon have the tools to bring these more novel ideas into the marketplace, but it's clear that some intriguing research is going into the hitherto largely unexplored ideas for mobile audio.

______________________________________________________

Latest Jobs

Xbox Game Studios

Redmond, Washington
10.5.22
Technical Lighting Artist

Innogames

Hamburg, Germany
10.5.22
Game Designer - Elvenar

Bandai Namco Mobile

Barcelona, Spain
10.5.22
Principal 3D Animator

Cryptic Studios

Los Gatos, California
10.5.22
Staff Core Software Engineer
More Jobs   

CONNECT WITH US

Explore the
Subscribe to
Follow us

Game Developer Job Board

Game Developer Newsletter

@gamedevdotcom

Explore the

Game Developer Job Board

Browse open positions across the game industry or recruit new talent for your studio

Browse
Subscribe to

Game Developer Newsletter

Get daily Game Developer top stories every morning straight into your inbox

Subscribe
Follow us

@gamedevdotcom

Follow us @gamedevdotcom to stay up-to-date with the latest news & insider information about events & more