Creating Games Controlled By Music and Sound

As a part time game developer, full time composer, I'm always interested in blending the worlds of music and games. I recently tried to create a video-game that  was controlled solely by the sounds a player made. 

I settled on using the parameter of pitch. You can see a video of my little proof of concept below:

And if you wish download (from dropbox) and play it here!

Although it is not very often utilised, the potential for using sound to control a game (outside of simple pedagogy) is really exciting and actually not very difficult to implement. An instrument or voice offers some very interesting control possibilities that a game pad or even a computer keyboard can't deal with.

For example; the sheer number of easily accessible inputs - pretty much anyone who knows how to vaguely play an instrument will have at least a grasp of where the notes of their instrument lie, and this means they have access to a controller with potentially hundreds of inputs that are much easier to use than on a standard keyboard. Imagine an RTS or RPG with its myriad of controls and quick binds (awkward and unintuitive) - translate this complexity onto a violin or piano and all those binds become accessible and more importantly, fluid. This frees these games to change their pace dramatically, transforming their complexity into an action oriented realm, maintaining their emergence and depth while being able to process user input at a break neck speed and in a fluid manner that mimics the moment to moment viscera of the action genre.

Other possibilities:

However there are a few warnings to bear in mind when trying to create this sort of game:

With all of this my main aim was to challenege people simply to think about the possibilities of controlling their games through a medium that can enrich their aural perception or musical sensibilties rather than just the coordination of their fingers.

One of the things I haven't really got into is how these different control systems really impact design - I've mainly looked at mapping existing control schemes onto sound input. However, if developers really start using this technology then the whole ludology of sound games will have to shift. I have no idea what this could lead to, but I can imagine games which fall more and more into the fantastic flow of music and pure sound. Where the aesthetics of the play experience stem from the expression of creating beautiful noise and are encapsulated as much in the game's systems of input as in the rules.

If anything, games which use true musical and sound input need to escape from the tyranny of the rhythm game, of the (infamous) Guitar tutor and other pre-defined genres and carve out their own world of expression. If you are a developer try it out! I'd be excited to see the results. I made a post about the specifics of doing this in unity in a similar post on my other blog to help get people started. It is amazing how something as simple as the method of interaction with a game can completely change the meaning of a game experience. It is something that all game developers should play with and experience.

Declan Kolakowski : @infiniteoctagon