Author: by John Fiorito

[Reprinting one of Game Developer magazine's most acclaimed 2008 postmortems, Insomniac goes behind the scenes on late 2007 standout Ratchet & Clank Future to explain the creation of the iconic PlayStation 3 platformer.]

One of the first features developed for Ratchet & Clank Future: Tools of Destruction was the "Groovitron" -- a bomb that was part boom box and part laser light show. Throw the bomb, force your foe into spontaneous dance, and then kill him to a disco soundtrack. It was hilarious to watch and just as fun to play.

But the music almost died when we realized that we'd have to give every character, enemy, and creature a set of unique dance moves. The Groovitron would require hundreds of animation cycles, special case programming, and extra work across the entire project.

In many ways, this reflected the challenge of bringing our heroes onto new hardware; the complexity, sophistication, and power of working on a new platform meant that our team would need to perform at an equally complex and sophisticated level. We had our work cut out for us but we felt we had an opportunity to create the Ratchet & Clank game we'd always imagined.

Ratchet & Clank Future: Tools of Destruction (RCF) is Insomniac's fifth Ratchet & Clank game. We needed to deliver an experience on par with its predecessors while making a fresh debut on the PlayStation 3.

Our challenge became even more daunting when we added to this a tight one-year development cycle (plus one year of preproduction with a skeleton crew), a production budget only half or two-thirds of similar games, and no experience working on the PS3.

It was necessary to recreate the Ratchet & Clank universe from scratch for PS3, and what follows is an account of the many issues we faced, the decisions we made, and the methods we used to do this.

What Went Right

1. Set a vision.

In late 2005 most of Insomniac was working mightily to get Resistance: Fall of Man up and running on early PS3 hardware (read the postmortem in the February, 2007 issue of Game Developer). Meanwhile, about a dozen of us were trying to figure out where to begin on a new Ratchet & Clank game. We were lacking hardware, an engine, game code, and even assets. We were truly at ground zero.

Figure 1: The opening view of Metropolis as we imagined it in 2005.

To start, we decided to visualize the Ratchet & Clank universe PS3-style by recreating Metropolis, one of the iconic locations from our PS2 series. We did this by building a "diorama" of the city, adding vehicles, and sending a camera through it.

We built our test city using the Resistance engine, stitched together a frame by frame camera fly through, and added audio effects to simulate the experience of being in Metropolis.

We were pleased with the outcome, but we knew that this was a guess at best and revealed more about our hopes for the game without the memory, frame rate, and game design constraints of a real level.

The reaction from our producers at Sony and later from people who saw the Metropolis video at 2006 GDC was astonishing. We were being compared to feature film CGI, and there was great enthusiasm for the game.

When Sony told us that future gameplay deliveries needed to "drop jaws" as Metropolis did we wondered if we could ever match the results in-game (see Figures 1 and 2). At this point we still did not know what lay ahead for RCF, but we knew one thing -- we had a vision.

Figure 2: The in-game opening view of Metropolis in 2007.

2. Pre-production time.

Armed with our demo, we set about trying to plan the game. It was the beginning of 2006, and we had 10 months to get ourselves ready for production, which would begin in earnest when Resistance shipped and the full team joined us.

We still had a lot of unanswered questions; What would Ratchet look like on PS3? How would we animate characters that were four to 10 times more complex than the previous generation? What constituted a "level?" How big was the game going to be, and how would we make it all work?

To start, we established the overarching goal of creating one fully-realized and playable level. Hopefully, this would give us two valuable pieces of information; first, an understanding of everything needed to get our game up and running, and second, a set of benchmarks to plan the content for the rest of the game.

We also had three distinct production paths that we needed to keep aligned -- asset creation, gameplay prototyping and implementation, and game design. To keep things organized, we set a series of escalating milestones. These included first prototype, first functional, and first playable builds that would eventually lead to a completed level.

Our milestones were shared across the entire team so that we maintained unified production goals. In between, we divided specific assignments into one- to three-week blocks to keep the workload comprehensible to the members of our team. We met our goals, but wound up overshooting in terms of our design scope. See "what went wrong" for more on this.

3. Balancing creativity and deadlines.

After our PS3 launch title Resistance had successfully shipped, our 15-person preproduction team began to swell to more than 100 people (70 full time and numerous "shared" resources). To carry out our game design, we needed to keep the entire team productive from day one.

As much as possible, we needed a stable production environment with a constant flow of game code and game assets ready for whoever needed it next. Time was our biggest enemy and we needed to do whatever possible to use it wisely.

We attempted to establish a production structure that rationalized the complexity and scope of our game but also gave everybody working on the project flexibility. One of the tenets of Insomniac's culture is to allow creative contribution and control at all levels, and we had to balance this with another Insomniac truth -- we never miss a deadline.

Within our major milestones we established production cycles, eight-week blocks of time where our various teams created components of the game. At the end of each cycle, these pieces were brought together, often resulting in a play test, media presentation, or both. During the project, we were able to establish and track progress according to these recurring cycles while individuals or teams could fit their unique workflows into a cycle.

At the same time we learned that creating functional gameplay rather than finished components not only accelerated progress but embraced iteration and change along the way. On past Insomniac productions we often tried to take features to completion before moving on, which complicated the inevitable changes and added unnecessary work.

As production intensified, we adopted additional strategies to keep up our momentum. Early on we instituted "Friday walkarounds." This was where the creative director and project manager would visit each production department (art, animation, design, gameplay, etc.) to see work in progress or evaluate completed assignments.

We tried to keep this casual -- usually we would view work at its creator's desk. However, by sticking to a regularly-scheduled routine we built a formal evaluation structure that focused the team on weekly, self-determined progress. Later, as the game was taking shape, we held Monday morning "war room" sessions where our designers and programmers would report the status of the game and what lay ahead for the coming week.

Again, this took place in an informal setting, but ended up being the other bookend to our Friday walkarounds that organized RCF production into discreet one-week segments. Still later, we added a "Daily Load Test" report that our Q/A team sent out to everybody working on the project. This was a simple test that reported if a level would load, if it could be completed, and what issues stood in the way.

It came at the time during the project where maintaining stability on a daily, or even hourly basis was essential, and gave everybody an accurate way to understand the most critical issues.

4. Lessons from the past.

One of the biggest aids to RCF production was the experience gained on Resistance. We were now working on a second-generation PS3 title, and we benefited greatly from working on a game engine that survived the rigors of launch title production. Meanwhile, our technology continued to develop.

RCF included a new form of texture streaming that allowed us to add great variety and lushness to our surfaces and further optimizations meant we could target 60 frames per second instead of 30.

By shipping a game on PS3 we proved to ourselves we could do it successfully and gained the kind of understanding that only comes by doing. We did not know everything to expect, but we had a good head start.

5. First party relationship.

Another big help was our "first party" developer/producer relationship with Sony. On all fronts we received support that was immeasurable. At a time when PS3 development kits were scarce, we had 150 of them in our studio. Sony Europe and Japan provided localization support for 13 languages, and our game was tested throughout America, Europe, and Asia.

Creating a basic structure and organization of RCF's development didn't require any rocket science -- though there's plenty of it in our code base.

The most important lesson we learned was that using the above measures helped put our team in a position where we could succeed on a very ambitious production. Once we realized this, we believed that we would succeed, and achievement no longer became an assignment. It became our expectation.

What Went Wrong

1. New-gen scale.

In general, we were unprepared for the natural challenges that came with working on a new-generation title, even during pre-production. Everything seemed to take longer than we wanted it to, and making changes was not as simple as it had once been.

Establishing a design for Ratchet proved especially challenging as we considered redefining the character. He went through numerous iterations before we felt we got him right (see Figure 3).

Figure 3: Early character studies for a next-generation Ratchet.

At the same time, we were still months away from the PS3 hardware launch, so technical stability was inconsistent at best. With these problems and the resulting lack of in-game feedback, it was especially difficult to lay out level designs.

Progress was slow, but we were still making some. By the end of preproduction we had one semi-complete level that was functional but not very stable. This would eventually become "Kerchu City" (see Figure 4).

Figure 4: Kerchu City as it appeared at the end of preproduction, fall 2006.

In addition to a level, we gained our first production experience that enabled us (or so we thought) to look at our overall macro design to determine if we had enough time and resources to complete it. We did not.

At the end of preproduction RCF's design called for 25 unique planets, 5 space combat missions, 1 hour of cinematic cut scenes, a hazily-defined co-op mode, and an even more ambiguous online component.

We wanted the game experience to last about 15 hours -- and we achieved this -- but the final scope of our game was 16 planets, 3 space combat missions, 45 minutes of cinematic animation, and single player mode only.

When we scaled back, it was not a popular decision but it represented a turning point in the project. We now had a design that the production team felt was achievable (though still ambitious), and we had removed content from the game before it ever went into production. Nothing was "cut," and during actual production no work was ever discarded.

We had spent almost one year piecing together a single level of our game. With less than a year to go, and 18 levels left to create, we remained optimistic because we now had a plan that we believed in.

2. Launch titles are not production benchmarks.

As noted above, our knowledge gained from building a preproduction level and shipping Resistance was an invaluable aid to our planning process. Without this data many of our production estimates would have been way off. What we failed to realize, and what hurt us at the end, was our lack of understanding in terms of what it took to finish a game of this scope.

Our launch title experience did not (and could not) provide measurable production benchmarks. It was a sprint to the very end and involved many unknown variables. There really was not a true post production period as Resistance development continued right up to its launch.

Working on a simulated level during preproduction was also inaccurate since we did not account for many of the issues that need to be dealt with when shipping a real game (such as memory and frame rate optimizations, obscure crash bugs, and localization errors).

We gave ourselves eight weeks from our project alpha date to "polish" our game and get it out the door. This was the same amount of time used on our PS2 titles, which had been loosely based on our PS1 model. Needless to say, it got a little crazy at the end of the project, and we've learned to add additional post production time to our future efforts.

3. Difficulty creating stable work builds.

During RCF, our greatest periods of progress occurred when all of the pieces clicked, and the project seemed to take on a life of its own. The end result would be a stable working build of our game that could be play tested and further refined.

Unfortunately, stability remained elusive throughout development and often required Herculean efforts from our tools, tech, and gameplay programming teams to achieve. There were times when we could not produce a clean build of our game for up to three weeks.

Because of our compressed development cycle, we needed to continue creating assets and wiring the game but we had to do it blindly as our stabilization efforts took place.

Once we fixed the problem(s), the resulting change lists submitted to the project were so massive that they invariably broke the game, which again required us to re-stabilize. This frustrating downtime hampered our ability to tune the game iteratively. This represents an ongoing struggle at Insomniac.

4. Developing two games simultaneously.

In addition to working on new hardware, it was a new challenge to have two games in simultaneous development, and we felt our share of growing pains. Previously, we focused the collective effort of our studio on one title at a time. Any important issue could receive the attention of our entire company if necessary.

But now, we had two games to think of, and needed to work with people who were considered shared resources and not always available.

What proved more difficult than sharing people was hiring people. When RCF went into production we were understaffed, especially on our programming team. Yet, we did not adjust our plans and remained hopeful that we'd find the right person soon.

We did not complete our hiring until three months before our release, and this meant that our understaffed departments had to take on extra work, which is not a sustainable strategy.

5. Expecting the unexpected.

RCF also experienced its share of curveballs. We were given the opportunity to show our game as part of the TV show Extreme Makeover: Home Edition, provided we create a unique character to tie in with the episode (see Figure 5). This resulted in some good publicity and one happy boy in Oklahoma, but it cost us a week.

Figure 5: Captain James, who appeared in RCF during an episode of Extreme Makeover: Home Edition and lives on as an unlockable secret in the game.

We were running builds on PS3 devkits but Sony Q/A only worked on test kits, a miscommunication that forced us to scramble to fit our builds on disc earlier than planned.

We prepared a downloadable demo of our game to coincide with E3 but it was an in-progress demo that only ran at 30fps. So we scrapped it and made the demo again as we were closer to finishing the game.

Future Tools

I doubt any of these production problems are unique, and many of them have been solved by other studios. Our biggest challenge was to deal with them in the context of creating a game, which certainly raised the urgency and stressfulness of the situation.

A lot of things did not work out for us, but a lot more of them did. We were able to stabilize many of our development techniques but know there will always be more ways to improve. And as we continue our mission to make great games, we've moved a step closer to a smoother production model.

Game Data

Developer: Insomniac Games
Publisher: Sony Computer Entertainment
Release Date: October 23rd, 2007
Platform: PlayStation 3

Number of Developers: 70 full-time, 30 shared, 25 contractors (music, voice acting, testing, localization).

Length of Development: 23 months: 11 months preproduction, 12 months production.

Hardware Used: Artist workstation: Dell Xeon 3.2G with Quadro FX3400. Programmer workstation: Quadcore PC with as much memory crammed into it as possible.

Software Used: Proprietary engine and development tools. Artist software: Maya, Photoshop, ZBrush, Turtle. Asset management and revision control: Perforce. Design/gameplay scripting: Lua. Compiler: custom GCC compiler provided by Sony. Character lip synching: Annisoft.

Number of Times Art Director Spoke Ill of Dragons: Zero -- for he is crunchy and good with ketchup.

Technologies Licensed: SpeedTree, Bink, Microsoft Visual Studio, Anark (UI menus), DevTrack (bug database).

Project Size

Game size: 22.5gb
Project size: 1000gb
Bugs: 16,000
Source Files: 4429
Lines of code: 980,184