Postmortem: Insomniac Games' Ratchet & Clank

The scene: Twenty developers lounging on a sun-drenched porch overlooking Barham Boulevard in Los Angeles, drinks in hand, enjoying the warm breeze and listening to traffic rumble by below. The occasion: Our first post-Spyro brainstorming meeting.

It was late spring 2000, and even though we were still in production for Spyro: Year of the Dragon (our last Spyro), we knew we had to start planning for our first PS2 project. Our problem was twofold: we had decided not to develop any more Spyro games, and we were deciding whether we wanted to stay with the platform-action genre. It's a familiar scenario for game developers: the road is wide open, but figuring out which direction to travel is excruciating.

We had meeting after meeting trying to narrow down the choices -- and with 20 people involved, things got tense and sometimes depressing. I was driving hard to move us away from the platform genre because Al Hastings, our vice president of technology, had very astutely suggested that this was the perfect opportunity not only to expand our abilities but to address other niches in the console market currently overlooked by U.S. developers.

After coming up with and discarding countless ideas, we settled on a concept best described as a dark adventure. We wanted to try a game with a bit more realism and immersion than our previous efforts. This meant moving away from bright environments, cartoony characters, and platform mechanics. This also meant creating a macro design and story that were far deeper than those of the Spyro series.

We called the concept "I5" (for Insomniac game #5), and the main character was a human girl with a staff. She would fight with the staff as well as use it to activate magic with special katas -- martial arts moves performed using directional input. There was a strong Mayan influence to the overall look of the game, and the characters and environments we planned were more realistic than anything we had attempted since our first game, 1996's Disruptor.

We pitched our game idea to SCEA and were fortunate to strike a deal very early in preproduction. Once we had Sony's backing, our preproduction team dove in and began working on PS2 technology, final macro design, and all of the elements that would help us create our first playable.

Within a couple of months, however, it was clear that things weren't going well.

First, we couldn't nail down the main character. She was too cartoony, and then too mundane; the colors we chose ended up looking weird on-screen, and we couldn't get the proportions right. In the past, proportion had never been a problem, since we had always worked with nonhuman characters. But we quickly realized that it's easier to spot flaws in human characters than in nonhuman ones. Even though our main character eventually looked acceptable, she still lacked that je ne sais quoi which would make her stand out.

Then there was the hardware. We were making the jump from PSX to PS2 in very little time, and Al Hastings was shouldering the entire burden with some help from Mark Cerny, who had written the original VU code used on the first-ever PS2 engine. Al and T.J. Bordelon, tools programmer, were, at the time, trying desperately to get the engine and tools to the point where the artists could use them to build and prototype environments and characters. Looking back, I can't believe they actually got everything to work, and work well, in a matter of months. Still, the technology was not yet state-of-the-art, and we all wondered how it would fare against the second generation of PS2 titles.

But the worst part of the process was the entire team's ambivalence about the project. No one was truly excited about the game or where it was heading. We were making it work through sheer effort. My job was to be the concept's champion, but maintaining a positive demeanor was proving more and more difficult. Morale was at its lowest in Insomniac's nine-year history.

We eventually ground out a first playable, and while it wasn't bad, it wasn't great either. And we wanted something great. Our Sony producers, who were very polite about their reservations, confirmed our feelings. Nonetheless, they had reservations. At one point Connie Booth, our SCEA executive producer, suggested that we might want to rethink the direction we were taking. While being very clear that Sony would support us with whatever we decided, she pointed out that not only would the PS2 adventure category be crowded upon our planned release date, she also believed that we were no longer playing to our team's strengths.

After digesting her words, Al Hastings, Brian Hastings -- Insomniac's vice president of programming -- and I (the three partners in the company) did some soul searching and realized that Connie was right. By pushing on, we could release a solid adventure game, one that might even do well. But slogging through another year of developing a game no one was excited about would kill the team.

So on March 20, 2001, we stopped preproduction of I5 and started over. We would be going back to our forte, action-platforming. This announcement moved the team's mood lever from reverse to overdrive. Everyone was energized and excited about the new prospects.

Within two weeks of this decision, we developed Ratchet & Clank's basic concept. In a matter of days, Dave Guertin, our lead character designer, nailed the two main characters, and soon we were brainstorming on the weapons and gadgets that players would be using.

Once we got started, we never looked back. That isn't to say problems didn't exist during the process, but it was the best and most enjoyable production experience we've had at Insomniac.

What Went Right

1. Prototyping. We had been prototyping gameplay since Spyro the Dragon, but never to the extent that we did with Ratchet & Clank. The game featured more than 35 weapons and gadgets, all of which had to be fun to use. The big problem we faced was that every weapon and gadget was woven into the macro design and the story. If we had to pull one out during production, the macro design would collapse, which would be disastrous for the production schedule.

We spent three months building and programming the weapons and gadgets. Many of them didn't survive the prototype phase because even though they sounded good on paper, we just couldn't make them work. A good example was the Revolverator, a weapon featuring a large drill bit which would spin enemies around and fling them away. We discovered that the spinning slowed down gameplay, and that it was difficult to hit enemies, since the collision for the drill bit had to be narrow to be believable. Another good idea on paper was the Mackerel 1000, a fish that would be a replacement for Ratchet's wrench. It sounded funny, but when we put it in the game the humor lasted for about three seconds.

We also prototyped enemy layouts and behavior to a much greater extent on this project. The majority of our enemies were well tested and tuned before each level went into production. This process saved us a massive amount of time, since we only built final models and did final coding once we were sure that the enemies would work. Conversely, on the Spyro series we were always ripping things out and starting over during production, since we rarely prototyped gameplay. With Ratchet & Clank, and for all of our future projects, gameplay prototyping has now become an ongoing process.

Finally, to clearly establish the look of the game, we used our I5 engine to prototype two of the game's planned environments before we had the real Ratchet & Clank technology up and running. It was all smoke and mirrors, but it allowed us to show on-screen what we imagined the final game would look like and put to rest a lot of our own fears about whether or not the game would stand out visually.

2. Sharing technology with Naughty Dog. Shortly after we decided to start over, Jason Rubin, Naughty Dog's co-founder, called me and asked if we'd be interested in checking out the technology they developed for Jak & Daxter. He explained that Naughty Dog didn't want anything from us other than a gentlemen's agreement to share with them any improvements we made to whatever we borrowed plus any of our own technology we felt like sharing. In an industry as competitive as ours, things like this just don't happen.

We went over to Naughty Dog's offices and took a look, particularly at their background renderer. They had developed some incredible proprietary techniques to render smoothly transitioning levels of detail and instanced objects very quickly. We brought the code back to our offices, spent some time getting a handle on their techniques, and then we were up and running with a much more powerful environment engine.

Needless to say, Naughty Dog's generosity gave us a huge leg up and allowed us to draw the enormous vistas in the game. In return, we've shared with them any technology in which they were interested, but so far we've been the clear beneficiary of the arrangement.

3. Setting reasonable design goals. Even though the concept behind Ratchet & Clank was ambitious for us (integrating RPG elements into an action-platformer), we were careful not to cram too much stuff into the initial design.

We had never made a game before where we didn't have to axe one or more levels at some point in the production process because we were out of time. The Ratchet & Clank macro design was more complex, so we couldn't afford to rip out a level at the last moment. Sony had created a tremendous marketing campaign that relied on a specific release date, so missing our delivery dates was not an option. Plus, we were already releasing pretty late in the year, and to miss one week of precious pre-Christmas sales would prove very costly.

For these reasons, we planned the game layout much more carefully than we had on past titles. We had a pretty good idea of how long it would take to build each level, but we also knew that plenty would go wrong during the production process. So even though we had time to do 20 levels, we cut back to 18 at the very beginning.

We also made sure that nothing went into the design unless we were very sure that it was going to work. Early prototyping was key here, but so was an attitude of general restraint. There were a few wild concepts that everyone was excited about, but had we integrated them into the macro, the project probably would have slipped. Ultimately we were able to put about 90 percent of what we planned into the game -- a record for us.

4.Focus testing. Most games go through focus testing at some point. Publishers and developers alike want to see how people react to the game and whether it's too difficult or too easy. Because it's the best way to tune the gameplay, we've focus-tested our games since the first Spyro. But with Ratchet & Clank we went overboard.

We had four major focus tests during production. Each focus test featured another 25 percent of the game until we were testing the full game at alpha. More than 200 consumers got to play the game before release, and the feedback we collected was invaluable. By recording and charting data from the game, we were able to tune item prices, adjust challenge difficulty, and change monetary rewards. Without this exhaustive process the game would probably have been unplayable.

Just as important, though, was the fact that each focus test forced us to get the game working. Along with the other deadlines it sometimes felt that we were always in crunch mode. The gameplay programmers in particular lived a nightmare existence between fixing bugs for the next focus disc and trying to move ahead with the new levels. But the constant burns kept us on track and on schedule. Given Ratchet & Clank's scope and complexity, if we had waited until the end of the project to burn playable discs, the bug list would have been overwhelming and we would have missed our ship date by months.

5. Collaborative design. Everyone in the company has always been free to contribute creatively to the projects. It's not a requirement, but for those who are interested it's an opportunity to affect the direction our games take. Programmers are encouraged to contribute to story, artists are asked for ideas on design, and so on. During Ratchet & Clank, a large percentage of the team contributed ideas outside of their particular areas of expertise, making the game one of the deepest and most varied titles we've developed.

This does not imply that we design by consensus. There's a solid structure in place to ensure that we adhere to the macro design and remain consistent with the game's "flavor." But adopting an approach that encourages design participation gives us a real wealth of creativity from which to draw while enhancing the sense of ownership everyone feels in our games.

What Went Wrong

1. Poor disc-burning process. Making the switch from CD-ROMs on the PSX to DVDs on the PS2 sounded like it would be easy. After all, we survived the challenge of recording PSX discs with quirky burners and nonintuitive burning software. What we didn't account for was the incredible amount of time that building and burning the DVDs would take.

We had to first transfer the code and data to the PC on which we would generate the files necessary to create a playable disc. Next we'd have to transfer the files to the burner PC. Then the burner software would have to create a disc image, and finally we could burn the disc. By the end of the project we were working with 4GB of data. Combining those steps with slow connections and a burner that we had to use at only double speed to prevent errors, the entire process took more than four hours to generate one disc. And there were many, many places along the way where something could go wrong, forcing us to start over again.

There were countless instances where a level would be out of memory or someone would change the memory card format, breaking everything. But we wouldn't know about it until the final disc had popped out of the tray and had been booted up on a test station. Two mistakes like this would cost an entire day.

So why didn't we change the process? Based on our PSX-burning experience, where the system was extremely finicky, when we had things working on the PS2 we didn't want to touch it and risk breaking everything. This was especially true near the end of the project.

As a result, a few of us didn't go home for days at a time near the end of the project. I remember promising our testers that if our first gold burns worked, I would do circuits of the office singing Britney Spears songs as loud as I could. Fortunately for everyone in the office, they didn't.

The result of our disc-burning pain is that we've now completely overhauled our system. We believe we've halved the overall disc production time for our current project.

2. Late start on cinematics. Ratchet & Clank has a much more lengthy and involved story than any of our previous projects. Oliver Wade, our animation director, compiled the scenes and found that we've got more than 60 minutes of movies. Even though most of them are about 30 seconds long, that's a lot of animation time. The problem was that we only gave our team of seven animators five months to animate them. That doesn't sound too bad until you consider that the animators creating the movies were also responsible for the in-game animations. Therefore they effectively had two-and-a-half months. If you don't include weekends, that's about 10 seconds per animator per day. And that's a lot.

Fortunately, the animators had finished most of the in-game animations by the time the movies were in full swing. But it was still a real challenge. Furthermore, animating the scenes was just the first step. We had to add programmatic and 2D effects and convert many of the animations into MPEGs before alpha, which stretched many people to the limit.

We got such a late start because we had to finalize the story, write the scripts, audition the actors, record the dialogue, and put the final sound files together before starting the animation. It helped somewhat that we took an iterative approach -- starting animations as soon as the first scenes were recorded -- but in general the tardy start created a lot of stress.

3. Immense level designs. Even though we tempered our ambitions for the macro design, sometimes we cut loose and created some absolutely huge level designs. We had a habit of wanting to make each level better than the last, and a few times this tendency resulted in layouts that made the artists want to kill the designers.

Early on, we didn't have a good understanding of what "too big" meant. The first level designs we created were reasonable, but then we decided that we really needed to show off the power of the Ratchet technology. We also had some ambitious gameplay ideas involving a fight on a moving train and a hoverboard race. This resulted in the Metropolis and Blackwater City levels, two of the biggest in the game. When the artists saw the layouts they said, "Are you nuts? There's no way we can build this in six weeks!" So the designers went back to the designs and tried to edit them, but the levels still ended up being massive.

To the artists' and gameplay programmers' credit they made these and other huge levels work, and they did it on time. And to the designers' credit, they continued to find better and better ways to put more gameplay into smaller areas without sacrificing creativity. In the end, our level design ambitions pushed the limits of time and resources we had allotted.

Out of this stress came a more team-oriented approach to level design, where we now involve a large number of people -- artists, programmers, sound engineers, and others -- earlier in the design process. Whether or not levels in our future games will be smaller remains to be seen. But with more people involved at the beginning stages, we can find solutions sooner to balancing the need for gameplay space in levels with the time we have available to build them.

4. Maya issues. Maya is a superb tool for building polygonal environments and characters, and it's also great for animation and for prototyping particle effects, rendering, and many other things. However, early in the project we had decided to use Maya as our construction, texturing, lighting, and gameplay placement tool. We had abandoned our in-house tool, Karma, which we had used previously to do gameplay placement, texturing, and lighting. What we didn't realize was that with the size of our levels, we would push Maya past the breaking point.

Even though we set people up with dual 1.2GHz Dells with superfast graphics cards and a gigabyte of RAM, Maya would still chug and frequently crash whenever our levels got up to around 40MB. And forget making all 500K polygons in a level visible. Fortunately, Al Hastings and T.J. Bdelon worked valiantly to create a suite of plug-ins and tools that worked with the Maya API. This solution didn't always prevent the crashes that plagued the artists or the occasionally corrupted level, but it kept us running and allowed us to create finished levels every six weeks.

While Maya has always been and probably will still be our first choice for art creation, we're moving back to our original approach of using proprietary tools for things like gameplay setup, lighting, and texturing.

5. Localization woes. From the beginning we planned to include the NTSC and PAL versions of the game on one disc. This plan created two problems for us. First, we had to send all of our assets to Europe for localization in French, Italian, German, and Spanish as early as possible. In most cases this meant pre-alpha, which really put the squeeze on the animators who were working on the movies. Second, we knew that we would end up fixing both functionality and localization bugs at the same time. We anticipated that this would create even more chaos duri ng the last few weeks before we went gold. And we were right.

Surprisingly, the biggest nightmare for us was the text localization. We had made the decision to allow subtitles for all of the movie scenes; plus we had a lot of text for the help system and a ton for the menus. We used spreadsheet databases to ensure some organization for all of the text (as opposed to entering localized text in the actual code, which we did on the Spyro series), and this allowed us make updates and changes quickly. But the system was also prone to user error when cutting and pasting changes into the database.

Because we were still fixing TRC (technical requirement checklist) bugs -- things like memory card messages -- we were making text changes up to a couple of weeks before gold. We had also added some text late in the process to support some of our postgame features.

We made mistakes, and the localization folks in Europe made mistakes when putting fixes into the database. In addition, it took forever to transfer our discs to Europe once they were burned (eight hours to FTP if nothing crashed, 24 hours for a courier). These facts combined meant that we were still desperately trying to resolve some TRC issues hours before the gold disc was due. Fortunately, the game shipped on-time in all territories, but I think it prematurely aged our producer in Europe, as well as a few of us here.

The Will to Kill

With this project, we had to fail to succeed. Had it not been for the pain we went through on I5, Ratchet & Clank might have never emerged. In the six months of preproduction on I5 we learned how to make games on the PS2, and we were able to hit the ground running when we switched to Ratchet & Clank.

Most importantly, we were very fortunate to have an extremely supportive publisher in Sony. SCEA's Shuhei Yoshida and Connie Booth helped us make the agonizing decision to shoot I5 in the head. But they made sure we understood that if we wanted to continue down that dark path of developing I5 for release, they would still support us. Furthermore, Sony never once threatened to cancel the I5 project or sever our relationship. Instead, they helped us to develop what Mark Cerny calls "the will to kill" ? meaning we grew the balls to voluntarily throw out everything we had worked so hard on for six months and start over.

The development process that Ratchet & Clank represents as a finished game is the ultimate example of how developer-publisher relationships can and should work. Sometimes good teams make games that aren't good. When a developer has the support of a great publisher and can cut off a nonperforming project in preproduction without fearing reprisals, everyone can save millions in production costs and apply the lessons learned to the next project. Doing so may cost money in the short term, but ultimately it may give birth to a blockbuster, strengthen the development team, and solidify the relationship between the developer and publisher.