Manager In A Strange Land: Turnaround Time

Enough general advice. Let's hit some specifics. One of the most obvious ways you can optimize your team is by decreasing turnaround time.

By turnaround time, I mean the time between making a change and testing the change. Different sorts of changes require different turnaround times. Some of them are as low as several seconds, such as using "Edit and Continue" in Visual C++ to stop the program, fix something, and keep running. ("Edit and Continue" is the coolest thing since source control.) After that we get into the minute range: changing one .CPP file will usually require around a one-minute rebuild and then the one to two minutes to load the level. (Although we do have a light level that takes only around 30 seconds, great for prototyping new features.) Changing an .H file often necessitates a complete rebuild, which can take anywhere from five minutes (for a debug PC version) to thirty (for an optimized PS2 version.) This is coder-centric, but artists and designers are in the same boat: they make a change, they need to export data, build data, load a level.

I find that when it takes more than a few minutes to see a change in the game, I lose "flow". I drop out of the zone. I start browsing web-sites, go get a drink, wander over to someone else's office, post another article to my blog, whatever: often the build finishes or the level loads and I don't even notice, creating additional idle time. And I forget what the hell I was doing. Frequently, when I see others "goofing off" at work, and go to investigate, there's a reason: they're waiting for data to build, they're waiting for a sync to finish. Although there are some conscientious people who sit and meditate on what they're working on while they wait for turnarounds -- and those guys are often the best employees on the team -- for most people, there's a point where turnaround times cost much more than just the wasted time spent waiting.

By way of comparison, I've done some programming in interpreted languages. Scripting missions for our game or writing database clients in Visual Basic, for example. And currently I've got a side-project going on at home where the build time is still under ten seconds. These experiences are pure pleasure. Flow is achieved. I feel like a potter at the wheel, sculpting my software. Some coders would be horrified by that analogy, as it shows a code-like-Hell inclination rather than the inclination to write correct code the first time around. Guilty as charged. Still, just because fast turnaround times allow us to be bad programmers doesn't mean we have to be bad programmers. Or maybe we should go back to job control and mainframes because that encourages us to triple-check and code review before we even compile?

Enough preaching. Here are some techniques to lower turnaround. I'm going to start with coder turnaround because that's what I know best, but on your project code may not be your biggest problem; you may be more worried about getting as much art, or as many levels, into your game as possible. In which case you should focus on content turnaround.

Measure your build times. You can't control what you can't measure. Just like optimizing code, optimizing build times requires an iterative process of measurement followed by single-variable changes.

With Visual C++ 6.0, you can pass in the /y3 option when running MSDEV.EXE and it will time your builds. Cool little undocumented feature. With .NET, go to Tools -> Options -> Projects -> Build Timing and set it to Yes. With command line tools you can use %time%.

Use metafiles. Symantec once had a C compiler called LightSpeed that had what they called a "Missing Link" - the link was almost instantaneous. Their secret, I believe, was that they didn't link, they just compiled all the C files into one big C file, and then somehow incrementally compiled single modules. I believe this because files listed later in the project could refer to the global variables declared in the earlier modules. Pure evil, obviously, especially if you don't know the compiler is doing it. Still, maybe we can use this for good.

If you create "metafiles" (.CPP files that include a dozen or so of your real .CPP files and nothing else), you'll see your build time improve, partly because your link time will go down, and mostly because your .H files will only be compiled once per dozen files instead of once per individual file.

Then you have a trade-off; when making an isolated change to a .CPP file, your build time for that single file is now longer. It's up to you to find the optimum balance between individual file compile time and full project compile time. One thing you can do is make a FILES_FAST.CPP metafile. If you're going to be working on a single file for a while, you can copy that file to FILES_FAST. It's the only file you put in there, so it takes very little time to rebuild. Best of both worlds, if you have the foresight to know which files you're going to be working on.

Of course, the downside to metafiles is it butchers your modularity. Suddenly, programmers working in one module could, in theory, see the static variables and declarations in another module, and violate their encapsulation.

Use precompiled headers. I've found that most programmers don't know how to use this feature with Visual C++. You can't just go into your project settings, turn on "Using precompiled headers", and expect it to work. What you need to do is this:

One warning: everybody on the team needs to know how precompiled headers work. A problem I frequently see is a new programmer tries to add a new .CPP file to the project and gets an error message about the precompiled headers. So they go in to the project settings and turn the precompiled headers off for everything. Voila, it works again. Forgive them, for they know not what they do.

There's a theoretical minimum time your compiler will take to do a complete rebuild. If you compile and link a 1000-line program and measure how long it took, you can then come up with a pretty good estimate of how fast you're going to be able to get your 200,000 line game to build -- unless you use a distributed build environment. We haven't bothered with that, because our full build times are around five minutes and products like IncrediBuild only promise to get that number down to three. If we had a ten-minute build, we would have signed up for IncrediBuild yesterday.

______________________________________________________