Manager In A Strange Land: Churning

Last column I discussed how you can speed up a content team by decreasing turnaround and getting them a decent asset control system. There's another thing that slows your content team down, and that's having a broken game. If your content team can't play the game, they can't test their stuff, and one thing they constantly need to be doing is testing their stuff.

This is something we're always struggling with. On Spider-Man, one or two of the levels would break about every day. We did a daily build, so we'd catch the problem within twenty four hours and fix it, so if somebody needed to work on a level, chances are that level was functioning.

On Spider-Man 2, the problem was exacerbated because we're basically doing one giant level, and all the data is built for that level before the game is even run. If any data doesn't build, we're all brought to a screeching halt. Every day the data is broken a few times. A daily build no longer suffices.

Exhortations to the developers to pretty-please stop breaking the game are not successful. Increasing the stringency of the validation process before submit -- we call it "The Drill" -- helped some but not much, and means that developers had to spend anywhere from fifteen minutes to hours running tests just to submit new data.

What finally did help is the end-to-end autobuild, or, as we call it, "churning". As soon as one build completes and is successful, the testing machine gets the next change. If there is no new changelist yet, the testing machine idles until another change is submitted. Then it syncs to the new change, rebuilds the data, and e-mails the team if there are any failures. We've accepted that people are going to make mistakes -- giving up on Steve Maguire's claim that it's possible to keep bugs out of the master sources -- what we have now is a system that catches those mistakes within minutes of when they're made.

We currently have five machines constantly building and validating data; one for each platform, including the PC, and the "monkey" machine, which runs a test script that loads each portion of the game while mashing controller buttons. The "monkey" machine so far has not been as effective as I hoped; usually somewhere there's a crash bug in the game that takes many hours to solve and the monkey is sitting idle during those hours.

Getting these autobuild scripts built isn't free. It takes a few weeks of coder time, and then requires regular maintenance. Frequently our autobuild machines will fail for one reason or another; a human has to check on them on a semi-regular basis (daily isn't often enough; hourly is too often) to make sure they're still churning away.

One rule you have to have when you allow sloppy submits is this: don't submit if you're going to leave work soon. It's very tempting when you get to the end of the day, and you've put the finishing touch on that feature you were implementing, to submit it and go home. But then you might break the build and someone else is going to have to fix it. Either be willing to wait until the autobuild machines have validated your submit, or wait until the next morning to submit.

One last thing: with Perforce, it's fairly easy to find points in the change history that are stable. You can tell people to never sync to the latest change, but only sync to changes that had been validated by the autobuild. You can even write a script that automatically syncs to the latest validated change.

______________________________________________________