Product Review: Xoreax Software's Incredibuild

Overview

Large render farms of high-end workstations have long been the secret weapon of the artist and as a programmer I've looked on with jealousy at the software provider's support, however minimal it has appeared, of providing distributed rendering systems for packages such as Maya, Kinetix's 3D Studio Max and Pixar's Renderman. This has been exacerbated for me even more by Weta Digital's gushing enthusiasm for their several hundred processor great wall o'renderers that created the large-scale battle scenes in The Two Towers.

Discounting large multi-processor mainframes the ability for a non-mainframe owning developer to compile a single large project across multiple machines has, for a number of years, been the sole domain of Unix & Linux platforms using distributed makefiles. Though difficult to configure and make robust for team use, once the system is up and running it can drastically reduce even the longest compile times. The major downside to it has always been that all systems involved in the build process must have the same compiler versions installed, along with header files, libraries, tools, etc. Hundreds of files, across possibly dozens if not hundreds of machines, that must be kept in lock-step synchronization with each other.

Until recently there has been no viable Microsoft Windows or Microsoft Visual C++ solution so programmers were stuck with their single platform compilation capability, spending, on very large projects, upwards of two hours waiting for a full rebuild of the tool chain or application executable.

But no longer…

Xoreax software released IncrediBuild and changed the way the game is played. IncrediBuild, currently on version 1.133 release and 1.14 beta is their answer to the programmer lusting after the equivalent of distributed makefiles spread over a compilation "render" farm.

What It Does & How It Works

What Xoreax IncrediBuild does is perform a distributed compilation of your C & C++ source files across multiple computers connected via a TCP/IP network - even 802.11b WiFi, even a 56Kbps modem - running Microsoft Windows NT, 2000 or XP - essentially, any Microsoft OS that has the ability to run services in the background that's connected to a network. The incredible part of this - absolutely no puns intended anywhere within this review - is that it can do all of this on machines that do not have Microsoft Developer Studio installed.

For all of this to work Xoreax recommends a minimum of a 200MHz Pentium with 64MB of RAM but I'm sure most game development studios these days would be hard pressed to find a machine that underpowered actually doing any useful office work. A single machine with Microsoft Developer Studio 6.0 along with the Xoreax IncrediBuild DevStudio Add-in installed is all that's required for a development machine, Agents on remote machines and the Coordinator do not require DevStudio to be installed forgoing both the expense of an un-used software package and the requirements of a capable machine. This removes any maintenance headaches that would be caused by ensuring that the same version of compiler, libraries, header files and Service Packs are installed on a diverse network. This also means that the IncrediBuild Agent can be placed on to office machines that are not directly involved with development.

IncrediBuild performs its magic by sharing out the compilation of source files to individual machines running the IncrediBuild Agent. A source file is considered atomic, no further subdivision of labour in compiling it is possible, so the more source files in a project, the better the performance gains witnessed.

Each IncrediBuild Agent, and there can be up to 100 Agents in a build farm though 40 is the recommended maximum, creates a small cache on the HD, and, utilising spare CPU cycles - it runs at the "Idle" priority by default - compiles C & C++ source files just as though they were being compiled on the machine that's running Microsoft Developer Studio. Should an Agent detect that the host processor usage has spiked from user interaction it stops processing and informs the IncrediBuild Coordinator to find another Agent to perform the work. Considering that powerful desktop machines sit idle most of the time, waiting for the user to press another key or move their mouse, there are an awful lot of wasted CPU cycles hovering around, completely under utilised.

The Network Configuration

Setting up an IncrediBuild network is straightforward, install the IncrediBuild Coordinator on a designated machine, and then proceed to load the IncrediBuild Agent -- you must be part of the Administrator or Local Administrator group to be able to use IncrediBuild -- on to as many spare computers as you can find, referring back to the Coordinator either by machine name or IP address, which should preferably be static. The Coordinator & Agents simply run as services in the background, remaining mostly transparent to the regular user. Each machine in the IncrediBuild build farm places a small icon in the system tray to indicate status, and thankfully, Xoreax provided the ability to hide the icon to lessen taskbar clutter.

In DevStudio, IncrediBuild integrates as an "Add-in", providing another Build menu and toolbar. After correct installation, open the largest DevStudio project you can find and click the "Build Current Project" button on the IncrediBuild toolbar. Remapping the F7 (Build Project) key to IncrediBuild makes it completely transparent. The IncrediBuild DevStudio Add-in supplants the standard Build Output window with it's own, providing extra tabs (see Figure 1) showing how many Agents are available, the progress of the build across each Agent, and how long your build is taking. The Agent progress window is also available from the icon in the system tray. Configuration of available machines (see Figure 2) is trivial via the Build Coordinator, giving the administrator the ability to add, what Xoreax refers to as "subscribe", and remove machines from the build farm through an intuitive interface To quote Ron Popeil on late night TV, the software is very much "set it & forget it."

As the build proceeds, the IncrediBuild DevStudio add-in aggregates all of the output, e.g. errors and warnings, (see Figure 3) from individual Agents into a single list that can be navigated normally by double-clicking or pressing the F4 & Shift-F4 keys for "Go to Next Error Tag"/"Go to Previous Error Tag" feature.

I first tested IncrediBuild on a small network of four machines in various configurations and later, on a sunny Saturday morning in January at a local Art College in Los Angeles on a network of newly installed workstations. The IncrediBuild DevStudio Add-in reports all sufficiently idle Agents as a single number in Megahertz, quintessentially an aggregate of all available CPU power, not the most scientific total, but it gives some idea of the network resources at your disposal. It's quite something to see your overall CPU speed reported as approximately 64,000 MHz, or 66 GHz in modern parlance when you have more than thirty 2.1+ GHz CPUs all ready to do your bidding.

The test network makeup was diverse enough to reflect that found in most companies, a speedy, 2.8GHz Pentium 4 Northwood equipped high-end laptop, a low-end, 1.1GHz AMD, a low-end, dual processor 950MHz Pentium 3 - any multi-processor machine will remain underutilised as IncrediBuild only makes use of a single processor though Xoreax has promised to rectify this in the future - and finally, a large collection of mid-range, 2.1GHz Pentium 4 machines.. The entire network utilised standard, 100MBit Ethernet. An 802.11b WiFi test was performed merely for curiosity, and worked flawlessly, but the timing results are not included.

The home network was configured to run through two consumer level LinkSys routers, a later reconfiguration of the routers allowed me to exercise IncrediBuild through a hardware firewall solution with no detrimental effects so it is possible to compile between remote office locations utilising standard TCP/IP. IncrediBuild has an encryption facility to safeguard your source files as they are passed around the network, all of the test runs were performed with it off, the default setting. With IncrediBuild encryption switched on I saw no performance difference.

Other than the College machines, all computers were running ZoneLabs ZoneAlarm and it worked almost flawlessly, ZoneAlarm popped up the usual dialogues asking for confirmation to access the Internet or act as a server. Only on one machine the IncrediBuild Agent refused to start up correctly until after a reboot, this may have had something to do with ZoneAlarm being installed but not running at the time of installation as ZoneAlarm never truly shuts down when you stop the service and that may have confused the Xoreax IncrediBuild installer. There was no problem with IncrediBuild when ZoneAlarm was switched off after installation.

On one machine the IncrediBuild Add-in installation proved troublesome, it was running a trial copy of Visual Assist and IncrediBuild failed to create the toolbar correctly, though the IncrediBuild menu showed up fine and the IncrediBuild system worked correctly. Thoughtfully Xoreax provides an option to re-create the menu and toolbar from the "Settings…" dialogue.

During testing, though not during the timing runs, I went around unplugging various machines from the network to simulate service outage, even so far as disconnecting the submitting machine from the Build Coordinator during a long complex build, and without a hiccough the whole operation kept processing files, albeit at a much reduced rate, amply showing the graceful degradation of service that the IncrediBuild software provides.

The load on Skuld, the 2.8 GHz laptop, was what could be considered average for a development machine - running Adobe Photoshop, two copies of Microsoft Developer Studio 6.0, one performing the actual project compile, WinAmp playing MP3s, two windows of Microsoft Office, one to write this review, one to read over a design document that had a lot of graphics in it, and Outlook checking e-mail in the background - and I can't say I noticed any slow-down when the Agent was compiling a project for a user on another machine. Skuld generally chews through a C++ source file with a full complement of nested "#include" headers at about 1 per second so it's a good base machine to be comparing against cost savings when weighing the speed gains given by a faster CPU and more RAM versus 3rd party software to perform a distributed task across idle development machines. Skuld was configured as both Agent and Coordinator when networked with the thirty College machines with no apparent detrimental effects other than a spike in network utilisation.

Buffy, the 1.1GHz AMD used SETI@Home configured to run constantly, the IncrediBuild Coordinator detected this high CPU utilisation and promptly removed the Agent from the list of those available. Certainly something to be aware of if you are considering installing IncrediBuild in a team environment, team members need to be aware that running processor intensive screen savers, the SETI@Home project or those that show off the capabilities of a GeForce card, is undesirable.

As the build progresses the DevStudio Add-in reports a running total of source files compiled, normally this ticks over at one or two per second on the laptop, on the College network it was ticking over at between thirty and forty source files per second. Every time I watched it I was mesmerised by the rapidly increasing source file counter. I'd suggest to Xoreax that they also add a source line running total too.

Interestingly the IncrediBuild DevStudio Add-in shows the progress of the build graphically, (see Figure 4) a horizontal progress-bar or coloured candlestick represents each file as it compiles on the various agents, the candlestick changing colour depending on the results of the compile, light blue for dependency updates and local tool execution, blue for compiling, green for successful compile, yellow for issued warnings, and red for fatal errors.

IncrediBuild obviously supports simultaneous builds submitted from multiple clients, constantly dividing up the available pool of CPU's until the scheduled jobs are completed.

The Projects Tested

Each project was rebuilt five consecutive times using the "Release" option, which generally takes longer to compile & link, with all optimisations enabled and full warning levels; the individual times - in minutes and seconds - are followed by an average of all the times for each build. The results include both compilation and link time. The number of files in an individual project is stated as an actual count of C++ .CPP source files. Header files, inline files, and template files were not considered.

Project #1

This was a small-size project that consisted of a cross-platform tech demo that was assembled during the Autumn of 2002. It consisted of a number of small classes linked to a pre-compiled Physics SDK supplied by Havok.

Most definitely a "work in progress", this is a project in development and requires constant tweaking of the physics parameters and code, which makes extensive use of DevStudio 6.0's Edit & Continue feature. Sadly an executable created using IncrediBuild is unable to work with this feature and I found myself a little frustrated having re-mapped IncrediBuild to my standard build project key to find myself having to rebuild the project under the standard DevStudio compiler to re-enable Edit & Continue. I really hope that in the near future Xoreax addresses this issue as I find myself unable to live without it when applying the good coding practise of stepping through every source line.

Project #2

This was a mid-sized 3D game project that shipped for Windows 95/98 in 1999 that linked with DirectX, Smacker and the Miles Sound System.

Project #3

This was a large-sized 2D game project developed during 1998/1999 that shipped for Windows 95/98/2000/NT in 1999 that linked with DirectX, the Miles Sound System and a few other libraries. The Developer Studio workspace consisted of three projects, a main executable and two dependent .DLL's that had to be built first. The main executable and one of the .DLL's used GNU Bison to turn script files into executable C++ source code that was generated dynamically at run-time so it was a good exercise of the IncrediBuild system in what would be considered a normal configuration for a large-scale, complex project.

When I was originally working on this project I was developing it on a 200MHz Pentium with 64MB RAM and a geologically slow SCSI hard drive. A complete rebuild was on the order of one or more hours.

Once the dynamically generated C++ files were created by GNU Bison on the main build/submitting machine they were parcelled out as normal by IncrediBuild to the rest of the Agents.

This project required me to increase IncrediBuild's swap-file size from the default 128MB to 400MB to clear up a build error due to a bloated pre-compiled header (.PCH) file. This was only necessary on the submitting machine, i.e. the laptop, so it is not necessary to reconfigure each Agent individually as the dynamics of the project change.

Project #4

This is the open source project M.A.M.E. A popular multi-machine arcade emulator available from http://www.mame.net/, the source code release I used was dated from August, 2001 because it was all I had to hand at the time of testing. Another good exercise of the IncrediBuild application as M.A.M.E. contains eight projects in the workspace, seven of which compile to statically linked libraries to be linked with the final executable.

M.A.M.E. proved anomalous in the timings, they were far lower across the board than I would have expected and I puzzled this for a few minutes until recalling that C code generally compiles faster than C++, and from my intimate knowledge of the source there are also fewer nested "#include" headers than in the other tested projects. The structure of the project dictates that most of the linking takes place in the static libraries - recall that in a distributed build the compilation continues while the creation of the library takes place -- the actual link time is minimal, almost instantaneous, when compared to the overall time.

Project #5

This was a large-sized 2D game project that shipped for Windows 95/98/2000/NT in 2000 that linked with DirectX, the Miles Sound System. Again this project required me to increase IncrediBuild's swap-file size from the default 128MB to 400MB to clear up a build error due to a large pre-compiled header (.PCH) file. Again this generated multiple dependent .DLL's and used GNU Bison for parsing script files.

Each .DLL took considerable time to link itself, 15 seconds for one large library. On a single machine configuration this effectively stalls the build. With the distributed IncrediBuild system the other machines continue to compile C++ source files independently, shaving several valuable seconds from the overall build.

Project #6

This was a large-sized 3D game project that shipped in 2002 for Windows 95/98/2000/XP that linked with DirectX, and various commercial and proprietary libraries. The compile time was considerably slower due to the large volume of dependencies. Again, I had to increase IncrediBuild's swap-file size from the default 128MB to 400MB because of the pre-compiled header (.PCH) file.

During testing of Project #5 on Skuld and the thirty College computers for the first two builds all of the workstations sat idle with the logon screen saver displayed. The last three test builds the room was quickly occupied by arriving students who began loading Adobe Photoshop and Maya, mostly from a shared network drive, web browsers and e-mail clients. Network congestion at this point was probably playing an important role and is reflected in the timings as the 25% spike in build time. IncrediBuild attempts to alleviate some of the network congestion utilising both compression and caching but it is still subject to the extremes of extreme network load caused by many users.

For this project the final executable link constituted most of the build time, accounting for more than half of all the time, but it also proves that the entire project, all 1.6 million lines of it, was rebuilt in less than forty seconds.

Pricing

Xoreax IncrediBuild is priced at $299 per seat, with volume discounts and site licenses available, plus the obligatory heavy discounting for educational institutes. Interested purchasers should visit the Xoreax website for up to date details. I can't honestly say that Xoreax is priced competitively as there are no other products available in the market that compete with it, it's steep for personal use but applied to a sizable team with the expenditure spread over the cost of multiple project P & L's the final costs are significantly diminished.

Xoreax estimates that the cost of a single seat pays for itself in around 3 months. Based on a naïve pricing of 10 licenses of $3,000, or $300 per seat to obtain a client license and then install and configure it is possible to make some quick back of the envelope calculations. Let us suppose that conservatively it reduces build time by 10 minutes per day per programmer, and let us suppose that there are 15 programmer on the team, translating in to around 2 hours per day that is gained company wide, if each programmer costs the company $30/hour on average, factoring in wages, benefits, cost of development systems and office space, which is a very low estimate, so each client license pays for itself in around 50 work days and gaining the development team the equivalent of an extra programmer for 2 weeks in the process, which is infinitely more valuable to me as a Lead Developer than the expenditure. So 50 work days or 3 months is equivalent as to make no difference.

With a lot of software it is difficult to quantify programmer productivity gains; IncrediBuild is a rare application where the gains are prominent other than on small scale projects consisting of two or three programmers. For large projects I would definitely recommend it as a purchase, adding to the arsenal of high-powered tools available to developers.

Summation

For companies that are forward looking enough to have a build monkey - an independent build station - compile times to produce the latest version of the game executable that artists and designers can work with are significantly reduced. From the point where the programmer performs a check-in of source code to a usable build the time is reduced to less than a minute for many projects, instead of the fifteen minutes or more as is usual. This is especially suitable for increased productivity if the programmer must wait for the auto-generated report from the build monkey to confirm everything compiled correctly, translating in to less distractions for programmers. The build monkey would also be the ideal location for the IncrediBuild Coordinator as the machine is usually un-attended for most of the working day and rarely undergoes any configuration changes.

IncrediBuild really shows its strength when you edit the one header file - though there are usually more than one in a large project - that always forces a rebuild of the entire game, every experienced developer has felt that agony. And with more projects using C++ templates it only compounds the rebuild issue, becoming especially relevant towards the end of a production cycle when schedule time grows shorter and rebuild times grow longer as developers begin debugging and making small changes to prepare a shippable product. IncrediBuild also provides benefits after having performed a large "check-in," or "get" of an home grown API that is constantly updated by your in-house library development team such as that used at Angel Studios. As is evident from the build times there are insignificant returns on small sizedprojects, but as the project matures the gains can be astronomical.

IncrediBuild is one of those software packages, like MSVC++ itself, or Visual SourceSafe that just "works" and once you've used it for a few weeks it disappears in to the background and you no longer think of it anymore except for those points I've outlined in the various projects that were tested. You still lament slow compile times, but on a sizable network they are an order of magnitude lower.

The game development world is not made up of Windows only titles, and currently IncrediBuild does not support target platforms other than Windows. Xoreax states that they are addressing issues with the Microsoft Xbox- I suspect it is to do with the Intel Processor Pack and Link Time Optimiser- which it is currently incapable of handling but this still leaves the other two major platforms - Sony PlayStation 2 and Nintendo GameCube - completely unsupported, it would be nice to have the ability to run SN Systems' ProDG GCC-based compiler in a distributed environment too.

I can understand the issues that Xoreax faces with running unknown code on a remote machine as it opens up a whole can of worms for viruses, Trojans and various other abuses but the support of a general system for distributing anything that works on multiple files and is processor intensive, such as resource & asset exports, that can be farmed out to multiple machines and controlled via a makefile would prove incredibly useful.

In studios creating large-scale professional projects with entrenched development pathways, as found at e.g Activision or Angel Studios, the peculiar needs of each project that have many inter-dependencies on assets, particularly those that are dynamically generated, may prove difficult to integrate. It would be relatively easy to trip up the system with a custom or "clever" makefile.

Once Xoreax fix the Microsoft Xbox issue I'll be looking to use it on the next Xbox project. For a full list of IncrediBuild issues consult the web site at http://www.xoreax.com/ Maybe Xoreax will consider porting IncrediBuild to the Microsoft Xbox taking advantage of that 700+ MHz debug station that is sat mostly idle.

Xoreax IncrediBuild does a wonderful job for a Windows or cross-platform product where the majority of the development takes place on a Windows host using standard MSVC++ tools and only later cross-compiled to the target platform(s). It makes you almost wish your project were considerably larger so that you could exercise the distributed compiler even more. During use I kept reaching for the "Rebuild All" button just to see it perform a distributed compile. Microsoft? Can you integrate this in to .NET? Are you listening?

For Further Information

Xoreax IncrediBuild
GNU Make
SN Systems ProDG
ZoneLabs ZoneAlarm