informa
/
Production
Features

Where'd It Go? It Was Just Here! Managing Assets for the Next Age of Real-Time Strategy Games

With the increasing number of assets and people involved in game projects, manually maintaining game assets takes on an ever-increasing portion of the project. In order to reduce, and hopefully eliminate, this time from future game projects, Ensemble Studios decided to evaluate its own asset management needs and implement a system for storing and managing all game assets. In this article, Herb Marselas discusses how they translated their asset management needs into an effective asset management system for future games, and the technologies utilized in doing so.

 

Age of Empires II: The Age of Kings consisted of more than 40,000 game and production assets, ranging from bitmaps and textures to 3D models, sounds and music, and source code files. However, with the exception of the source code, managing game assets at Ensemble Studios has largely consisted of editing, copying, and renaming files on local and shared network drives. This process has sometimes resulted in a number of problems, including misplacement, corruption, or accidental loss of game assets. All of these problems result in effort that must be spent finding or re-creating missing assets.

With the increasing number of assets and people involved in game projects, manually maintaining game assets takes on an ever-increasing portion of the project. In order to reduce, and hopefully eliminate, this time from future game projects, Ensemble Studios decided to evaluate its own asset management needs and implement a system for storing and managing all game assets. The purpose of this article is to discuss how we translated our asset management needs into an effective asset management system for future games, and the technologies that we utilized in doing so.

Needs of the Many

During the development of previous Ensemble games, game assets were managed using a directory structure that was centrally located on a network server and copied to the user's workstation as needed. Assets were edited locally with final changes copied back up to the server or edited directly on the server itself. To indicate successive revisions of an asset, incremental numbers were sometimes added to the end of the filename.

This combination of local and server files created confusion when two users both attempted to work on the same shared server asset, or they made different local versions that were later copied back up. It was also difficult to determine which older revisions of assets were truly good enough to keep and which could be thrown away.

Can you count the assets? Hundreds of new assets were created for the Age of Empires II: The Conquerors expansion pack, including those for this Mayan city deep in the Yucatan jungle

However, even with these potential problems, there were several big advantages to a centrally located, directory-based asset system. The first advantage was that the servers hardly ever went down. There were few times during the course of developing Age of Empires II: The Age of Kings (AoK) when server or network problems disrupted access to game assets.

Another definite advantage was simplicity. Every user was already familiar with copying files between Windows folders. Updating an asset on the server, or adding an asset, or putting a new asset into the game was as simple as copying between Windows folders. We were able to use the pros and cons of the current directory-based asset management system to create a list of requirements for the new asset management system that built on the positives but removed the negatives.

Beyond these requirements, a new asset management system had to be able to handle a file of virtually any size, as art and sound files can range in size up to hundreds of megabytes. Another requirement was that the new system should be based on serving the asset to the user's local workstation for editing. This was especially critical, as our main 3D content package, 3D Studio Max, had problems editing files across the network.

The new system also had to be capable of exporting a complete set of game-ready assets from those under asset management. This would remove the onus from the project teams of trying to verify that they really had copied the very latest version of an asset into the appropriate game directory. As long as the latest revision of the asset was in the asset manager, they could be assured it would get in the game.

The final requirement was for a simple workflow system to help the art team keep better track of where assets were in the art pipeline. The workflow system would have three nodes, allowing an asset to be tracked from prototype, to ready for game use, to finalized.

Having established the requirements for the new system (Figure 1), we then faced the looming question: Build or buy?

Figure 1. Ensemble's requirements for their asset management system.

A Single Solution?

Because the programming team was already using Microsoft SourceSafe 6, the first task was to examine the viability of using it as an asset management system for the whole team. While SourceSafe offers a good user interface, a stand-alone version, and an API to create tools to interface with it, a number of concerns arose immediately. The biggest of these were issues of dealing with files, and even a moderate number of users.

With only 15 programmers using it, our SourceSafe system was having performance and consistency problems that resulted in a number of hours spent each month in maintenance and recovery. Working with Microsoft support, we found that the problems we were experiencing affected some number of SourceSafe sites with no discernable cause. Other problems with SourceSafe included a severe performance problem when attempting to check in large assets (anything bigger than about 10MB), even when the files were stored directly, and there was confusion when a user had files checked out on more than one workstation. Although we chose not to use SourceSafe, it did have two features that we could not ignore: its simple user interface and seamless integration with Microsoft Visual C++. These became our guidelines for usability in selecting an asset management system (see Tables 1 and 2).

Table 1: NT-based asset management products
COMPANY
PRODUCT
REPOSITORY
API?
ASSET SERVER?
LIMIT ON TOTAL BYTES OF ASSETS?
Bulldog Two Six Relational Database (RDBMS) Yes Yes Limited only by RBDMS
eMotion Cinebase3 RDBMS Yes Yes Limited only by RBDMS
Filemaker Filemaker Pro RDBMS Yes Yes Limited only by RBDMS
Microsoft Visual SourceSafe 6 File-based Yes No Yes, 4GB
Merant PVCS Proprietary database Yes No Not documented
NxN Ailenbrain RDBMS Yes Yes Limited only by RBDMS
 

 

In our quest to try to keep the whole team using a single asset management tool, we also evaluated Merant PVCS. With PVCS we came to some of the same conclusions that we had reached with SourceSafe. While PVCS is a good tool for simple programming projects, we found it to have many of the same shortcomings as SourceSafe for game asset management. It also lacked a good, intuitive stand-alone UI.


Finally, we looked at NxN's Alienbrain. Alienbrain is more like an asset management toolbox than an off-the-shelf asset manager. This meant that if we did use it, we couldn't just drop it in and go. We would have to learn how to use its interfaces, then build an asset management system on top of them.

Table 2: Is there a single solution?
PRODUCT
PROS
CONS
BOTTOM LINE
SourceSafe
  • Simple, easy-to-use interface
  • Integrated with Microsoft Visual Studio and C++
  • Programmers already use it
  • File-based repository
  • 4GB data limit
  • Can't handle really big files
  • Gets confused when a single user checks out files on multiple workstations
  • Good for teams working on assets < ~10MB in size who don't want or need a back-end server
  • Competent user interface is easily picked up by programmers and nonprogrammers alike
PVCS
  • Integrated with Microsoft Visual Studio and C++
  • User interface not friendly or intuitive
  • Proprietary database
  • Has trouble handling some types of Visual C++ projects
  • Good for teams working on assets < ~10MB in size who don't want or need a back-end server, but who need more than 4GB of asset space
  • Programmers may not be happy about the Visual C++ project problems, and everyone may hate the user interface
Ailenbrain
  • Large API
  • Uses RBDMS for asset repository
  • Large API
  • Stand-alone UI, but no integration with Visual C++ or 3D Studio Max
  • Good for teams that need a back-end asset repository but don't have back-end programming, design and architecture experience
  • The large API may be overwhelming
 

Three other asset management products we looked at briefly were eMotion's Cinebase 3, Bulldog Two.Six, and Filemaker Pro. In general, these are all competent asset management tools. However, they suffer from the same issues that plague the other products we reviewed in greater depth -- a lack of front-end integration and workflow. They are potentially more powerful back-end solutions depending on your need, but in the end a lot of time will be spent creating a custom solution relying on their individual APIs.

The Final Solution

The biggest issues for all these tools, however, are the lack of front-end integration and even the simplest type of workflow. If we were to use any of these asset managers, we would have to learn their APIs and then spend the time creating a workflow system and integration with front-end tools such as 3D Studio Max.

Because we already had expertise in the design and architecture of large-scale databases, we decided to spec out how long the implementation would require if we created an asset management system from scratch. We estimated the time needed to create our system would be approximately six man-weeks of programming time. This did not include time for testing or for major new requirements or features that cropped up during development, testing, and deployment.

The asset management system we ultimately implemented consisted of four major components. The largest component was the relational database on the back-end to track and manage the assets, and the client-side data access layer to it. On the client side, there were two user interface components. One was a plug-in (Figure 3) to integrate the asset management functionality seamlessly with Max, and the other was a stand-alone front end for those users. The final component was an exporter used to create game builds from the latest versions of assets.

Back-End and Client

On the back-end, we chose to use Microsoft SQL Server 7 as the central database (Figure 2). Oracle or Sybase would have worked just as well, and may be required if we increase the number of users significantly. However, SQL Server 7 running under Microsoft Windows NT 4.0 is enough computing power to handle our current and future needs at this time.

Figure 2. Back-end Network and server configuration.

We also decided to store the asset revisions directly inside SQL Server as binary large objects (BLOBs). We could have stored them on a network drive, but we felt that SQL Server could stand the additional load, and storing them in the database provided additional security.

To improve performance of the database, especially with the large number of assets stored inside it, the database server was configured with two ultrawide SCSI controllers. Each controller then supports two ultrawide SCSI hard drives. This configuration allowed us to place the database system files and logs, the small asset management data, the large BLOBs, and the index data on separate drives. This improves performance by allowing the database to spread its access patterns across all four drives. Database security itself is handled directly through the Windows NT domain user authentication system. This means that granting users database access is as simple as adding them to one of the existing NT domain groups.

On the client side, we built a data access layer using ODBC. This gave us several advantages. First, it's simple to maintain. It's also easy to learn. Moreover, there is no dependency on bound data controls (such as MFC), which allows us to actually build an in-game connection to the back-end SQL Server by just adding the additional link to the ODBC libraries.

Figure 3. Integration of client-side components.

All of the asset manager's client functionality was then created in a single layer on top of ODBC and other core technologies (Figure 3). The specifics of the Max plug-in and the stand-alone user interface are then abstracted into separate files on top of the core functionality. This creates a system where all of the code is completely shared through most of the two separate user interfaces.

Regardless of whether the user accesses the asset manager through Max or the stand-alone user interface, both systems present the asset manager to the user with just a Windows Explorer interface (Figure 4). Because all of the users are already familiar with navigating a tree structure, this has significantly reduced the users' learning curve when using the system.

Figure 4. Creating a project folder using Windows Explorer interface.


Keep the Artist in Max

During the development of AoK, the art process consisted of a number of individual steps. Once an artist had created a unit for in-game use, it then had to be rendered out, postprocessed, then added to the game by a lead artist or designer. This meant that it could be a long time between the time artists worked on an asset and when they actually saw it in the game.

As we started implementing the asset management system, our mantra became "keep the artist in Max." In other words, give the artists everything they need to create, manage, and view their models and textures in the game directly in Max. The only time they should have to leave Max is to run Adobe Photoshop, which they can also launch from a button in Max.

This mantra led to the creation of two additional Max plug-ins to support our asset management system. The first was a texture/bitmap browser that allowed the artist to search, view, check out, and use any bitmap asset stored in the asset manager (Figure 5). The texture browser was built as a Max extended viewport plug-in using MFC and Lead Tools imaging control. The Lead Tools provided us a very powerful yet simple-to-use ActiveX component that could be used directly in the MFC dialog box that formed the basis of the texture browser.

Figure 5. 3D Studio Max with RTS3 in-game extended viewport (upper right) and bitmap browser extended viewport (lower right).

The second plug-in was also a Max extended viewport -- it was actually the entire game running inside Max as a viewport. This plug-in was relatively simple to create by making a version of the game that built as a .DLL rather than an .EXE file. Some additional code was required in the window handling code to compensate for the fact that the game was now running as a child window and had to interact with the Max input system.

A small amount of functionality was also required to allow the artist to specify which of the objects in a Max scene should be put into the game. Instead of creating a cumbersome communication system between Max and the game, the artist's scene is exported to a file in a format that the game can load. The game is then told to load the file and display its contents. Because this is the whole game engine running inside Max, the artist can examine the model in the context of other existing game assets and scenarios. As complicated as getting the game engine itself to run within Max was, creating both extended viewports was very straightforward compared to managing Max files and seamlessly integrating with Max itself.

Max files themselves might initially be located in any local or server directory, and they may refer to texture files that exist in any of the Max texture paths. Adding a Max file to the asset manager meant moving the file to its managed directory, then scanning the file for any texture files and moving those to the same directory as well, and finally updating the paths of textures referenced by the Max file.

If the Max file contained in-game models, not only was the Max file added to the asset manager, but an in-game version of the file was generated and stored in parallel. This allowed the back-end export process used to create the game build to be much simpler. It just had to copy data out of the database and store it in files.

Every texture also generated one or two additional files on check-in. A thumbnail image was created and stored to support the texture browser, and textures used in-game automatically generated a texture in the in-game format.

Integrating with Max

Integrating directly into the Max user interface and menu system was one of the hardest challenges we faced in creating a seamless asset management system (Figure 6). Unlike the Microsoft Visual C++ IDE, Max does not have a well-defined interface for asset management tools to plug into. Also, it's not possible to create menu items in Max using the Max SDK or MaxScript.

Figure 6. 3D Studio Max menu integration.

Because we wanted to make the integration as seamless as possible, we had to rely on Win32 programming to manipulate the main Max window and menu system directly. Although this appeared to be a complicated solution, it gave us the flexibility we desired to create new menu entries, and the ability to override and enhance existing Max functionality.

To facilitate this integration, we used a Max general utility plug-in (GUP) to glue Max and the asset management system together. The GUP is one of a number of DLL-based plug-ins that Max supports for modifying or extending existing functionality. Before reading the following explanation of how we integrated the GUP plug-in directly with the Max menu system, you may want to download the source code from the Game Developer web site at http://www.gdmag.com/.

The Max plug-in architecture is based on deriving developer-implemented classes from base Max C++ classes. In this case, our MaxUIModGUP class is derived from the Max GUP class. Max identifies plug-ins in two ways: through their file extension (.GUP for a GUP plug-in), and with a simple class factory called ClassDesc.

When this DLL plug-in is built, we change the file extension from .DLL to .GUP and place it in the MAX PLUGINS\ directory. As Max scans for each set of plug-ins it recognizes, it knows that this plug-in is at least used as a general utility plug-in. Once loaded, Max uses four simple functions to interrogate the DLL. LibDescription returns a simple text description of the plug-in. LibNumberClasses returns the number of class factories (or ClassDescs) in the plug-in. LibVersion is the version of Max that the plug-in will work with. And most importantly, LibClassDesc returns an instantiation of our own derived version of ClassDesc called MaxUIModClassDesc. Max can instantiate our MaxUIModGUP class using the MaxUIModClassDesc::Create member.

In the case of some plug-ins (for example a viewport plug-in), this class factory could be called multiple times. For a GUP it is only called once at startup. This is why we can simplify the code somewhat by using global variables to store our flags and state information.

Once instantiated, the MaxUIModGUP::Start member function is called. MaxUIModGUP has access to the main Max window handle using the inherited member function MaxWnd. Once we get the window handle, we can subclass it with our own message handler (SubclassWndProc) and return success. Subclassing the window ensures that we get the Max window messages before its message handler does.

In implementing our scheme to add new menu entries and enhance existing functions, we only care about two Windows messages: WM_INITMENU and WM_COMMAND. WM_INITMENU is sent to the window when a menu is about to become active. This allows us to look for the main menu, and modify its functionality. However, we must be careful to modify only the main menu, and then to modify it only once.

Because Max creates a number of menus, we use GetMenuItemCount and GetMenuString to make sure the menu we're getting is the main menu. Once we've ascertained that we have the correct menu, the modifyMenu function inserts two new entries into the file menu. For this sample, we're adding a menu option that will force a complete redraw, plus a separator to make the menu look pretty. In the end, DrawMenuBar is called to make sure that the menu is properly updated when it draws.

Back in the SubclassWndProc function, we need only add the WM_COMMAND case to look for the new menu entry we created (IDC_MAXMENUMOD), and process it accordingly. One additional piece of functionality that has been added to this WM_COMMAND handler is a wrapper for the Max File Open menu entry. This code stores off the current Max project filename, calls the default handler to open a new file, and then displays a message box to inform you if a new file has been opened.

Overriding the File Open function in this way may seem a bit dangero

Latest Jobs

Sucker Punch Productions

Bellevue, Washington
08.27.21
Combat Designer

Xbox Graphics

Redmond, Washington
08.27.21
Senior Software Engineer: GPU Compilers

Insomniac Games

Burbank, California
08.27.21
Systems Designer

Deep Silver Volition

Champaign, Illinois
08.27.21
Senior Environment Artist
More Jobs   

CONNECT WITH US

Register for a
Subscribe to
Follow us

Game Developer Account

Game Developer Newsletter

@gamedevdotcom

Register for a

Game Developer Account

Gain full access to resources (events, white paper, webinars, reports, etc)
Single sign-on to all Informa products

Register
Subscribe to

Game Developer Newsletter

Get daily Game Developer top stories every morning straight into your inbox

Subscribe
Follow us

@gamedevdotcom

Follow us @gamedevdotcom to stay up-to-date with the latest news & insider information about events & more