informa
/
Programming
News

Reflection in C++: The simple implementation of Splinter Cell

In this reprinted #altdevblogaday in-depth piece, Lionhead and Ubisoft veteran Don Williamson shares the simple reflection API developed to replace Unreal's object model in Splinter Cell: Conviction.
[In this reprinted #altdevblogaday in-depth piece, Lionhead and Ubisoft veteran Don Williamson shares the simple reflection API developed to replace Unreal's object model in Splinter Cell: Conviction.] The first part in my Reflection in C++ series gave a high level overview of many of the possibilities open to you when adding reflection to your games. In this second part I'm going to go into details and cover the system used to aid the rendering engine in Splinter Cell: Conviction (SC5). The motivation for the development of the SC5 engine was a clean break from the past. We were working with a very, very large code base that used Unreal 2.5 with many years of modifications and rewrites. While immensely stable, visually very good looking and a code base you could bet a few million dollars on, it slowed the development of new techniques required to push the SC franchise onto the next generation of consoles (the XBox 360, circa 2005). Compile times were painfully slow, link times were in the order of minutes and it suffered from the classic Unreal issue of requiring huge rebuilds whenever you changed a .uc script file (used to define your interfaces to the editor, among other things – partially solved in Unreal 3). There were many levels of pipeline and engine indirection added to ship titles that were slowing down the iteration and development of new techniques, simultaneously contributing to a lack of runtime performance on the target platform. After a couple of months, we were at a few seconds for compile/link on the PC, sub-1 second single file iteration times, vastly simpler/faster import pipelines, multi-threaded performance that was orders of magnitude faster on the Xbox 360 than the old engine, the ability to live edit C++ rendering code and no compile dependency on Unreal. As a result we lost a few key technologies along the way that would have helped immensely, such as the ability to carve indoor space up using BSP volumes. We also ended up recreating some already existing features such as light association control and skeletal attachments. But the full story is something for another day and Stephen Hill's fantastic GDC 2010 piece on the development of some of the rendering technologies can give a bit more insight (Rendering with Conviction). This post will cover just the reflection API developed to replace Unreal's object model. The implementation was very simple, written in a couple of days, constrained to a single cpp/h file pair and slowly grew as the needs of the engine evolved. It also helped us develop the new engine while the old engine was still active, supporting 50 odd developers and keeping game progress undisturbed. It is my hope that I can demonstrate how simple (and sometimes naive) solutions can help ship great games, as long as you're willing to rub shoulders with some pretty big limitations. An example of such a system is Reflectabit, written a few years ago in my spare time. It's unfinished but contains enough to serve as a tutorial of sorts. This post is from memory as I don't have the code to hand anymore. The Type ID The first task in any reflection API is defining what a "type id" is: how can you reference types in code? For SC5 we needed a type ID that:
  • Could reference built-in types (int, char, etc) as well as custom class types.
  • Was unique for each type.
  • Was persistent between program invocations.
  • Could be used for serialization.
The simplest of solutions is to use an enum:
  enum TypeID
  {
    TYPE_INT,
    TYPE_CHAR,
    TYPE_MYTYPE,
    // ...
  };
Each time you add a new type, you add it to the end of the enum. This needs a means of mapping a C++ type to its enum:
  // Each reflected type must specialize this
  template <typename TYPE>
  inline TypeID GetTypeID()
  {
    // Compile-time assert: Type not implemented (avoiding a return value is enough, really)
  };

  // Specialization examples
  inline template <> TypeID GetTypeID<int>() { return TYPE_INT; }
  inline template <> TypeID GetTypeID<char>() { return TYPE_CHAR; }
  inline template <> TypeID GetTypeID<MyType>() { return TYPE_MYTYPE; }
This is not particularly extensible or maintainable in larger projects: if you're working on a changelist that adds a new type and you are using it for serialization, any incoming changes from members of your own team have the potential to clobber all your data and make the act of submission a chore that is potentially very dangerous. Type IDs will accrue over time and you must ensure that types are added at the end of the list. It's not suitable for a reflection API that ships as part of a 3rd party library that expects its client code to add its own type, although there are many examples of such systems in decades old code (WM_APP). One good benefit of this method is that you can immediately see the type name in a debugger when inspecting values of type TypeID. However, you can't see them at runtime unless you add a means of also mapping the enum value to a string. Of course, there are ways to do this, for example:
  // TypeIDs.inc:

  // List all Type IDs
  TYPEID(TYPE_INT)
  TYPEID(TYPE_CHAR)
  TYPEID(TYPE_MYTYPE)

  // TypeIDs.h:

  // Build the enum table
  #define TYPEID(type) type,
  enum TypeID
  {
    #include "TypeIDs.inc"
  };

  // Use the pre-processor "stringiser" operator to specify the name of each name
  #undef TYPEID
  #define TYPEID(type) #type,
  const char* g_TypeNames =
  {
    #include "TypeIDs.inc"
  };
This is a well-used technique in many shipping C/C++ products where its variants have been branded X Macros. At this point it's all getting a bit messy/overkill; we ruled it out on SC5 without much thought as there was a much simpler solution:
  // Each reflected type must specialize this
  template <typename TYPE>
  const char* GetTypeName()
  {
    // Compile-time assert: Type not implemented
  }

  // Specialization examples
  inline template <> const char* GetTypeName<int>() { return "int"; }
  inline template <> const char* GetTypeName<char>() { return "char"; }
  inline template <> const char* GetTypeName<MyType>() { return "MyType"; }

  template <typename TYPE>
  u32 GetTypeID()
  {
    // Calculates the string hash once and then caches it for further use
    static int type_id = CalcStringHash(GetTypeName<TYPE>());
    return type_id;
  };
As long as your hash function is good this is a great way of defining your type ID support:
  • Practically, you are guaranteed unique, persistent IDs as long as your typenames are unique (you can prefix namespaces if you like).
  • Types can be added in isolation; all you need to do is implement your GetTypeName alongside its type in a header file.
  • The type names are readily available in the debugger and are part of your executable.
The choice of hash function is important but don't overthink the issue. SC5 used CRC32 for both type names and object names but this is technically not the purpose of CRC32 (it's a trivial method of error detection in data packets). We used it for hashing of both type and object names and while I was working on the project we had one hash collision with some materials that was easily sorted with a rename (collisions were tracked offline in a MySQL database). These days I use MurmurHash3, however DJB2 is a nice simple implementation and the field is steadily progressing (e.g. see CityHash). Whatever you choose, collision visibility is essential as collisions can cause very subtle data integrity issues. Before going any further, a small review of some other methods of generating a type ID may be of interest. The first is to use C++ RTTI to replace GetTypeName:
  template <typename TYPE>
  const char* GetTypeName()
  {
    return typeid(TYPE).name();
  }
Note that this is not using any runtime aspects of C++ RTTI support beyond calling a function in the type_info object returned, which needs to store its information somewhere in memory. Technically this means you should not be penalized for its use at runtime (it doesn't alter the size of any of your objects) but I've only tested this on MSVC platforms. This is a remarkably simple solution that doesn't require you to specialize GetTypeName. One issue you can encounter is that RTTI is a very loosely standardized feature of C++ and different platforms may return different names for each type. I believe GCC mangles the result in some way, although none of these issues are insurmountable. Curiously, if you disable RTTI in an MSVC project, typeid still works, demonstrating the theory of no runtime penalty. However, other compilers such as GCC fail to compile. Another non-standard side-step involves the use of the pre-defined __FUNCTION__ (C99) macro or whatever equivalent your compiler most likely has these days:
  template <typename TYPE>
  const char* GetTypeName()
  {
    // GCC's equivalent is __PRETTY_FUNCTION__
    return __FUNCSIG__;
  }
This is MSVC-specific but the output is:
  const char *__cdecl GetTypeName<struct MyType>(void)
An important realization is that this string is unique if your type name is unique so you don't really need to change it. If that bothers you, however, you can do a quick parse of the string and cache it locally in a static char buffer, or similar. A final method I'd like to cover is one which has caught me out in a moment of error:
  #define GetTypeName(type) #type
It's alluringly simple and works, except for one fatal flaw; it can't be used within templates:
  template <typename TYPE>
  void SomeUtilFunction()
  {
    // Returns the string "TYPE"
    const char* type_name = GetTypeName(TYPE);
  }
Types in memory The next step is in defining how types are represented in memory, their dynamic retrieval and how they can be used to create objects of that type. This involved a few structures:
  // Used for both type names and object names
  struct Name
  {
    unsigned int hash;
    const char* text;
  };

  // Function types for the constructor and destructor of registered types
  typedef void (*ConstructObjectFunc)(void*);
  typedef void (*DestructObjectFunc)(void*);

  // The basic type representation
  struct Type
  {
    // Parent type database
    class TypeDB* type_db;

    // Scoped C++ name of the type
    Name name;

    // Pointers to the constructor and destructor functions
    ConstructObjectFunc constructor;
    DestructObjectFunc destructor;

    // Result of sizeof(type) operation
    size_t size;
  };

  // A big registry of all types in the game with methods to manipulate them
  class TypeDB
  {
  public:
    // Example methods; implementations discussed later
    Type& CreateType(Name name);
    Type* GetType(Name name);
  private:
    typedef std::map<Name, Type*> TypeMap;
    TypeMap m_Types;
  };
We didn't use the STL to define our types; I'm using it above to demonstrate intent. The Name type always stored both the null-terminated string pointer and hash of that string. Type names were always present, stored in a read-only segment of memory when the compiler encounters any calls to GetTypeName. Object names were never stored in memory. Instead, they were stored in a MySQL database which was queried by a Visual Studio debugger plugin whenever it wanted to display a name in the watch window. This was a great way of always having object names present in console builds without consuming runtime memory, even in our most final release builds. Types are created dynamically as part of game initialization and a simple implementation of CreateType would be:
  template <typename TYPE>
  Type& CreateType(Name name)
  {
    // Only allocate the type once (GetType will call CreateType if the type doesn't exist)
    Type* type = 0;
    TypeMap::iterator type_i = m_Types.find(name);
    if (type_i == m_Types.end())
    {
      type = new Type;
      m_Types = type;
    }
    else
    {
      type = i->second;
    }

    // Apply type properties
    type->type_db = this;
    type->name = name;
    type->size = sizeof(TYPE);
    return *type;
  }

  // Example registration on initialization
  TypeDB db;
  db.CreateType<MyType>();
This gives enough information to be able to allocate enough space for objects of a given type but a means of constructing/destructing that object has yet to be defined. In C++ you can't create function pointers to the constructor or destructor (see C++98 12.1 for more info) but given a memory address, they can be called:
  template <typename TYPE> void ConstructObject(void* object)
  {
    // Use placement new to call the constructor
    new (object) TYPE;
  }
  template <typename TYPE> void DestructObject(void* object)
  {
    // Explicit call of the destructor
    ((TYPE*)object)->TYPE::~TYPE();
  }
This now allows CreateType to fully define Type:
  template <typename TYPE>
  Type* CreateType(NAME name)
  {
    // ... alloc type ...

    // Apply type properties
    type->size = sizeof(TYPE);
    type->constructor = ConstructObject<TYPE>;
    type->destructor = DestructObject<TYPE>;
    return *type;
  }
This is enough information to dynamically create objects of a given type, which will be discussed later. Finally, the default type database needs to register all C++ types that were supported in its constructor:
  TypeDB::TypeDB()
  {

Latest Jobs

Sucker Punch Productions

Bellevue, Washington
08.27.21
Combat Designer

Xbox Graphics

Redmond, Washington
08.27.21
Senior Software Engineer: GPU Compilers

Insomniac Games

Burbank, California
08.27.21
Systems Designer

Deep Silver Volition

Champaign, Illinois
08.27.21
Senior Environment Artist
More Jobs   

CONNECT WITH US

Register for a
Subscribe to
Follow us

Game Developer Account

Game Developer Newsletter

@gamedevdotcom

Register for a

Game Developer Account

Gain full access to resources (events, white paper, webinars, reports, etc)
Single sign-on to all Informa products

Register
Subscribe to

Game Developer Newsletter

Get daily Game Developer top stories every morning straight into your inbox

Subscribe
Follow us

@gamedevdotcom

Follow us @gamedevdotcom to stay up-to-date with the latest news & insider information about events & more