Daily news, dev blogs, and stories from Game Developer straight to your inbox
Sponsored By
Sponsored: How Telltale Games handles version control
The type of version control system a studio uses is often a religious issue when it comes to software dev tools. Telltale Games has found much success in using a new VCS: Plastic SCM.
November 28, 2016
8 Min Read
Sponsored by Codice Software
Author: by Francisco Monteverde, CEO, Codice Software
Article brought to you by Codice Software
Telltale is an award-winning independent developer and publisher of video games. They create unique experiences within the rich worlds of Minecraft, The Walking Dead, Batman, Game of Thrones, and many others.
When it comes to software development tools, the version control system (VCS) is often a religious issue. There are pros and cons to every version control system. Telltale has found much success in using Plastic SCM, from Codice Software, as their main VCS.
Telltale operates on a very tight release schedule. Last year they shipped 14 episodes, and through that time, Plastic SCM faithfully kept everything going.
This article explains what are the challenges and demanding requirements a successful game development studio like Telltale put in their source control tool, and how they implemented it.
We would like to thank, Zac Litton (VP Engineering at Telltale) and James Chan, (IT Manager and Source Control Administrator at Telltale) for sharing their views and information with us. A short testimonial is available in the video, "Telltale Games: Managing Growth."
Episodic & Cinematic Games
Telltale has been in business for 10 years and wanted to become the best narrative licensed game company.
The studio expanded from 50 people to 350 over the course of five years, which stressed the tools, the technology, the processes and the pipeline that need to adapt at the same time.
They are doing so with a “digital first” episodic model, making consumers to enjoy and engage in a two-hour experience playing their series. These may have five episodes over the course of six months, in a similar way to TV shows, but opposite to the forty hours’ experience, developed through the course of two years, from traditional big titles.
Their games run on 14 different platforms, and it is challenging to create the same cinematic experiences in all of them, as well as keeping the pace of producing an episode every two months for three large productions at a time as they have now.
As people engage with the license and the content, the users and the community influence the evolution of the story or characters of the next episodes, so the development team needs to be flexible and adapt to it.
Producing games at this peace and respond to user’s feedback at the same time makes Telltale a very special studio.
Close to the audience
Telltale bet on shorter play experiences, episodic games and use digital marketplaces, where content is distributed and accessed to their audience instantaneously.
Key industry trends like supporting mobile games, constant updates and monitoring how users are playing the games, quickly reacting to their feedback and behavior are characteristics the company had in its DNA.
Natural Source Control & Collaboration
As Zac Litton explains, as game developers, you want tools that do their job well and get out your way, so you can concentrate on making exciting games. This is especially true for operational ones such as source control.
The essential aspects that Telltale expects from a source control tool are:
Maintain Fast Release Cycles: they currently release five episodes for three large titles over six months, with a team with 350 plus members. They need a source control with a strong server technology that quickly downloads the gigabytes of information that changes every day.
Tools that are easy to use for Artists: people need to concentrate on making games. Version Control must save time, making them more productive and efficient, download gigabytes of information quickly, do the basic version control operations easy, and getting out of their way.
Support for Remote Partners and Security: in addition to the resilience of the connection, they need to enable access to assets to remote teams and contractors. They all need to work remotely with fast access to the repositories. Administrators need to be able to grant access to designated part of the repositories by setting fine grained ACLs.
A reliable tool provider you can reach out and get help when you need it: this is really important for a piece of software that is going to be used by everyone in the studio every day.
The Setup
The main elements of Telltale’s setup to support their source control operation are:
Large Repositories (3.2TB): the largest repository is currently about 3.2 TB. They need downloads and uploads to be fast. Most artists only need a subset of this, so the version control needs to be able to work only on assets they need, allowing them to precisely select what is going to be downloaded to their workspaces.
Server & Storage: The main server is hosted on a dual socket Xeon E5-2630 with 32GB of RAM that houses both the Plastic SCM Server and the MySQL daemon. Storage is provided over a 10Gb network interface.
High Availability & Backup: a secondary server uses the replication feature in Plastic SCM to keep the two servers in sync, so they have constant synchronization, obtaining High Availability and Backup all in one.
Proxy servers: the international partners run Plastic’s proxy server in their environment to have fast access to the repositories.
What Plastic SCM brings to Telltale
The ability to manage Very Large Repositories (3.2TB): downloads and uploads are now super-fast. The Plastic SCM network protocol has no problems saturating a gigabit network when needed.
Artist Friendly Version Control System: Plastic Gluon UI and workflow is a subsystem designed specifically for easily configurable sparse checkouts, at a per file level, and it is very easy to use.
Locking Capabilities: When an artist needs to work on an asset, like a Maya file, they have Maya plugins that interface with Plastic SCM and it configures the Maya file and all its dependencies into a local workspace so that they can perform an exclusive lock on the file and then commit it later.
Security and Access Control: Plastic SCM has extremely fine grained ACL for the IT staff requirements and security purposes, this is very important when working with external groups.
Support and Fast Access for Remote Partners: the international partners run Plastic’s proxy server in their environment, and the admins there keep the cache hot by syncing with the servers periodically. The experience for their staff feels almost local even on very high latency, low bandwidth connections. (True distributed operation is also possible, but centralized + proxy seems to be better on artist-centric environments where easy operation is a must).
Real Time Replication: Plastic SCM can replicate the repositories on another server in real time, providing High Availability and Backup, all in one.
A partner behind the tool: they need a company that they can reach out and get help and support when they need it, delivering the highest levels of service.
Creating a Source Control for artists
Back in 2014, Telltale and Codice Software teamed up to create Plastic Gluon. Codice did all the engineering, and Telltale provided the user workflows and feature requirements for an artist-centric workflow.
This is not the first time a company tried to create a slimmed down, artist-friendly version control system, however.
The basic workflow supported by Gluon; the one most artists feel comfortable with, is extremely simple: you grab the entire repository in your workspace (or just part of it), you checkout files to work on them and check them in when you are done. This is how most of the studio interfaces with game data. No branching or merging required to excel with the simple and fast workflow artists demand.
And still supporting power users, CI and Git connectivity…
Power users, typically developers with a DVCS background, prefer to stick to a branch per task workflow. Then the traditional Plastic SCM and command line operations offer all the advanced features like branching, merging, replica and shelving.
On the continuous delivery and integration side of things, Plastic provides plugins for Bamboo , TeamCity, and others.
If your CI system is not supported, there is a feature of Plastic called GitServer that offers a Git connector. Therefore, every Git tool can push/pull to Plastic SCM.
Every Plastic SCM server can serve repositories using the Git protocol (Git and HTTP supported). This means that every Git client can push/pull to a Plastic SCM server directly.
Thus, any tool in the Git ecosystem can now be directly used to connect to Plastic SCM using their native Git functionalities.
Engineering teams could work entirely in Git for source code if they are more comfortable with it. From an administrator standpoint, it is one less database to backup and engineers use whatever workflow they are most comfortable with.
Migration tips
According to James Chan, there are a couple of considerations that you should know if you ever plan to migrate large databases over to Plastic SCM.
Plastic SCM supports the Git Fast Export format to import a full Git repo, so as long as your current source control system supports the Fast Export format you are in luck.
If it does not like in their case with Visual Source Safe, they had to convert to Git first and then used the fast export from Git to Plastic.
The problem with doing that with a very large 2.6TB database is the amount of downtime it was going to take.
So here is what James ended up doing: he took a snapshot of VSS with no history and froze the VSS database. Work at the studio could then continue as normal, except without history. Then he added and committed all the files from the snapshot into the Plastic SCM repository. Later, he did the VSS to Git Migration and imported into Plastic SCM, then connected the HEAD of the import to the tail of the snapshot. Voila! Zero downtime migration.
More on Version Control
Not every studio is alike; there are large studios, indie one's and even individual developers and artists. So, if you want more information, references, and use cases, visit Plastic SCM Game Studios case studies.
Check out this video about Plastic SCM @TellTale: Telltale Games: Managing Growth.
Read more about:
Sponsor Resource CenterYou May Also Like
