The Glitz and Glamour of Good Workflows

This post was originally written for Joy Machine's site.

This will likely be the one of the only development logs written without a bunch of screen shots. I'm so sorry. It hurts me more than it hurts you.

In fact, given some of the tasks I've taken on this week: believe me, it hurt me a great deal.

Before pushing much further into development, I had to take a step back at the beginning of this week and get everything straightened-up a bit from the somewhat haphazard approach I had been taking with source control and backups thus far. It's fairly easy, especially with Unreal Engine 4, to get into somewhat of a messy jam as far as source control goes. Given that the engine is, well, the digital equivalent of every elephant on the planet all on the same trampoline (I say that endearingly?), the potential for creating a completely unwieldy and very error-prone workflow for major integrations and changes is unspeakably easy. I know this. I was there around this time last year.

So, I took the time to rectify it! It was fun! So fun! So fun! So fun! So... (okay not even remotely fun). But, I want to go into some of the practices I put into place (and some of the tech and third-party APIs that got added along the way) so others can benefit from my pain. Much like the last time I talked about Perforce (someone actually emailed me to say that they stumbled on that post and it saved them so much work, which was beautiful). 

GITHUB

GitHub didn't always have Git-LFS. It has a soft repository size limit of 1GB, and a much harder limit of 2GB. I have had to take many a repo down in the past because it easily and quickly eclipsed that size limit. Source repositories aren't just about what you have active at any given time -- they're much more about what you have and have done for the entire life of the tracked file set. So, even if your at-the-moment overhead isn't tremendous, that doesn't mean that the 0.5GB asset you checked in accidentally early on isn't still taking up space. And, as far as games go, Git has never really been meant for us. It's much more for the efficient historical tracking of text files and the changes within them over time.

Git-LFS was introduced (I imagine) to address the greatly-increased needs of developers reliant on a large quantity of binary files. And it offers fairly large storage plans at a small additional price in order to handle them.

Problem for me: I just don't like Git-LFS. Regardless of the additional storage size, Git simply does not handle... Sorry, you wouldn't know this, but I just took a half-hour break to figure out why Perforce was unhappy with one of my changelists and refused its submission. I said "but no" and it said "lol we don't care." Video games!

As I was saying: Git simply does not handle massive repositories particularly well, especially when those repositories contain a lot of binary files. Git-LFS may make storage and revisions more manageable from a server perspective, but they absolutely do not ease the user's experience.

PERFORCE

Now, Perforce handles large repositories regardless of their contents like a champ. It has a lot (and I do mean a lot) of quirks as it does so, but at the end of the day it's really the only tool for the job on major game projects. It brings with it a command-line interface unlike any other, and I don't say that as a positive, but the GUI (P4V) is generally well-equipped to handle most day-to-day operations.

That said: what Perforce provides in overall large project management and revision histories, it really, really drops the ball on ease-of-automation (scripting, primarily). I tried for about five painful hours earlier this week to setup a simple on-submit script that would generate release notes for me over time. This is the result of that (pardon the language; it was a lively debate between man and machine):

Point I'm slowly arriving at: all tools have their pros and cons. Also, Perforce, if you're reading this:

THE RESULTING STRUCTURE

I now have three repositories that I manage without any actual amount of painful overlap whatever. Other than the one time I accidentally merged the branch of the NVIDIA GameWorks repo for HairWorks into my engine build. I have a lot of uses for the various GameWorks techs, but there is literally not a single conceivable use that I would have for well-rendered and physically-plausible hair in Steel Harvest. Not one. Unless we do some crossover with Brutal Legend. And, surprisingly, of all the NVIDIA tech I've worked with over the last year and a half, the HairWorks UE4 integration is, by far, the most widespread and complex. C'est la vie?

Anyway, we now have a Perforce server on AWS that will see me through the end of the project with joy and vigor. There is currently a large quantity of assets on their way through the tubes as we speak.

And in addition to setting up a GitHub organization for Joy Machine, I also setup two separate GitHub repositories (super secret ones): one for the Steel Harvest game code and one for the custom fork of Unreal Engine 4. And the interplay between the whole family is just beautiful.

LIST OF OTHER THINGS SO I  CAN GET BACK TO WORK