Optimising UE4 Project Builds With Cloud Native Infrastructure And Containers

In the last few months, COVID19 has taken the world by storm. Economy not being the biggest tragedy, from large airlines to small business owner brick-and-mortar corner shops, businesses have entered a crisis situation. Tech companies are not immune. Neither am I, as I’ve been caught right "between jobs” while interviewing for a new position. Amongst other things, open roles are being canceled and entire business models seem to be falling apart, for the time being.

Due to employees having to quarantine themselves at home, "remote work” just became a very hot topic in the tech industry scene. This implies IT departments are scrambling to connect remote workers to their internal and cloud infrastructure while keeping the engine running. Jasper Barnes (Talent Acquisition Manager at Team17 Digital) celebrated a ”milestone week”, as they successfully cope with the situation:

"It's been a milestone week for working from home at Team17 Digital Ltd, we've both onboarded out first new starter remotely into the Nottingham team AND had an offer accepted from our first 100% remote recruitment process.” - (source: LinkedIn)

Companies are currently responding the best way they can to the status quo, as it develops. I wonder how many of them have already realized that the world has effectively changed. Hopefully, the pandemic situation will normalize over the course of the next months, possibly years, but things will not go back to the way they were. Many tech companies that were reluctant to support remote working employees are now in a position in which they are forced to if they wish to survive.

What this means is, while there are benefits and challenges of remote working, they are forced to deal with the more arguable challenges of it, like trusting their remote employees to still be productive. From this, more success cases will arise to reinforce the case towards remote working and that cat is not going back into the bag, haha!

The Case For Unreal Docker Containers

Anyways, while the above is related to the topic, let’s focus on the blog post title. In between job hunting and surviving the world pandemic, I have been looking for ways of applying my skills and potential talent into video games tooling/build/release engineering space. There are interesting problems to solve there. My favorite serious side project, as a Cloud Native Engineer, is research & development into Build Pipelines with Docker Containers for Unreal Engine projects. I maintain somewhat regular contact with several key elements in the video game industry.

One of which, Valentin Galea (Technical Lead at Splash Damage and general super nice guy), had this to say to me recently, regarding the topic of UE4 and Cloud-Native Infrastructure:

"Traditionally UE4 is problematic to be run "in the cloud" due to the huge content demand of a typical AAA game (but even some indie might suffer). If you are doing source compilation alone might be better. Even for that there is are no good solution - especially around having UE4 run in Docker for ex. Another interesting approach would be distributed compilation - it still takes 15 min to compile from scratch even on the best AMD Threadripper machine. Incredibuild works but costs a lot of money, Fastbuild the open source alternative is not well maintained when it comes to UE4.”

Wise words. Incredibuild, while an amazing product with its "burst to the Cloud” capabilities, for example, is indeed costly and still somewhat tied to particular workloads. General workload platforms as Kubernetes, have the advantage of providing a single cost-effective platform that runs both on-premise and private/public cloud, not only for game build jobs, but other kinds of workloads too.

Free Open Source Tools like Terraform allow engineers to collaboratively automate, design, test and manage the lifecycle of build platform infrastructure and other resources. A Terraform Provider is merely a wrapper around any REST API and manage any API resource which has a lifecycle. Tools like Terraform follow the principles of Infrastructure-As-Code and have the most Return Of Investment in Continuous Integration and Delivery environments. Community Terraform modules also enable engineers to iterate faster, which also applies to the creating and lifecycle management of Kubernetes clusters.

Every single project of every single non-game industry related company I’ve worked for in the last three years is running their workloads on Kubernetes. They are putting together "commodity Open Source software" as Kubernetes, Gitlab, Prometheus, Grafana and others, to implement their build and delivery platforms. Federated Kubernetes clusters, as in the ability to control from a single control plane, several isolated clusters, either or both on-premise and in the private/public Cloud, are on the horizon for some and a reality for others.

In the recent past, while chatting with Build Engineers from other AAA studios at UnrealFest 2019, while still in early stages it seemed, with some lack of optimizations, they did mention that Windows Containers were in production use. It is an area that I'm quite looking forward to following the development of. I'm a hardcore believer in cross-industry know-how and applicational concepts and there are lessons to be learned between the general IT and game industry. Particularly where it comes to knowledge sharing, as I've been made aware that within the gaming industry, mostly happens at large conferences like GDC.

One Vision

Every problem is an opportunity for innovation, so, there is potential for the development of a series of "bleeding edge today, industrial-grade tomorrow” distributed compilation tools, like Admiral, that can help the industry consolidate and optimize their workloads. I am sure there are AAA studios working towards this.

My own ambition is to contribute and expand on the work of the Unreal Containers community. I would like to create Open Source services and tools to increase the productivity of, in particular independent developers and small to medium-sized, studios. The best outcome would be to optimize their Build and Release Engineering, so they can focus more resources on what studios can do best, build high-quality amazing games.

Undoubtedly, but not only, a shameless plug, at the moment I’m also trying to move my career forward by getting a job soon while surviving this pandemic. I am in the process of setting up a Patreon account to sustain myself while supporting my game industry R&D. The offering plan is most probably something along the lines of:

Hopefully, I will be able to continue to put an effort on this and give something back to an area that is responsible for an amazingly large number of hours of fun in my life.

That is all for now :) I occasionally also post on my own website and get in touch with me on Twitter with feedback or just a hello!