Real reasons (not) to build custom game engines in 2024

It’s 2024, and indie developers still face the same question: should you use a proprietary engine or build your own? Recently, Game Developer published an article about Rez Graham’s talk at GDC 2024 in San Francisco. It hit home for me as someone who built their game on a custom engine instead of using what was already out on the market. During the development of Highfleet, I often questioned whether that was the right call. So, is building your own still worth it, or should you stick with the big-name engines? And if it’s worth it – when?

Custom engines were necessary for pushing performance limits or creating something truly unique. However, with industry-standard engines like Unreal and Unity offering increasingly powerful and flexible tools, the need for custom engines continues to decrease. On the one hand, this creates a degree of uniformity, as many games are developed using the same foundational technology, which gives the games a familiar appearance to players. At the same time, it lowers the barrier to entry, allowing more talented indie developers to bring their ideas to life. The key is determining whether the time and effort required to build a custom engine aligns with your project’s goals.

Three misconceptions about building a custom engine

1. A game engine is just about rendering sprites

One common misconception (that we also initially had) when deciding to build a custom engine is thinking that an engine is only about rendering sprites, animations or shaders. In reality, engines come packed with a wide range of functionality that you’ll have to implement yourself if you're building from scratch, such as:

Underestimation of the scope of features that modern engines provide can lead to months or even years of extra development time, especially when you have to implement things that most proprietary engines already handle out of the box.

2. I know my game’s needs and can build a custom engine around them

This one is incredibly common. You might think that you fully understand what is required in your specific case and create an engine that is the perfect fit. But in reality, game development rarely goes according to plan, especially in indie projects. You might start with a simple concept, but as the project grows, you will need to scale the game world or add more complex effects. That’s when you realize your custom engine wasn’t built for these new demands, and then performance starts to suffer.

Take the render pipeline, for example. Let’s say a small indie team builds a custom 2D engine, and everything runs fine — until they decide to scale the map 10x or add more detailed special effects. Suddenly, the engine struggles, and performance drops. At this point, the team has to either optimize or, in many cases, rewrite parts of the engine.

Maybe the game was built around DirectX 11, but now they need features from DirectX 12, which forces a full migration. Our Highfleet engine went through this process, moving from DirectX 9 to 10, and eventually to 11, with each transition requiring significant rework.

Imagine – a studio sets out to create a 2D platformer, like most traditional ones. The designer outlines the vision, and the developers build a custom 2D engine tailored to the game. But after much of the work is already done, the designer decides the game would look better with 3D trees in the background, casting soft shadows on the 2D characters. For a developer using a proprietary engine, this would still present challenges, but for someone with a custom engine, it could be catastrophic. The engine wasn’t built for 3D rendering, and 3D wasn’t even a possibility in this case, so now the team would need to go back to square one: researching theory, planning the architecture and render pipeline, coding the new functionality, integrating it with the existing engine, testing, optimizing, and testing again.

Even smaller, less drastic, but frequent changes to the requirements for a custom engine can turn it into a never-ending development project, taking valuable time away from refining the actual gameplay.

Screenshot from 2D platformer game Shovel Knight. Image credit: CNET.com

Proprietary engines are built to handle these kinds of changes from the start, allowing you to adapt much faster and focus on what really matters — your game.

3. Building your own engine is faster (or cheaper)

It’s easy to think that building your own engine will save time and money, but that’s rarely the case. To develop a game using an engine like Unity or Unreal Engine 5, you need to invest time learning them. While this can take a while, it's still a much smaller time investment compared to building your own engine from scratch and learning the underlying technology needed for it.

For example, once a programmer understands Unity or UE5, they can quickly implement new features. When a new version of DirectX comes out or new rendering techniques (like RTX) become popular, the engine developers handle the integration of those technologies. All the programmer has to do is update their engine and apply the new features in their game.

When you’re building your own engine, though, updates don’t come automatically. If a new version of DirectX is released or new rendering trends emerge, you have to study the theory behind those technologies, and manually integrate them into your engine, and only then can you start using them in your game. This can be a complex and time-consuming process, especially for a small team. For example, a quick YouTube tutorial on “How to add RTX lighting in UE5” is 40 seconds long. But to add RTX lighting to your own engine, in a way that meets all your game’s needs and is well-optimized, would take much more time and effort.

Another big advantage of proprietary engines is the vast amount of pre-existing assets, shaders, and algorithms made by other developers, which you can easily plug into your game. In contrast, with a custom engine, you’re constantly reinventing the wheel, chasing industry trends, and building everything from the ground up. Even technologies that have existed for years — such as various lighting algorithms, shadows, bloom, bump mapping, diverse post-processing effects, weather systems, water surfaces, etc. — will require manual implementation from your side. Yes, open-source projects with the right licensing, developer forums, and sites like ShaderToy can be incredibly helpful resources, but at the end of the day, it’s up to you to integrate all of that into your game.

Shadertoy.com, Greek Temple shader by Inigo Quilez

This is exactly what Rez Graham was talking about at the GDC conference I mentioned earlier. The key benefit of using proprietary engines is what he called "automatic knowledge." By using a well-established engine, you gain access to institutional knowledge shared by other developers.

And about the cost — yes, Unity and other proprietary engines do charge fees, but those costs are minimal compared to the time it takes to develop your own engine. Unless you’re selling millions of copies, building your own engine doesn’t offer much financial benefit. You could argue that it pays off in the long run, but that’s only true if your engine is robust enough to support multiple projects in the future.

What about libraries?

Indeed, some of the features proprietary engines offer can be achieved in custom engines by using third-party libraries. For example, you can handle physics with libraries like Box2D, and sound with BASS. However, while these libraries can be helpful, you still need to integrate them into your project, and they won’t cover all the needs of a game developer in each area.

The main challenge with using standalone libraries is that they only handle specific tasks, and it’s up to you to ensure everything integrates smoothly. Proprietary engines, on the other hand, already have these systems built in, reducing the need for heavy customization and allowing you to focus more on game development rather than technical integration.

What does this mean for the industry?

As you’ve likely guessed, the challenges mentioned above push most developers to choose proprietary engines, which is a logical and justified decision. However, looking at the bigger picture, this creates a certain degree of technological monopolization within the gaming industry, sadly leading to less project diversity.

Because many developers use the same engines, like Unity, they might end up using the same assets and shaders, causing games to feel similar or even – generic. Additionally, proprietary engines are often optimized for more traditional game types. If a developer is working on something truly unique, there are cases where a custom engine could better serve the project’s needs.

A great example of a justified use of a custom game engine is the well-known Noita. The game has a truly unique concept that likely wouldn’t have worked with any existing proprietary engine, a uniqueness captured in the name of its engine, "Falling Everything." Yet, Noita doesn’t rely on advanced graphical effects, it’s single-player, and it was built exclusively for one platform — PC.

Screenshot from Noita. Image credit: Steam

Top three reasons to build a custom engine

1. You have no other choice

Screenshot from Mount & Blade. Image credit: Steam

Sometimes, you don’t have the option of using an existing engine because the game you want to create relies on new technology that isn’t supported by current engines — or can’t be adapted as easily as you’d like.

This might mean building a large-scale simulation that needs really low-level control to run efficiently (like Dwarf Fortress or Noita). In cases like these, using a standard engine could slow you down because it won’t give you the flexibility to optimize at the level you need.

Another example is when your game mechanics are so unique that no existing engine fits the vision. Games like Mount & Blade needed custom engines because their gameplay doesn’t follow conventional patterns, and trying to make a standard engine work would have meant constantly running into limitations or forcing it to do things it wasn’t originally built for.

 2. You have (really) big plans

Screenshot from The Witcher 3: Wild Hunt. Image credit: dualshockers.com

If you’re thinking long-term and planning to grow your studio, building your own engine could make sense. This isn't just a one-off investment for a single game — it’s something you’ll use across future projects as you scale. But, it’s important to understand the magnitude of resources required to pull this off.

By making your own engine, you can optimize your workflow specifically for the kind of games you’re making. This is the path that CD Projekt RED took. Their REDEngine was developed over many years and was used in hits like The Witcher series and Cyberpunk 2077. However, what’s telling is that even with all the arguments in favor of using their own engine — sufficient human and financial resources, extensive experience in engine development, and significant sales volumes that would make third-party engine licensing fees substantial — CD Projekt RED still decided to retire REDEngine and partner with Epic Games to use Unreal Engine.

3. You want to be technically, legally, and financially independent

If you're using a third-party engine and encounter a bug or missing feature that impacts your game, you often have no choice but to wait for the engine’s developers to fix it. With a custom engine, you can solve these problems immediately as they come up, giving you full control over your development process.

Another downside of proprietary engines is the cost. For example, Unreal Engine 5 (UE5) takes a 5% royalty on gross sales after your first million dollars in revenue. But what’s worse is that the owner of the engine can change the terms of use at any time. This risk became very real during the Unity pricing controversy in late 2023. Although they rolled back some of those changes, the incident highlighted a serious risk for developers.

With a custom engine, you avoid these uncertainties — no unexpected fees, no sudden changes in licensing, and total control over your tech.

Conclusion

Writing your own engine is a complex task. It’s a natural fit for large studios working on AAA projects since they have the manpower and resources to develop, maintain, and update their engines to stay on top of modern technologies. For these studios, the cost of developing their own engine is easily offset, especially by avoiding fees associated with third-party engines. Plus, they’re often working on multiple projects, making the investment in a custom engine even more worthwhile.

Indie developers can also benefit from creating their own engine, particularly if they’re working on a highly specialized single-player game that doesn’t require multiplayer support, multi platform compatibility, or technologies like complex graphical effects, ray tracing, or global illumination. Additionally, indie devs may want to avoid the risks that come with relying on proprietary engines and their changing terms. However, even in these cases, it’s essential to weigh the pros and cons carefully — using a proprietary engine might still be the better option in the end.