Game Engines: Past, Present and Future, part 2

By Kevin Normann

Link to part 1

My karate instructor would be proud.  I contained my enthusiasm long enough to complete part 1 and have used the extra time to refine my thoughts and consider feedback from others. I am now ready to delve into what excites me about the future of game development.  The central theme of that future being cloud based technology.  We are already using cloud technology in our applications to varying degrees, but we are not yet using it in our development.  This is about to change.  

Change is inevitable which can be scary for some, but as developers give these new technologies a try, I expect any concerns will be readily replaced with excitement.  Processes in the new world might look and feel different at first, but they address so many of our current pain points that I think the change will quickly feel comfortable.  It will be like taking off a pair of old, stiff work boots and putting on a pair of well fitted running shoes.  Your feet will feel so good that you will be eager to join the race!  

I recognize that these topics are big and subjective.  My hope is that this article will be the beginning of a conversation.  I encourage the reader to follow up with comments so that we can explore these topics deeper.  Okay, let’s get into it!

It’s a Paradigm Shift

There was a day when game development was about figuring out the hardware well enough to build a game on it.  While game design has always been important, designs were more simple back then, but making the games run well on the hardware platforms of the day was very challenging and getting more challenging.  This is what led to video game engine middleware.  Moving from developing directly against the hardware to targeting a video game engine was a big paradigm shift that did a lot to ease the pain points.  It allowed the industry to graduate to a new level.  

I believe the change to cloud-based development will be a bigger paradigm shift and will provide an even greater relief to the pain points that are being felt today.  In fact, it is such a broad reaching change that I think we need to settle on new terminology to help us clearly communicate.

New Terminology

We still use the term “video game engine” to reference current technology even though today’s video game engines come with an “engine” and an “editor” as well as a host of incorporated and optional “middleware” and “plugins”.  Even though the term “video game engine” grew less and less accurate, we still all know generally what we are talking about.  In my first article, I tried to stretch this term even further by saying “next-gen video game engines” and realized that it just doesn’t fit any more.  I tried a few other ways to convey the idea, but always struggled.  I even struggled again in the opening paragraphs of this article.  I think that the word “platform” is more accurate to describe today’s video game engines, and thought about using that, but I had a problem doing so because the already over-used word is being adopted for yet another use in the coming generation.  

In the conversations related to next-gen video game development that I have been a part of, the word “platform” refers to the cloud-based backbone that holds all of the other technologies together.  I have come to accept this meaning for the term myself.  So what then do we call the full technology package?  Since the big new component is the cloud platform itself, we might call the full technology suite a “video game development platform.”  But I think this is inadequate because the introduction of this platform significantly enhances the nature of the technology in ways that this phrase fails to convey; specifically the dramatic improvement of team (and beyond) collaboration capabilities.  It would be more accurate to call it a “video game development ecosystem” to reflect the fact that so many are working and thriving within the same environment at the same time.  But this is long and hard to pronounce so for the rest of this article I am going to use the phrase “cloud-based game development system” or “game development system” to refer to the complete package.  

To restate, in this article, a “game development system” refers to the “engine” and “platform” as well as all “tools” and “plug-ins” built on top of them, the “middleware” that has been incorporated into them, and the broader “ecosystem” that supports development with them.  Does this make sense?

What Excites Me

A big part of what excites me about these new game development systems is the anticipation for how they will make it easier for the average developer to take on bigger challenges and produce more polished games quickly.  Rather than just listing off features, I want the reader to understand my thought process.  To do this I need to talk about what motivates us as developers and the things that keep us from realizing our goals.

Fueling Our Industry

My twin would quickly point me out as the more eager optimist between the two of us. It became clear to me as a young boy that if I was going to realize my great “visions of awesomeness”, I would have to learn to contain that enthusiasm in order to focus on the tasks at hand and let just enough of it out as I went along to fuel my work.  I know that I am not alone.  

I believe it is a child-like enthusiasm that fuels our entire industry at its core.  But if enthusiasm is the fuel, then there must be something consuming this fuel.  What is it?  What is it that consumes our enthusiasm?  What saps us as we push to realize our creative visions?  

The short answer is that it is the obstacles and distractions that inhibit progress toward our goals.  This does not mean that challenges are inherently bad.  In fact, contemplating, addressing, and overcoming challenges boost our enthusiasm.  Yet, there are challenges that do sap our enthusiasm.  They sap our enthusiasm specifically because they provide little or no progress.  I would like to define these kinds of challenges better because they are the kinds of challenges that are specifically addressed by the coming cloud-based solutions.

Productive And Unproductive Challenges

I am reminded of when my brother and I moved to Colorado to a house on the footsteps of the Rocky Mountains.  Looking out the western facing windows at those majestic snow covered peaks was a thrill to wake up to every day.  We were eager to hike them.  But before we could hike them we had to deal with many other obstacles first such as finalizing details of the move, getting jobs, and planning our lives around school.  Within the context of climbing the mountain, these challenges were frustrating because they kept us from our goal.  There were other challenges that were related to the hike such as acquiring maps of the area, gathering our gear, and picking the right day to climb.  These challenges were much more interesting and fun because they were on the path to our goal.

When we finally got to climb the mountain we found it wasn’t as easy as we had expected.  We hiked for hours to only about a thousand feet up from where we started.  It was clear that we were not well-equipped for the journey.  Our shoes were low-cut which allowed dirt and rocks to fall into them, we were wearing shorts which was a problem because our route was full of scrub oak and sharp rocks that scraped our legs, and we only had a flask of water each which had already run out.  We sat down to reflect on our progress.  While the view was inspiring, were we prepared to complete our journey to the top?  How much more effort would it take?  We discovered that while positioned on the side of a mountain, it is hard to see how far you are from the summit.  Were we 1 hour or 8 hours away?  We didn’t know.  We also didn’t know if it would get harder or easier going forward?  If we had made the journey before, we would know what challenges were left, but we hadn’t.  We resolved to head back down and plan better for our next attempt.

There are many analogies that can be made between climbing a mountain and developing a video game.  You have an idea of the challenges that you will face, but you can’t be sure just what it will take to overcome them.  Some challenges are inherently motivational and others are inherently de-motivational; at least within the context of a specific goal.  The nature of these challenges can have a drastic effect on the effectiveness (or health) of the team.  Let’s call challenges that motivate a team toward its goal as “productive” or “healthy” challenges and those that do not as “unproductive” or “unhealthy”.  Among the unproductive variety, I can think of two types: “needless” challenges and, for lack of a better word, what I will call “insurmountable” challenges.

Needless Challenges

What I call “needless” challenges are those that demand our attention yet addressing them doesn’t get us closer to our goal.  These are the “rocks on the road” to where we are going like the rocks that fell into my shoe while hiking.  Because I was wearing the wrong shoes I was frequently dealing with these problems in my hike.  

Game development has these same kinds of problems that aren’t related to the core task.  They sap our enthusiasm because they pull our attention away from our objective if only for a short time.  This includes things like “I just pulled down other people’s changes and now my game won’t run.”  Or, “I can’t complete my task until I can get access to the same game level that Joe is editing right now, so I will have to shelf what I am doing and come back to it later.” Or even worse, “I just spent hours editing files that I forgot to check out and now either Joe or I have to lose our work and start over.” Examples of these kinds of unproductive challenges go on and on and read like a detailed critique of the pains involved in modern game development.

Insurmountable Challenges

By “insurmountable” I mean that for whatever reason a challenge either could not be solved or could not be solved well given the time and resources available to the team.  These challenges force us to scope back, side step, or completely drop features ultimately compromising the game.  Working hard to reach a goal only to come up short can be a tremendous cost to a team’s morale.  

It is easy to imagine what an insurmountable challenge might look like when climbing a mountain.  Related to video game development, this could be an engineer struggling to figure out the right math to pull off a desired shader effect, or an artist who’s tool doesn’t support a required feature.  These challenges might not seem insurmountable to the reader, but to a person lacking the key bit of information or who doesn’t know where to go to acquire the missing information, these can be show stoppers.  Most of the time when a team hits an “insurmountable” challenge, it is only because they don’t have access to a subject matter expert or technology that would turn the unproductive challenge into a productive  one.

Challenges Build On Each Other

I have discussed two kinds of unproductive challenges that sap a team’s enthusiasm; there may be more.  I am not saying that there is a one-for-one relationship between enthusiasm and productivity, but there is definitely a strong correlation.  In fact, I think there is an exponentially negative affect to a team’s enthusiasm as these negative forces increase.  By that same token there is an exponential benefit to the team’s morale that is felt by reducing or removing these negative forces.  Think about it… If a team is not spending its time working on unproductive challenges, then it can spend that time focusing on the productive challenges that move it closer to its goal while boosting the team’s enthusiasm for taking on any remaining challenges.  Does this make sense?

The real hit comes from the fact that unproductive challenges don’t just get in the way of your goal; they stifle creativity.  One minute you are “in the zone” and eager to share your ideas with a colleague, the next minute your game doesn’t run.  You are frustrated, yes.  But what is worse is that now you can’t show your colleague any of the brilliant work you’ve done.  This means all of your brilliant ideas for where to go next will have to wait until later.  Perhaps you can get back to the same clarity of thought on this topic again, but maybe you won’t.

I think we’ve all worked on projects where everything seems to go right and projects where everything seems to go wrong.  I know that when things are going well, putting in extra hours to meet deadlines feels easier and even enjoyable whereas, when their not, the opposite can be true. This has an exponential effect.  Staying on the good side of this dynamic is important to keep a team productive and is directly determined by the process choices that the team makes.

It's Easy to Accept the Status Quo

Monitoring and channeling a team’s enthusiasm is an important part of successful management in any industry, but has special challenges in ours specifically related to technology.  The tools and technology used by a game development team is ripe with opportunity for introducing both productive and unproductive challenges.  I have seen bad decisions lead to processes that place tremendous burden on artists, designers and anyone trying to work with the technology.  In some cases team members will “make do” with a poor process for years not knowing that a simple change to the technology framework could make a significant improvement to their workflow.

I remember joining one team that was struggling so much with errors that showed up whenever multiple people would edit levels simultaneously that the engineers wrote code to place the editor in read-only mode for all but one user at a time.  This was a ninety person team where only one person at a time was allowed to use the game editor.  It was a huge bottleneck. What was even more amazing to learn was that this process change had been made during the development of the first product release six years earlier.  Four other product teams had released four more products all suffering from this terrible workflow.  The team by this point was made up of entirely new people (no wonder), yet all of the new people had accepted the poor process as well.  I immediately had the multiuser feature turned back on and forced the team to hit the problems again.  This stifled productivity and was met with much griping… at first.  However, after about three weeks of focus by the engineers, the editor was made stable and the six year burden was lifted.  Everyone on the team was elated.  It was immediately apparent to them just how bad of a drain on their productivity the old process had been.  This poor decision undoubtedly came at a high cost to the affected projects yet wasn’t addressed for years.  I think this demonstrates how easy it can be for people to accept unproductive challenge as just being a part of the job.

When you extrapolate the effect of poor workflow across an entire team over a large project or over many projects for years as in the example above, it is easy to conceive of the tremendous cost involved even if the impact is hard to measure.  When I take on a lead engineer role, my top priority is to work with the engineers to establish a technology stack that reduces needless challenges while providing the best possible tools for addressing potentially insurmountable challenges and avoiding road blocks for the team.  I believe the new cloud-based game development systems will be a remarkable step forward in this regard.  Small independent developers will find it easier to produce larger, better-polished experiences.  Large teams will feel an even more dramatic improvement.  Large companies will re-organize their development around these technologies to realize the greatest impact of all.

Inefficient Development Process Used Today

Here is a basic rundown of the fundamental development process used today by most of the industry.  This should be familiar to you and will likely trigger many painful memories from your past (if not from earlier today).

In order to work on the game a developer must do the following:

What challenges come from this process?

Wouldn’t it be nice if…

Well, these are the kinds of improvements that are coming with cloud-based game development systems.  Related to a team’s productivity, I think the benefits described above are clear.  What a real-time, cloud-based system provides is a much more streamlined approach to a developer’s process while greatly enhancing the collaboration and communication among its members. The net effect is a shorter, smoother, more engaging road to realizing the team’s vision that allows the team to stay focused on quality and creativity.  When you consider what the cloud has done for shared communication and work in other verticals it starts to become clear that there is tremendous opportunity for improvement in game development.

How else do these improvements impact development?

If these were the only benefits, I would be excited already, but the ramifications are actually much greater:

- Collaborating in real-time inspires and spreads enthusiasm in an infectious way:

 - With cloud-based development, anyone on the planet with an account could be granted permission to view and/or edit a game and with a single click, be where you are…seeing what you are seeing.

- Work in the cloud is sharable.  If one team creates a zone, level, interface, component, AI, system, or experience that is useable by a different team, the expert system and its assets can be easily migrated to the other project.  This has advantages for large companies, independent developers, and especially for middleware providers.

- Work in the cloud can be protected.  Permission levels can be used to protect parts of the game from being modified, accessed or even run in a more intuitive way rather than simply based on file permissions.

- Work in the cloud is persistent. There are many great games that have sold millions of copies that could not be brought back out today without re-creating them from scratch.  Companies like EA have a robust process driven by whole departments for mothballing completed and cancelled projects so that they can be brought back out at a later date if needed.  With the cloud, projects of all sizes will simply sit quietly and wait for any eventual use that may come.  Again, this helps everyone.

- Reviews and comments of various parts of the game and its resources are easy and persistent as well.  This provides insight and inspiration during development and beyond.

- Telemetry data capture and review is trivial, preserved, and provided not just for tracking user experiences at run-time but also during development to help spot bottlenecks in workflows or to aid debugging.

- Your work environment travels with you.  What this means is that you can sit at someone else’s station to help them with their work and, with the click of a button, change the workspace to be exactly like your own including all user settings, window placement, shortcut keys, and expected processes.

I have not covered all of the positive ramifications of cloud-based development.  I am sure that five years from now there will be obvious major benefits that none of us have yet envisioned, but I think you can see where this is going and why I am excited.

Back to my original thesis

The natural life cycle of successful game development platforms is that they are created to address the problems of their day, have a good run of producing well-loved games, but eventually get locked into paradigms and architectures which prove harder and harder to evolve due to an increase in code complexity while supporting more customers.  Simply put, the architectures get sluggish to incorporate new paradigms and over time get locked into virtual cement.  So, while game engine developers don’t stop looking for and incorporating new features, it is more and more costly to make the serious kinds of changes that would be required for revolutionary steps forward.  While I have no doubt that the top engines of today are thinking about how to incorporate new cloud technologies in what they offer, the revolutionary kinds of developments that I am discussing here can only be done effectively when a clean architecture is built around them.

This is why I am excited about new game development systems currently in development.  History and experience tells me that it is these new, upstart companies with what, at the time look like “radical” visions, but later prove to be obvious pivots combined with a good dose of moxie that are best suited to take on established leaders and capitalize on market opportunities to show us all a better way. In this case, it is these new game development systems architected from the ground up around cloud-based paradigms to form clean abstractions that will usher us all to a new level of game development.