Game Engines: Past, Present and Future, part 2
By Kevin Normann
My karate instructor would be proud. I contained my enthusiasm long enough to complete part 1 and have used the extra time to refine my thoughts and consider feedback from others. I am now ready to delve into what excites me about the future of game development. The central theme of that future being cloud based technology. We are already using cloud technology in our applications to varying degrees, but we are not yet using it in our development. This is about to change.
Change is inevitable which can be scary for some, but as developers give these new technologies a try, I expect any concerns will be readily replaced with excitement. Processes in the new world might look and feel different at first, but they address so many of our current pain points that I think the change will quickly feel comfortable. It will be like taking off a pair of old, stiff work boots and putting on a pair of well fitted running shoes. Your feet will feel so good that you will be eager to join the race!
I recognize that these topics are big and subjective. My hope is that this article will be the beginning of a conversation. I encourage the reader to follow up with comments so that we can explore these topics deeper. Okay, let’s get into it!
It’s a Paradigm Shift
There was a day when game development was about figuring out the hardware well enough to build a game on it. While game design has always been important, designs were more simple back then, but making the games run well on the hardware platforms of the day was very challenging and getting more challenging. This is what led to video game engine middleware. Moving from developing directly against the hardware to targeting a video game engine was a big paradigm shift that did a lot to ease the pain points. It allowed the industry to graduate to a new level.
I believe the change to cloud-based development will be a bigger paradigm shift and will provide an even greater relief to the pain points that are being felt today. In fact, it is such a broad reaching change that I think we need to settle on new terminology to help us clearly communicate.
We still use the term “video game engine” to reference current technology even though today’s video game engines come with an “engine” and an “editor” as well as a host of incorporated and optional “middleware” and “plugins”. Even though the term “video game engine” grew less and less accurate, we still all know generally what we are talking about. In my first article, I tried to stretch this term even further by saying “next-gen video game engines” and realized that it just doesn’t fit any more. I tried a few other ways to convey the idea, but always struggled. I even struggled again in the opening paragraphs of this article. I think that the word “platform” is more accurate to describe today’s video game engines, and thought about using that, but I had a problem doing so because the already over-used word is being adopted for yet another use in the coming generation.
In the conversations related to next-gen video game development that I have been a part of, the word “platform” refers to the cloud-based backbone that holds all of the other technologies together. I have come to accept this meaning for the term myself. So what then do we call the full technology package? Since the big new component is the cloud platform itself, we might call the full technology suite a “video game development platform.” But I think this is inadequate because the introduction of this platform significantly enhances the nature of the technology in ways that this phrase fails to convey; specifically the dramatic improvement of team (and beyond) collaboration capabilities. It would be more accurate to call it a “video game development ecosystem” to reflect the fact that so many are working and thriving within the same environment at the same time. But this is long and hard to pronounce so for the rest of this article I am going to use the phrase “cloud-based game development system” or “game development system” to refer to the complete package.
To restate, in this article, a “game development system” refers to the “engine” and “platform” as well as all “tools” and “plug-ins” built on top of them, the “middleware” that has been incorporated into them, and the broader “ecosystem” that supports development with them. Does this make sense?
What Excites Me
A big part of what excites me about these new game development systems is the anticipation for how they will make it easier for the average developer to take on bigger challenges and produce more polished games quickly. Rather than just listing off features, I want the reader to understand my thought process. To do this I need to talk about what motivates us as developers and the things that keep us from realizing our goals.
Fueling Our Industry
My twin would quickly point me out as the more eager optimist between the two of us. It became clear to me as a young boy that if I was going to realize my great “visions of awesomeness”, I would have to learn to contain that enthusiasm in order to focus on the tasks at hand and let just enough of it out as I went along to fuel my work. I know that I am not alone.
I believe it is a child-like enthusiasm that fuels our entire industry at its core. But if enthusiasm is the fuel, then there must be something consuming this fuel. What is it? What is it that consumes our enthusiasm? What saps us as we push to realize our creative visions?
The short answer is that it is the obstacles and distractions that inhibit progress toward our goals. This does not mean that challenges are inherently bad. In fact, contemplating, addressing, and overcoming challenges boost our enthusiasm. Yet, there are challenges that do sap our enthusiasm. They sap our enthusiasm specifically because they provide little or no progress. I would like to define these kinds of challenges better because they are the kinds of challenges that are specifically addressed by the coming cloud-based solutions.
I am reminded of when my brother and I moved to Colorado to a house on the footsteps of the Rocky Mountains. Looking out the western facing windows at those majestic snow covered peaks was a thrill to wake up to every day. We were eager to hike them. But before we could hike them we had to deal with many other obstacles first such as finalizing details of the move, getting jobs, and planning our lives around school. Within the context of climbing the mountain, these challenges were frustrating because they kept us from our goal. There were other challenges that were related to the hike such as acquiring maps of the area, gathering our gear, and picking the right day to climb. These challenges were much more interesting and fun because they were on the path to our goal.
When we finally got to climb the mountain we found it wasn’t as easy as we had expected. We hiked for hours to only about a thousand feet up from where we started. It was clear that we were not well-equipped for the journey. Our shoes were low-cut which allowed dirt and rocks to fall into them, we were wearing shorts which was a problem because our route was full of scrub oak and sharp rocks that scraped our legs, and we only had a flask of water each which had already run out. We sat down to reflect on our progress. While the view was inspiring, were we prepared to complete our journey to the top? How much more effort would it take? We discovered that while positioned on the side of a mountain, it is hard to see how far you are from the summit. Were we 1 hour or 8 hours away? We didn’t know. We also didn’t know if it would get harder or easier going forward? If we had made the journey before, we would know what challenges were left, but we hadn’t. We resolved to head back down and plan better for our next attempt.
There are many analogies that can be made between climbing a mountain and developing a video game. You have an idea of the challenges that you will face, but you can’t be sure just what it will take to overcome them. Some challenges are inherently motivational and others are inherently de-motivational; at least within the context of a specific goal. The nature of these challenges can have a drastic effect on the effectiveness (or health) of the team. Let’s call challenges that motivate a team toward its goal as “productive” or “healthy” challenges and those that do not as “unproductive” or “unhealthy”. Among the unproductive variety, I can think of two types: “needless” challenges and, for lack of a better word, what I will call “insurmountable” challenges.
What I call “needless” challenges are those that demand our attention yet addressing them doesn’t get us closer to our goal. These are the “rocks on the road” to where we are going like the rocks that fell into my shoe while hiking. Because I was wearing the wrong shoes I was frequently dealing with these problems in my hike.
Game development has these same kinds of problems that aren’t related to the core task. They sap our enthusiasm because they pull our attention away from our objective if only for a short time. This includes things like “I just pulled down other people’s changes and now my game won’t run.” Or, “I can’t complete my task until I can get access to the same game level that Joe is editing right now, so I will have to shelf what I am doing and come back to it later.” Or even worse, “I just spent hours editing files that I forgot to check out and now either Joe or I have to lose our work and start over.” Examples of these kinds of unproductive challenges go on and on and read like a detailed critique of the pains involved in modern game development.
By “insurmountable” I mean that for whatever reason a challenge either could not be solved or could not be solved well given the time and resources available to the team. These challenges force us to scope back, side step, or completely drop features ultimately compromising the game. Working hard to reach a goal only to come up short can be a tremendous cost to a team’s morale.
It is easy to imagine what an insurmountable challenge might look like when climbing a mountain. Related to video game development, this could be an engineer struggling to figure out the right math to pull off a desired shader effect, or an artist who’s tool doesn’t support a required feature. These challenges might not seem insurmountable to the reader, but to a person lacking the key bit of information or who doesn’t know where to go to acquire the missing information, these can be show stoppers. Most of the time when a team hits an “insurmountable” challenge, it is only because they don’t have access to a subject matter expert or technology that would turn the unproductive challenge into a productive one.
Challenges Build On Each Other
I have discussed two kinds of unproductive challenges that sap a team’s enthusiasm; there may be more. I am not saying that there is a one-for-one relationship between enthusiasm and productivity, but there is definitely a strong correlation. In fact, I think there is an exponentially negative affect to a team’s enthusiasm as these negative forces increase. By that same token there is an exponential benefit to the team’s morale that is felt by reducing or removing these negative forces. Think about it… If a team is not spending its time working on unproductive challenges, then it can spend that time focusing on the productive challenges that move it closer to its goal while boosting the team’s enthusiasm for taking on any remaining challenges. Does this make sense?
The real hit comes from the fact that unproductive challenges don’t just get in the way of your goal; they stifle creativity. One minute you are “in the zone” and eager to share your ideas with a colleague, the next minute your game doesn’t run. You are frustrated, yes. But what is worse is that now you can’t show your colleague any of the brilliant work you’ve done. This means all of your brilliant ideas for where to go next will have to wait until later. Perhaps you can get back to the same clarity of thought on this topic again, but maybe you won’t.
I think we’ve all worked on projects where everything seems to go right and projects where everything seems to go wrong. I know that when things are going well, putting in extra hours to meet deadlines feels easier and even enjoyable whereas, when their not, the opposite can be true. This has an exponential effect. Staying on the good side of this dynamic is important to keep a team productive and is directly determined by the process choices that the team makes.
It's Easy to Accept the Status Quo
Monitoring and channeling a team’s enthusiasm is an important part of successful management in any industry, but has special challenges in ours specifically related to technology. The tools and technology used by a game development team is ripe with opportunity for introducing both productive and unproductive challenges. I have seen bad decisions lead to processes that place tremendous burden on artists, designers and anyone trying to work with the technology. In some cases team members will “make do” with a poor process for years not knowing that a simple change to the technology framework could make a significant improvement to their workflow.
I remember joining one team that was struggling so much with errors that showed up whenever multiple people would edit levels simultaneously that the engineers wrote code to place the editor in read-only mode for all but one user at a time. This was a ninety person team where only one person at a time was allowed to use the game editor. It was a huge bottleneck. What was even more amazing to learn was that this process change had been made during the development of the first product release six years earlier. Four other product teams had released four more products all suffering from this terrible workflow. The team by this point was made up of entirely new people (no wonder), yet all of the new people had accepted the poor process as well. I immediately had the multiuser feature turned back on and forced the team to hit the problems again. This stifled productivity and was met with much griping… at first. However, after about three weeks of focus by the engineers, the editor was made stable and the six year burden was lifted. Everyone on the team was elated. It was immediately apparent to them just how bad of a drain on their productivity the old process had been. This poor decision undoubtedly came at a high cost to the affected projects yet wasn’t addressed for years. I think this demonstrates how easy it can be for people to accept unproductive challenge as just being a part of the job.
When you extrapolate the effect of poor workflow across an entire team over a large project or over many projects for years as in the example above, it is easy to conceive of the tremendous cost involved even if the impact is hard to measure. When I take on a lead engineer role, my top priority is to work with the engineers to establish a technology stack that reduces needless challenges while providing the best possible tools for addressing potentially insurmountable challenges and avoiding road blocks for the team. I believe the new cloud-based game development systems will be a remarkable step forward in this regard. Small independent developers will find it easier to produce larger, better-polished experiences. Large teams will feel an even more dramatic improvement. Large companies will re-organize their development around these technologies to realize the greatest impact of all.
Inefficient Development Process Used Today
Here is a basic rundown of the fundamental development process used today by most of the industry. This should be familiar to you and will likely trigger many painful memories from your past (if not from earlier today).
In order to work on the game a developer must do the following:
- Install the correct version of the game development engine.
- Connect to a source control system which is mapped to a local folder structure where the game will be synced, built, run, and developed.
- Change local files to implement features.
- Review local log files to sort out issues.
- Then, before submitting changes, the developer must reconcile his or her local files against the changes of others that were submitted to source control since the last time the game was synced locally. Once submitted, these changes then trigger other team members to repeat this process to merge, prove and submit their changes.
What challenges come from this process?
- If you are working on more than one project at a time, or different versions of the same project, you may need different versions of the game engine installed on your local machine. Some engines try to make this easier by providing a user interface to manage installed versions, for others engines this is a serious hassle.
- After merging the existing game to prove local changes before submittal, it is possible that other conflicting changes are submitted. In order to be as safe as possible the developer should repeat the process over and over until no changes are updated before submitting his or her local changes. The odds of this happening may be small for small teams, but they grow exponentially for larger teams. This problem becomes significant for teams of say 80 or more. I remember one team specifically that had close to 300 developers where these collisions happened multiple times a day and resulted in tremendous numbers of lost man-hours. Imagine the cost to the company when 300 people cannot work for even a single hour, let alone what could amount to many days per month of a project.
- If you are going to make changes to digital assets or game files (levels, meshes, tables, physics, tweaks, UI, audio, special FX, etc.) you must first verify that no-one else is modifying those same levels or assets. This is where source control can help if you are disciplined in using it. But even with source control non-mergeable work still occurs all too often.
- Even when you coordinate with someone (using source control or not) waiting for the files to be available for edit requires time management and extra thought on the part of the developer. Breaking up a project into smaller pieces can help, but this has its limits and no matter what, it all has to come together somewhere which can produce its own bottleneck.
- If someone else is doing similar work in parallel, there is no convenient way for you to see the work in progress without interrupting them at their work.
- When two people decide to work together on a specific part of the game, sit at the same desk, share a screen over some chat style app, or plan their check-ins, this effort requires coordination and because ultimately only one person can “drive” at a time, is a bottleneck in and of itself.
- Often someone (person A) is having a problem in their local work environment and need the help of someone else (person B), possibly an engineer, to address it. However, the other person (person B) can’t effectively track down the problem within the first person’s (person A’s) environment and wants or needs to set up the same conditions within their own workspace. This effort is an unproductive challenge for both team members which interrupts their work.
Wouldn’t it be nice if…
- Wouldn’t it be nice if the project you were working on itself knew what version of the engine it required and simply built against that version of the engine without any thought from you?
- Wouldn’t it be nice if changes that you made were immediately viewable by others in a group that are all working on the same part of the game so that they could review and give feedback of your work in real-time?
- Wouldn’t it be nice if your changes were automatically reconciled against the work of others in the group with no (or almost no) forethought, post-thought, or any other thought on your part?
- Wouldn’t it be nice if anyone could choose to play the game in any instance at any state during its development even while parts of it are being implemented simply by pushing “play”?
- Wouldn’t it be nice if anyone could sit down on anyone else’s machine and switch it to use their own environment settings with the push of a button so that every window, shortcut, and behavior is immediately familiar to them while they help someone else with their work?
- Wouldn’t it be nice if work was immediately preserved so that changes made a week ago or a month ago could be reviewed to see what the developer did and how they did it (even re-enacted in “real-time”) regardless of what version of the editor or assets was used in that earlier version of the game?
- Wouldn’t it be nice if all of your logs were preserved so that, if you needed to, you could compare the output you were getting today to the output you were getting last week when you didn’t see the problem? Or, better yet, wouldn’t it be great if everyone’s logs were preserved in a common location so you could easily review Joe’s to see how yours might differ from his?
Well, these are the kinds of improvements that are coming with cloud-based game development systems. Related to a team’s productivity, I think the benefits described above are clear. What a real-time, cloud-based system provides is a much more streamlined approach to a developer’s process while greatly enhancing the collaboration and communication among its members. The net effect is a shorter, smoother, more engaging road to realizing the team’s vision that allows the team to stay focused on quality and creativity. When you consider what the cloud has done for shared communication and work in other verticals it starts to become clear that there is tremendous opportunity for improvement in game development.
How else do these improvements impact development?
If these were the only benefits, I would be excited already, but the ramifications are actually much greater:
- Collaborating in real-time inspires and spreads enthusiasm in an infectious way:
- Watching other parts of the game make progress in real-time while working on your own tasks is motivational. Most developers haven’t experienced this or only experienced it in small doses because the systems of today don’t provide for this level of cohesiveness. In cloud-based game development systems, when someone texts the group to say “Hey, look what Joe just added!” developers all over the team can jump over and look at that inspirational coolness from their own desks within their own work environments with minimal interruption.
- There is something more that is communicated when watching someone construct a part of the game in real-time that you don’t get when you only see the final result. For me a good analogy is when you look at a few rendered frames of an animation compared to seeing the full animation stream played back. Watching someone work in real-time conveys a thought process, suggests alternatives, and invites rapid feedback that can lead to better ideas and a better product.
- With cloud-based development, anyone on the planet with an account could be granted permission to view and/or edit a game and with a single click, be where you are…seeing what you are seeing.
- For small development teams, this means that if you can find an expert to help you past your road-block, the bar for that person to jump in and help you is two orders of magnitude easier than in the non-cloud based approach.
- For large corporations, this means sharing expertise across the company becomes not just more manageable, but natural and intuitive. I used to teach a class to new hires at EA University each week. With cloud-based development systems, the class doesn’t have to end at the door. Students could immediately apply what they learned in class on their own project teams known that, if they hit a snag, they could get my immediate help without causing either of us headache. From my desk I could jump into what they were doing as though I was a member of their team with the game already installed on my machine and with of their current work available to me at my desk.
- Those working on closed platform projects (such as Xbox, or PlayStation) would have a much easier time getting help from first-party support teams on their specific issues because, again, anyone can be invited into the project to see challenges first hand and render fast feedback or even author fixes.
- Work in the cloud is sharable. If one team creates a zone, level, interface, component, AI, system, or experience that is useable by a different team, the expert system and its assets can be easily migrated to the other project. This has advantages for large companies, independent developers, and especially for middleware providers.
- Work in the cloud can be protected. Permission levels can be used to protect parts of the game from being modified, accessed or even run in a more intuitive way rather than simply based on file permissions.
- Work in the cloud is persistent. There are many great games that have sold millions of copies that could not be brought back out today without re-creating them from scratch. Companies like EA have a robust process driven by whole departments for mothballing completed and cancelled projects so that they can be brought back out at a later date if needed. With the cloud, projects of all sizes will simply sit quietly and wait for any eventual use that may come. Again, this helps everyone.
- Reviews and comments of various parts of the game and its resources are easy and persistent as well. This provides insight and inspiration during development and beyond.
- Telemetry data capture and review is trivial, preserved, and provided not just for tracking user experiences at run-time but also during development to help spot bottlenecks in workflows or to aid debugging.
- Your work environment travels with you. What this means is that you can sit at someone else’s station to help them with their work and, with the click of a button, change the workspace to be exactly like your own including all user settings, window placement, shortcut keys, and expected processes.
I have not covered all of the positive ramifications of cloud-based development. I am sure that five years from now there will be obvious major benefits that none of us have yet envisioned, but I think you can see where this is going and why I am excited.
Back to my original thesis
The natural life cycle of successful game development platforms is that they are created to address the problems of their day, have a good run of producing well-loved games, but eventually get locked into paradigms and architectures which prove harder and harder to evolve due to an increase in code complexity while supporting more customers. Simply put, the architectures get sluggish to incorporate new paradigms and over time get locked into virtual cement. So, while game engine developers don’t stop looking for and incorporating new features, it is more and more costly to make the serious kinds of changes that would be required for revolutionary steps forward. While I have no doubt that the top engines of today are thinking about how to incorporate new cloud technologies in what they offer, the revolutionary kinds of developments that I am discussing here can only be done effectively when a clean architecture is built around them.
This is why I am excited about new game development systems currently in development. History and experience tells me that it is these new, upstart companies with what, at the time look like “radical” visions, but later prove to be obvious pivots combined with a good dose of moxie that are best suited to take on established leaders and capitalize on market opportunities to show us all a better way. In this case, it is these new game development systems architected from the ground up around cloud-based paradigms to form clean abstractions that will usher us all to a new level of game development.