- With Matt "Queso" Niederberger, Jerry Tang, and Warren Hwang

Intro

Since the release of the Guns of Icarus series on Steam and PC, we’ve been continually updating the player experience. From the original PvP only edition to Guns of Icarus Alliance, the latest and complete edition with PvE and faction warfare infused, we’ve learned a ton over the last few years. Yet, since the very beginning, we had always aimed to go beyond PC and take the game to console. Bringing Guns of Icarus Alliance to the PlayStation 4 was a dream that we’d been working toward since the start. Gaming on console had been, and still is, a significant part of our lives, and we still get a bit starry eyed with the realization that our game is going to be on PS4. The journey to get to this point had been a winding and arduous one for us, learning and overcoming myriads of challenges along the way. With the game now set to release on PS4, we wanted to take a bit of time to reflect and share some of the lessons we learned on this journey.

The main tenets of Guns of Icarus were teamwork and bringing people together. Therefore, cross-platform play was something we aspired to from the very beginning. Not only did we want people from console and PC to play in the same match, we also wanted them to fully communicate through all the means available, such as voice, text, and command signals. Realizing the vision of fully integrated cross-platform play - between PS4 and PC (Win/Mac/Linux) on Steam and other platforms - brought about another set of novel challenges that we had to find creative ways to overcome, as there hadn't really been precedents that we could follow or emulate. The purpose of this write up is to recount the different sets of issues - ranging from cross-platform development, performance, and user interface considerations to engine and platform specific challenges - and summarize how we overcame them. Hopefully this would be a useful reference for anyone embarking on multiplayer projects and thinking about cross-platform support.  

Cross-Platform

Limitations and Advantages

The decision to go cross-platform between console and PC was a complicated one to make for any multiplayer title. While we were guided by our vision for the game, this decision did come with a long list of technical challenges. One lesson learned was that we could not make design changes independent of platform, meaning that if we discovered challenges that were different for mouse and keyboard users, or find out something was common convention on one platform but not the other, we had to find a unified solution that worked for both. This also limited the way we solve other problems such as performance. Essentially, we realized that we had to make a game that was optimized for two unique platforms. This meant that our technology had to be flexible, and that we could not go all in on any technologies or hardware that might not exist on other platforms. We basically had to optimize the game all over again for each platform we wanted to release on. And for Guns of Icarus, moving to console meant that we had to take a step back and change some of the ways we developed new contents and features. We moreover no longer had as much freedom to push updates whenever we felt like it, and we had to be careful adding features that didn't fit on both platforms. In addition, because our console odyssey took such a long time, we had to re-examine our UI/UX and make extensive redesigns with controller navigation in mind. This had consequences in terms of community and player reception; a portion of our players who were used to or entrenched in our old navigational flow did not take kindly in our redesign efforts. The UI redesign then became also an exercise in community and crisis management.

With all this being said, keeping a game cross-platform ultimately had a number of advantages as well. With Guns of Icarus, we had always pointed everyone from different regions or operating systems to the same match on the same server. For better or worse, we wanted to make a case to bring players together in that way, and so we’ve architected our backend to reflect that. In a way, with supporting cross-platform for console, this was merely a platform extension to our back end. However, in our opinion, keeping a single user base was generally great for a game. It was a lot easier to maintain one group of active players than two. And in any multiplayer project built on teamwork, letting players from different platforms group up and play could mean more long term and invested players. Cross-platform support also meant maintaining a single version of the game instead of multiple discrete versions that could potentially be even more of a version control nightmare. From our end, only having to maintain one version of the game was a huge benefit. We were not balancing every item and map twice. We weren’t writing all our code twice, and it was all the same game. Most importantly though, doing cross-platform play was just REALLY fulfilling. It was awesome to see everyone playing together. Being able to say “come play this game with me” and not having to worry about having friends on different systems was a dream come true to a good portion of players in our game.

Cross-Platform Voice

Guns of Icarus started as a PC game (Windows, Mac, Linux) that integrated Steam’s API, and therefore we implemented our chat on top of Steam’s Voice Chat system, which was done via peer to peer networking. However, this voice system required Steam running for all players, and cannot be used for PS4 players.

PS4 has its own voice chat SDK. Unity wrapped it with Net Chat Plugin. It provided some basic voice chat features, but not enough to cover all the voice chat features Guns provided. And the bigger issue was, it cannot be used for PC players.

In order to support cross-platform voice, the first issue came to our mind was whether there would be a 3rd party voice chat solution that can work for both PC and PS4 players. Without it, we could still support both PC/PS4 players playing together, but they would likely not be able to use voice chat to communicate. Given the high demand of collaboration in the game, it would not be ideal. Along the way, we built contingencies, such as a voice signaling/command system and a more nuanced AI crew command system. However, these were meant to be plan Bs, and the elusive grail of cross-platform voice chat was what we were after.

At the time, there were very few cross platform games, and even fewer games that supported players from both platforms to play together, so we could not find how other games worked.  There were no ready-made 3rd party solutions, and our own attempts to build this voice chat bridge did not get very far. The task seemed daunting to impossible for a long while.  

Last year, we found Photon Voice (developed by Exit Games) as a voice chat solution that works on PC. Since we used Photon as our own game server, we had some experience with it, and knew that its networking package works on PS4. We started talking with Photon’s team and realized that this could be a way forward to make cross-platform voice a reality.  

The package itself was quite straightforward and easy to use. However, Guns of Icarus’s existing voice chat system was complex. We designed our in-match voice chat based on a multi-channel structure, taking a bit of inspiration from the military. Crews onboard the same ship could only talk to each other. The captain of each ship had an additional channel to talk to other captains on the same team. This way, we helped manage the flow of information so that players were not overwhelmed and inundated with everyone’s voices, which could be highly chaotic.  

As a result, integration took us some time, as we needed to support players chatting with players in the same match lobby, as well as players in the same party, which could be two different groups of people. Also, in match lobby we supported match chat, team chat, captain chat and crew chat, and recipients needed to know not only who was talking but which channel that player was using. These were not supported by the original code provided by Photon Voice, and therefore we tweaked some of its source code at higher level. The solution was flexible enough that we managed to make it work exactly like what Steam voice did, and it worked cross platform!  

However, there was a caveat that we didn’t know until very much later, found by Sony’s meticulous FQA team, that PS4 players were not able to output only the chat audio to the headphone, but keeping the other music/sound to the TV. The feature was supported by Unity’s Net Chat Plugin by routing the incoming voice data to PS4’s voice device, whereas Photon Voice handled the voice with Unity’s AudioSource class, so that they could not be separated with the rest of the sounds in the system. This was something we reported to the Photon team, and it was something that they were actively working to support. Hopefully soon, everyone using Photon as the solution would be able to navigate this issue much easier. One thing great about diving into the unknown abyss was that we had the opportunity to work with great people such as the Photon team to make contributions to improving their product and platform. This was one of the more rewarding part of our development.

Navigating Unity

The Bare Minimum: Getting the Game to Run on PS4

AoT Compilation Issues

Guns of Icarus had a lot of generic object instantiation (like List<X>) at runtime (using System.Reflect), which caused us a ton of headaches, because the PS4 didn’t have an actual .Net runtime environment like PC, and would just throw out a bunch of AoT compilation errors at runtime.

Our initial solution was to declare all those used generic classes explicitly in one of our scripts. So that Unity would know those classes exist and build them properly. However, there were so many of them and it was very difficult to list them all, not to mention making and maintaining that list was extremely error prone.

The better solution for us was “Managed bytecode stripping,” described in https://docs.unity3d.com/Manual/IL2CPP-BytecodeStripping.html. First, from Unity’s Player Setting, under PS4 tab / Other settings, set “Scripting Backend” to IL2CPP, with IL2CPP optimization level set to “Optimized Compile, Unused code removed, Optimized Link.” Then put a link.xml file somewhere in one’s script folder, include all the assemblies/classes that would be used in one’s client program. This should deal with most of the issues.

Shader Porting

Early on in the port process, we of course ran into issues due to the PS4’s unique rendering environment, which is not quite OpenGL or DirectX. Guns of Icarus used heavily customized shaders, and all of these had to be adapted to the new platform in small ways. This was largely simple grunt work (using the correct semantics and function names, e.g. SV_POSITION instead of POSITION), but the sheer volume of shaders to test and verify added up.  Sometimes those shader errors were obvious compilation-time catches, but often they didn’t show up until the shader in question appeared (or didn’t, when it should have) on screen on the console itself.

Especially for something as hardware-specific as this, then, testing on the device itself became critical, which meant build times had a serious impact on our development time. It became important to minimize this impact by setting up isolated rendering test scenes and making “builds” that included only that scene. Even then, there were isolated issues we didn’t catch until much later in the process, so no doubt we could have been more thorough in these early rendering tests.

Staring at the Menu

This is not a good way to navigate menus, but a great way to get hand cramps

Once we finally did have our game technically compiling and running, we needed a way to actually test anything, beyond making sure the main menu loaded properly. This meant getting ourselves some sort of kludge or hack solution for navigation so we could get anything done at all. It turned out though, that Unity maintained mouse support on the PS4 platform. So now we had the Dualshock 4 trackpad hooked in as the world’s worst navigation solution, emulating a mouse. It was inelegant, caused all sorts of minor bugs and annoyances, and was incredibly hard to control and actually click anything. But at least we could get to the majority of our menus, and start testing gameplay.

This is not a good way to navigate menus, but a great way to get hand cramps

CPU Performance Bounds

Early performance tests were quite literally off the charts. The top 40% won’t even display without hiding categories.

Once the dust had settled on getting Guns of Icarus to run on PS4 at all, it became clear that we were far from getting Guns of Icarus to run on PS4 well. Console development was a new challenge, with different rules. On PC, we set the minimum requirements. To a significant extent, we got to decide what hardware the game needs to run on. With 5 years of development experience behind us, this made the jump to a more rigid platform with a singular set hardware a bit problematic to say the least. Don’t get us wrong, the PS4 is a powerful platform.  But we had to work the problem backwards, which brought us our biggest trouble, the PS4’s CPU.  

In some ways, game development is still catching up to the multithreaded world of modern computing. We certainly are. Which meant that we ran headlong into 8 cores at 1.6 GHz, when our game’s system requirements were, on paper, 2 cores at 2.4 GHz, of which we only seriously taxed one core. The GPU was fine, and we barely made a dent in our available RAM. It all came down to getting our game loop as fast as possible.

Large scale multithreading was out of the question. Guns of Icarus was heavily integrated into the Unity Engine, and a lot of work we did on the client side involved API calls that weren’t thread safe, or even permitted to run on other threads. A noticeable portion of our scripts were small and did little more than move transforms around, which wasn’t multithreaded, so the overhead alone of trying to spin off work to other threads already exceeded the potential benefit. Thankfully Unity already had some options to move certain systems onto other threads. Their sound system ran on another thread without issues. Additionally, we were able to leverage their experimental Graphics Jobs system to keep a lot of the rendering time on other threads. Even with those improvements, we were left with a lot of time spent in our own scripts. Everything from networking, to lens flare calculations, to updating text in a piece of UI, all dipped into our available CPU time. Every piece of it needed to be profiled (and we were constantly crashing the profiler so we instead had to manually dump the profile data to disk), and then broken down by hand until we found the exact code slowing us down. We added hundreds of profiling hooks, typing out UnityEngine.Profiling.Profiler.BeginSample over and over until we found what we were looking for.

Since we wanted true cross platform parity, and we had an already active PC game, it was too late to start cutting content and features. That meant that we had to carefully shave milliseconds off systems, piece by piece, until we hit our targets. Sometimes this meant updating objects less often if they were far from the camera, where it was hard to notice smaller changes. Sometimes it meant prefilling object and effects pools and trading a small piece of load time for better performance. Sometimes it meant just digging into the code and seeing if there’s any small inefficiencies eating time that shouldn’t be.

Many of the problems had unique solutions, which meant they took long hours to track down and test. For example, checking a bounding box on every cloud in the game, multiple times a frame, wasn’t a massive hit on performance, but it was enough to be significant. Fixing it meant doing faster distance based prechecks, caching results so it wouldn’t rerun the calculation every time it was checked, and only redoing the calculation every couple frames, rather than every single frame. This happened to be one of the cases where optimizing this late came to our advantage, since we could make some reasonable assumptions about the size of clouds, and the speed of your airship moving through them. We used those assumptions to recreate the effective behavior in a tiny fraction of the CPU time it took originally.

Design Limitations - Anecdote: Why is the World Map breaking the intro video!!??

One challenge of working in an engine without source code access was hitting bugs we couldn’t properly track down. And sometimes those bugs showed up in places that didn’t make any sense. One of the strangest, most nonsensical bug we had ever seen in a piece of code occurred near the end of production. While re-implementing the intro video to work with the PS4’s video player, we were having trouble getting the color to render correctly. No matter what we poked at with the shader, or the code for the video player, it just wouldn’t show up correctly. Eventually we went to Unity’s devs for help. They asked for a reproduction project, so we set off to making one. But even with identical code, we couldn’t reproduce it. It had to be something else in the project. Worse, this bug only showed up on the PS4 hardware, so we had to make builds of the game to test it, and we had no idea where the bug was. We started taking out entire scenes of the game until the bug went away. After half a dozen builds we narrowed it down to… a totally unrelated UI scene. In fact, the solution was moving a GetComponent call from Awake… to Start. As far as we could tell, none of that craziness should have happened. In fact, even though we knew where it was happening, we were still not sure why, and we didn’t really have a way to find out. Ultimately, we were just happy it worked, and we were not going to try to poke at again. Looking back, it was this type of issues that kept us entertained and the development process interesting :D

The video as it was supposed to show up:

The video as it was showing up on the PS4:

Menu Madness

Buttons, Buttons Everywhere and not a Link to Click

Because Guns of Icarus started as a PC game, the earlier UI (lots of it...) was designed for PC players - with a complicated structure so that everything fit in one page, with hover tooltips, popups, and smaller buttons, etc. At the time we started, UGUI didn’t exist, and we had to build a lot of the UI custom. Once UGUI became available, we upgraded all our UI to UGUI in order to have menu navigation work in PS4, for we realized that our custom solutions would not work well for PS4 development. However, once we made this decision, we soon found the many issues that we anticipated coming true. One fundamental issue was that the UGUI’s navigation logic simply didn’t work well in our game.

For instance, if a button (or other Selectable)’s navigation was set to Default, when player navigated a direction, the system would look at all other active Selectables, and find the one that had the highest “score”, which was the combination of two parts: closer, and in the right direction (based on each component’s center point). While it seemed reasonable, in reality, we often found it navigating to unwanted targets, because one part contributed too much to the score versus others.

UGUI allowed us to individually specify each component’sfour neighbors, so all that navigation could be manually/programmatically controlled. However with our complicated UI structure, with many components instantiated/disabled at runtime, this method didn’t work well for us.

In the end, we decided to override UGUI’s navigation logic (all UGUI’s source code is available at https://bitbucket.org/Unity-Technologies/ui/src/). Most logic was the same, but we added a layer of logic, to take game object hierarchy into consideration as well. This allowed us to gain a bit more control over navigation logic (sometimes preferring hierarchical neighbors), while keeping the code / structure reasonable (meaning it won’t break if we tweak the UI). Since we would often have at least three navigable groupings of selectable elements at one time (header, subhead, and content), allowing the controller free reign to select all elements with the d-pad was undesirable to say the least. Locking in navigation groups to be controlled only with triggers or bumpers helped speed up and simplify the navigation for the user.

We also added some other custom code, such as better highlighting of currently selected element (w/o cursor this is a real issue), automatically selecting an element inside when a panel shows up, avoiding selection of components in the background when a “modal” dialog shows up, etc.

With all the work, navigation was still cumbersome for some of the UI pieces, because they were simply too complicated for controllers to use. In the end, we designed two versions of UI for PC/PS4 for some pieces, which was not ideal for maintenance reasons, but probably was the most reasonable approach given where we were. If we had started the game designing for both platforms, planning navigation ahead would be a lot easier.  It was another lesson that we learned.

Lastly, it seemed that Unity’s StandaloneInputModule would by default treat touchpad as mouse. So that even if we disabled its click behavior, touching it would still function as cursor move. This meant inadvertently brushing against the touchpad could have undesirable consequences in the middle of gameplay or navigating the menus.  In the end, we extended StandaloneInputModule.Process() to remove that behavior.

In Closing

Realizing the dream of releasing on the PS4 took us a challenging and lengthy journey. Through the development and certification processes, we had to rethink a number of things we did and find new and creative solutions for issues that we’ve never faced before, especially given that we were trying novel cross-platform implementations. The certification process was highly rigorous, and having this type of QA for the game before release was tremendously valuable. We definitely learned a lot throughout the process that would not only help us with submission the next time around, but in terms of designing different systems of any future game that we aim for console as well. Hopefully what we’ve shared here would be of use to devs looking at PS4, especially for a multiplayer project. For more details on PSN Integration and notes on the certification process, please also reference here.