Small Developers: Minimizing Risks in Large Productions - Part II

[Experienced game developer and manager Troy Dunniway, a veteran of studios like Microsoft, EA, Insomniac, and Ubisoft, discusses some of the major risks involved in transitioning from a small to large development team and how to identify and avoid them. This second installment tackles the nitty-gritty risks different disciplines will face.]

My earlier Gamasutra article Small Developers: Minimizing Risks in Large Productions - Part I talked about what general risks a small company might face as it tried to expand and take on larger projects. It gave a general overview of these risks: risks in running a game development business, risks in not having adequate or proper game preproduction, game development process risks which occur when you don't have a defined roadmap for how to work, and risks around your team members and staff.

In Part II, we will continue the article and dive into: schedule and project management risks which occur when you fail to properly schedule, risks involved in poor game design, art, testing and programming decisions and the risks involved in outsourcing. This article will focus on more specific problems on your projects which you must look out for.

1. Schedule and Project Management Risks

The risks of improperly scheduling a project are numerous and ongoing throughout. There is no right or wrong answer on how to develop or how to schedule. Some teams utilize a waterfall methodology; some use an agile method. Most use some kind of hybrid methodology.

No matter how you handle it, you will incur a tremendous amount of risk to a project if you don't schedule your projects, and you will also incur a lot of risks if you schedule poorly, fall behind, or make bad decisions in your scheduling.

Many producers just like to add a 20 to 50 percent (or more!) buffer to every task and schedule, allowing for unforeseen problems. While this is a good practice, it can't account for all problems. If you are scheduling the game or a feature you are working on, and know that there is some inherent risk in it, you should account for that from day one. "Plan for the worst; hope for the best," as they say.

The risks and solutions for how to properly schedule your game could fill an entire book. The important thing to keep in mind is that not scheduling your project, or doing it poorly, will usually lead to lots of problems. This is one of the reasons that larger teams usually employ one too many development directors, whose whole job is to manage the schedule for the project, or sometimes even just a team or feature.

The biggest thing you can do to avoid risks in scheduling is to establish a process where your leads are not tasked at 100 percent for the project -- rather, usually at 50 to 75 percent -- and can then safely spend time scheduling, hiring, managing the team, doing status reports, having 1:1 meetings with reports, and so on. Many teams expect their leads to be as productive on the project as everyone else, while also shouldering the additional responsibilities of being a manager. This tends to burn out leads more quickly and is a major source of aggravation for many.

If your team is too busy to stop and schedule, you have a problem.

2. Design Risks

Understanding when and how to make something extremely new and innovative and when to utilize existing technologies is an important decision. Many designers like to do things differently just because they can. Sometimes this can be good, but it usually leads to a project assuming the maximum amount of risk. It is good to understand what risks your game design is going to possibly impose on the project and how to minimize them.

Design risks can come in many shapes and sizes, so it is impossible to categorize them all. Things can be risky because the team has never done them, or because no game has ever done them. Things can be risky because they could require a lot of resources to do correctly, or because there are no tools or processes in place to do them. In the end, only the team can decide if there is a lot of risk or a little risk associated with each feature, and weigh in on how to minimize these risks.

Early on it is good to brainstorm wild and crazy ideas, but it is also good to put some limits on things and try to reduce some of the risk as soon as you can. This can include setting a limit to how many new things you are doing (I usually try and do no more than three to five new things in a project if possible), or putting a time limit on when the new stuff must be proven -- before you go to plan B, or cut it.

The design pipelines must also be identified for risks. You need to take a look at your tools and processes and make sure that they are optimized to let designers create, tweak and balance content as quickly as possible, as well as allow the maximum number of designers to work on the same stuff as possible. The design tools problems are especially amplified for open world, role-playing and other less linear and more complicated games. If your tools pipeline isn't optimized, you run the risk of the design team not being able to start until others have finished, getting behind, and then not being able to catch up -- forcing the game to be delayed.

It's especially important to remember that designers are also majorly at risk from others on the project falling behind and delivering their assets late. Since the designers are usually the last to touch a level, and do most of the assembly, they are much more susceptible to problems faced by the entire team. Designers must try and ensure that the other teams stay on track.

In the end, making the wrong design decisions, and having a game director that is going to force the team to do them regardless of what everyone else thinks, can be a massive risk to the project and should be avoided at all costs.

3. Art Risks

Just like the design team, the art team also has a number of pipeline-related risks associated with it. Unless the pipelines are optimized and proven, you run the risk of things taking much longer than they should. Even things like the export of the art into the game engine, which seems like a no-brainer, can easily be messed up by a lazy or inexperienced programmer.

Most of the risks for the art team will be proven during the prototype. Probably the biggest risks come in the fact that sometimes the engine keeps changing, requiring all the art to be tweaked or re-exported time and time again. This tends to happen a lot to teams which are implementing a new engine. Otherwise, most risks in art tend to be fairly minimal. Having done it once, it is generally a "rinse and repeat" a few hundred or thousand times -- in theory.

Art Tools and Pipeline Change and Problems

It is very important for a team to get their art pipelines locked down. This problem especially exists for larger games, games using new engines, and games which have a long development cycle. In some cases, the specifications for the art in the engine can change, causing anything from a simple re-export to a complete redo of the art in the game.

Another problem is that the art tools can also be very hard to use for the artists, causing a lot of slowdown. This is usually because an engineer just hasn't had the time to optimize a tool, but can sometimes be for other reasons.

I worked on an ill-fated project where the art export process took one half to a full day per object to export and get properly into the engine. For a long time we thought this was a technical issue with the tools or engine, and the artists had just accepted it and were working without complaining, even though it was slowing the project way down.

In the end, it turned out it was a deliberate sabotage by a programmer who was having "issues" with the art director, and had purposefully made the tools hard and slow to use as payback for some grievance he had. So, sometimes you never know.

Art Optimization

The other major risk in art production usually comes later in the project, when you have thousands to millions of assets to track and implement in the game. Most teams will allow their games to run slow (usually less than 30fps) early in the development cycle, counting on the fact that the programmers can optimize the engine to run much faster and get the game up to the desired framerate at the end of the project.

In theory this can work, but in practice it is easier said than done, and can often require the art to be heavily optimized at the last minute -- which can lead to not only a lot of crunch time, but frustration for the art team.

In other words, trusting your engineering team to optimize the art at the last minute will always be a huge risk to the art team.

Some teams will just spend the time optimizing the art, while others think about this problem ahead of time and design many of the assets to be "optional", allowing them to drop 25 percent or so of the art in any area, without compromising the integrity of the level.

However, that being said, I have also seen numerous projects get into trouble because their art was way too expensive. Sometimes this was purposeful, and sometimes it was done accidentally -- or just because the artist didn't know any better.

I've seen 3D scenes where a simple trash can, which was supposed to be maybe 10 pixels high on the screen, was modeled with 50,000 polys, and the team couldn't figure out why it was running slowly; not having some guidelines and boundaries for artists can be very risky as well.

Another common risk is that teams tend to need to "sell" the game to executives, and art is usually the best tool to wow them. I have seen a tremendous number of demos created that look absolutely amazing; unfortunately, frequently a large portion of the gameplay systems still aren't functioning, but the area is using 100 percent of system resources. Inevitably, the demo isn't real, and now the team is stuck trying to tell management they messed up, or thinking they can optimize their engine to make it run faster. Both solutions have major risks and should be avoided.

You also need to make sure that the art is really what is slowing down the game before you set out to optimize it. Most 3D graphics chips and engines are very good at pushing polygons and textures, so if you're spending lots of time (on a high end console game) optimizing a few polys or pixels, there may be other problems. I know of several cases where programmers blamed the art for many of the slowdowns, and it turned out that it was the AI, physics, or another system really responsible.

4. Outsourcing Risks

The term "outsourcing" is generally applied to creating artwork using a team in another location -- often in China, India, Eastern Europe, or other places with cheaper labor. These teams can vary widely in skill, however, and are seen as risky by many.

Some cheaper teams without a lot of experience can be seen as particularly risky, but even these teams can have a majority of their risks mitigated by some best practices and proper management.

If you have never worked with an external art team, it can be tough. There can be time zone issues, communication problems (either due to the time zones or language barriers), technical problems, art quality problems and much more to deal with.

You can mitigate your risks, however, by making sure you test them well before you hire them, making sure they understand and are capable of executing on all technical requirements, are capable of doing the quality you want, have very clear directions, get timely and thorough feedback, and are properly communicated with.

So, while outsourcing your artwork can be difficult, it can also save you a lot of time and minimize your risk of having to hire and keep a large art team busy. It also allows you to utilize multiple teams to increase the amount of work being done, or to capitalize on different teams' strengths and weaknesses.

5. Programming Risks

Programmers also face a number of risks during a project. Probably the biggest risks to programmers, besides choosing the right technologies, are designers. A design team that is constantly changing its mind, not creating adequate specifications, designing on the fly, designing gameplay ideas which are infeasible, not willing to design what the technology can do, and things like that will quickly derail a programming team.

But, of course, the programming team also needs to make sure it is doing its job right: running source control, adhering to good coding standards, investing the proper time into the tools, and so on.

Engine Technology Risks

The game programming team runs the biggest risks in choosing the right engine and technologies to use. If the team decides to create a new engine, it will take on a huge risk. If the team is going to license an engine, there is still a lot of risk, especially if few on the team have used the engine before. So, to minimize the risks, the programming team needs to spend an adequate amount of time evaluating and learning the tech and all possible choices, before committing to anything.

This is a very hard and controversial risk. Some teams will choose to make their own engine while some will choose to license. If you have never worked with an engine you are going to license, and nobody on your team has used it, there are a lot of risks involved which you must spend time to investigate. When investigating which engine to use, a team must not only look at the performance of the engine, and what features it is capable of, but also how its tools work, and much more.

For example, many of the cheaper game engines which are primarily used to create casual games not only don't have the performance to run a large scale game, but the tools they use are often very inadequate for large teams to use efficiently.

Some engines claim that a well known game used them, but then if you talk to the developer, they may actually say that they re-wrote a good majority of the engine and had to apply major resources to get it working properly. So, even if you are doing an exact rip-off of a game which used the engine, you can't know for sure how much work it was to really implement it.

One ballpark, which sometimes you can gauge, is to look at the team credit list for a project which used the engine, and see how many engineers they had working on it. If the game took a long time to develop, or there were dozens of additional engine programmers listed in the credits, you may want to ask why -- if you can.

I have seen dozens of projects canceled, or severely delayed (by a year or more), because they made the wrong decision about which engine to use.

Middleware Risks

Using middleware can be a really great choice, or can turn into a nightmare. Middleware can work as advertised, or be incredibly slow and/or buggy. So, just like when choosing your full game engine, you need to choose your middleware carefully, and make sure you understand all of the features and performance risks involved.

Make sure you create a memory map, and make sure that your middleware will run within the limitation that is it supposed to run in. Many teams test the middleware in a clean environment, and don't realize what will really happen when you put it into a full game with tons of other stuff running.

Also, it is important to not believe the marketing of any middleware companies. Just because a project says that it is using the middleware, doesn't mean that the team is using it out of the box. The developer might only be using a portion of the middleware, or the team may actually have gone in and heavily modified the middleware, fixing all the problems in it, in order to get it to run.

6. Testing Risks

Getting too many bugs in your project is a major problem. Some teams prefer to just let bugs pile up for most of the project and only fix the showstoppers, while other teams tend to spend some percentage of their time each week trying to fix problems and optimize their code.

The longer you take to fix a bug, the harder it can be to fix and the riskier it is. However, you can also argue that if you spend too much time fixing bugs early on, you don't get as much done, and take on the risk of falling behind schedule or running into other problems.

The bug count becomes increasingly more important as you head into the final development of the game. Depending on your schedule, this could be at the same time as beta, or could be later.

However, as a general rule, if you have three to four months left on your project (assuming a standard 24 month development time), and you have thousands of bugs in the database, you probably won't ship on time.

So, it is important to monitor your bug count and assess this risk constantly through the last half or third of the project. Otherwise your programming team will rarely be able to catch up with their daily new features, bug fixing of existing bugs, and bug fixing the new bugs the test team finds each day.

Avoiding Risks

Now that you have an idea of how to identify risks, it is important to understand how to avoid risks in the first place. Avoiding risks means that you need to learn from your own and others mistakes. Avoiding risks means that you need to be smart and constantly looking for new problems, but also making sure you solve old ones so that they don't happen again.

The most important thing you need to know and admit is what you don't know. A lot of companies are formed by very talented people who think they know everything, but in the end realize that there was a lot of knowledge expertise that others had on their previous teams, and which is now lacking in their companies.

Even if you can't hire people into these roles permanently, you need to make sure you identify your problems. There are lots of experienced contractors and consultants out there which can be found to give advice and direction. Make sure the decisions you make are based on real knowledge and not guesswork, if you want to avoid risks.

In the end, you need to build a defensive development model. It's sort of like defensive driving. You need to be able to speed down the road at full speed, but be constantly aware of obvious and potential problems around you as the traffic patterns change. You need to give yourself enough leeway that you can change course or dodge out of the way of things, instead of slamming into them.

Just like in defensive driving, your "peripheral vision" during your project development and some attention should always be focused on looking out for oncoming problems and how they can be solved before they even happen. In the real world, this is extraordinarily difficult, and sometimes impossible, but if you stop trying to avoid risks, eventually one will come along which will stop you, and your project, dead.

Reducing Risks

This list is far from being a complete list of risks which could exist in your project. There are too many variables in each project to be able to have a list of everything which will pop up. In the end, you must use your head, the schedule, the feature list, past experience and gut instinct to look for things which could go wrong and flag them for analysis.

The best way you can reduce risk however is just to talk about it and create a plan to avoid it. Talk to others outside of your team and get their advice as well, to see if they have had similar problems. It is also good to read the project postmortems on Gamasutra or in Game Developer magazine. These tend to point out a lot of problems with projects, which often could have been avoided if they had just done a basic risk analysis.

The other aspect to risk -- besides its identification -- is to have a plan not just to avoid it, but also on how to get around it when it does inevitably happen. So, when risks are identified, you can either try to eliminate them as they are discovered, or you can create a plan for when you will stop taking the risk and fall back on a more tried and true methodology or feature.

Some risk in a project is a good thing, since if you have no risk at all, then you probably aren't innovating and trying to make the game as great as it possibly could be. If you find yourself saying there aren't any risks in your game, you are either really lucky, or just blind to reality.

It is also important to remember that larger scale projects usually have very different risks and problems associated with them and that you can't use the same risk assessment for a larger scale game as you may have for a smaller game.

Every game has totally different needs. It's critical that you don't fall into a complacent rut and just continue to use the same processes that you have used before, because it is what you know. It is important that you continue to challenge your development processes on each project and verify that your risks are actually minimized.

No matter what happens, just remember to continually communicate with your entire team and work together, and your problems will be much better if they are kept in the open and not hidden.

---

Photos by Scott Maxwell, openDemocracy and wwworks, used under Creative Commons license.