Sponsored By

When can Waterfall development work? And is Microsoft Project really useful? GarageGames CEO Eric Preisz tackles the question of when you should leave Agile to the side and go Waterfall -- and explains exactly how his company does it.

Eric Preisz, Blogger

November 21, 2012

18 Min Read

Agile, Agile, Agile. It's hard to disagree with a project management process named "Agile". It's like disagreeing with something named the "Patriot Act" or answering "No" to a department store checkout attendant that asks if you would like to save 20 percent on today's purchase. It's clear: Agile is cool, and every Scrum master certificate mill will agree with that statement.

But what happens when an Agile paradigm isn't appropriate?

"How is that possible?" says the guy with the skinny jeans and the PBR.

There are two scenarios where we think a more "waterfally" approach is appropriate (We'll explain the "more" part later... because no one could ever be purely waterfall if you think the name implies a rigid schedule).

The first is when a project is innately low-risk, and the second is when your stakeholder doesn't wholeheartedly buy into the Agile process. Examples of the second include Kickstarter and the majority of contract work.

For the first case, I think you can throw out any game development scenario where you are striving to achieve fun that isn't a full clone of something that is already proven to be fun. Fun is just too damn squishy, and in my experience, you don't know something is fun until you play it and say, "Hey, that's fun!"

Creating fun is a risky proposition. Vertical slices -- a technique where you make some small percentage of the game at near final quality -- and rapid prototypes help pave the way for a more structured project.

The second case is the focus on the remainder of this article. GarageGames is a service company and in the majority of the cases we see, the formula for customer's desire on an initial consultation is usually something similar the following:

I want X by the date of Y. How much will you charge me?

Problem Statement:

A customer wants a firm fixed price (FFP) quote for a game-related software development project. How does a company arrive at a price and date that is fair for the procuring and commissioned company?

To simplify the problem statement, what the customer wants to know is: What, How, When, and How Much is required. Additionally, you will need to determine Who from your firm will do the work. Those elements, solved together are a FFP contract, schedule, and Statement of Work (SOW). Here are the steps we use to build them:

  1. Determine initial scope -- Focus on the What.

  2. Determine if the project is suitable for FFP contracting -- Initial analysis of How.

  3. Build a rough order magnitude estimate -- Second analysis of How, first analysis of When and How Much.

  4. Determine scope of the Statement of Work -- Final analysis of What.

  5. Determine & deliver final quote -- Final analysis of How, When, and How Much and Who.

Determine Initial Scope

This usually occurs through one or more conversations with business development, technical, and project management. If a GarageGames employee can do all three it can often be covered in one conversation. The most important goal is to be output and goal focused. Don't get too caught up in implementation. Right now you are more interested in the what than the how or when. For us, this process of determining initial scope starts with a conversation about the customer's high-level goals.

Often times, the customer doesn't know what they want aside from their high-level goals; sometimes, you need to help them understand their goals (a bit of a red flag sometimes). It's understandable that they might not know; if they really understood the solution to their high-level goals then they wouldn't be coming to you now would they? Work with the customer to define and document the high-level goals. By understanding them, you are in a better position to either develop their plan or validate any existing plans.

Knowing how much can be helpful at this stage. Customers typically don't want to tell you their budget at this point because they are worried that you won't give them an honest quote --you have an asymmetric knowledge advantage that makes them uncomfortable. A customer's budget holds intrinsic clues about what can be accomplished and not knowing that number will make the process much harder to scope and define the project.

GarageGames asks customers to try and identify a budget range for the purposes of accelerating the quote process. We assure them transparency in our quote that addresses their fear and we've found that sharing more detail with the customer gives them a better experience. Transparency puts both parties on the same side of the table tackling a problem instead of negotiating against each other as foes. Working together to bring down cost is something we enjoy doing. We believe that if we save customer money they are more likely to spend that money on us in the future. Both sides win.

If they aren't willing or allowed to share a budget range with you hope isn't lost -- it's just more work to triangulate the cost from the bottom up.

Determine if the project is suitable for FFP contracting

At this stage, we are going to analyze the project by doing an initial analysis of how we will implement the project to meet the customer's high-level goals.

It's time to start thinking about risk. All projects, especially software projects will have some risk. How confident are you about your team's ability to accomplish this project? What is the variance of success and failure? At GarageGames we measure risk using a confidence scale of 1 to 10.

With an eye towards risk management, we do our initial analysis of how we will implement the solution. To do this, we work with technical leads to formulate an approach.

We decompose the output above from high-level outputs into smaller features. This process is known by many as a Work Breakdown Structure (WBS). At GarageGames, we tend to get to a more detailed decomposition than many. It takes more work, but it protects us from taking a job we can't complete by exposing more known issues.

For each of the features, we have the leads describe the expected implementations. How will you develop this feature? How confident are you that this approach will work? Document the confidence and determine a gut feeling about your confidence in the implementation of the entire project. A task estimate may look like this:

Implement Frustum Culling   Estimate: 20 hours   Confidence: 7

One of the biggest mistakes made by companies is saying "yes" to a job they can't perform. If your gut feeling is that you aren't very confident in your ability to accomplish the work, please do yourself a favor and either 1) walk away or 2) insist on a time and materials contract where you are paid hourly for the work as you accomplish. Saying yes to a FFP job you aren't confident in will likely hurt your business as bad as your prospective client. Just say no!

If you are having challenges decomposing the steps or the cumulative confidence numbers are low, this might be a bad job for your company.

GarageGames Confidence Scale

Below is the scale GarageGames uses as a standard for confidence numbers:

  1. No clue -- don't make decisions on this. We need to talk more.

  2. Hardly a clue -- don't make decisions on this. We need to talk more.

  3. Someone else has done this and I read about it; job not well defined -- don't make decisions on this. We need to talk more.

  4. Someone else has done this and I think I understand this; job might not be well-defined -- don't make decisions on this. We need to talk more.

  5. Done something similar to this before and this relates to that work -- this estimate has a variance of +/- 50 percent of estimate.

  6. I think I understand this fairly well and I understand the goal.

  7. The average case for programming when the requirements are understood.

  8. A confident case for programming; average case for a lot of art work.

  9. I've done this before and it's very similar.

  10. I'm probably not working on software... or... no matter what; I'm only going to work on this for a period of time (i.e. 1 week of QA).

Build a Rough Order Magnitude Estimate (ROM)

If we feel comfortable moving forward under a fixed priced contract, it's time to do the second analysis of how, and how much. We will also do the first analysis of when. We use Excel.

First take a look at your set of features and implementations. It's time to define some milestones. Milestones are waypoints towards your end goal of delivering the product. A good milestone is measurable and defined by the outcome of your work. Write these down as eloquently as possible. Hopefully, you can describe 90 percent of the milestone in one sentence. A detailed description of the milestone is best described as a test plan written by QA. If a developer can test their work before it gets into the hands of QA, the less time you will need to iterate.

Milestone Definition: The first play experience of Level 1.

At each milestone, think about the steps you will need to take to reach that outcome. Look at your implementations and segment them in a way that puts risker items earlier in the development process. Why? Because if an implementation goes awry, you can tell your customers as soon as possible so that you can work together to resolve it. A lack of time is your biggest enemy when things go wrong.

Many project managers forget the following tasks. Set aside time for the following:

  • Shims (scaffolding code required to stand-up an implementation or increase testability and efficiency)

  • Refactoring before milestones build on new code foundations

  • QA

  • Reactions to QA (why find a bug if you aren't going to fix it)

  • Usability

  • Customer Feedback and correction time

Now that you have your milestones, write them down. Add in the implementation tasks. Use a hierarchy to help you organize work and decompose as necessary. A simplified snipped of a ROM might look like the following:

Milestone

Estimate

Confidence

 

 

 

Milestone 2 -- User Kills a Simple Enemy

 

 

Art & Design

 

 

Build Level

32

9

Build Initial Player Character

80

8

Build Initial Weapon

40

8

Programming

 

 

Implement Player Damage

32

7

Implement Basic Enemy AI

16

6

Implement Enemy Damage

8

8

Implement Enemy Death

32

7

A snippet from a ROM. We prefer to use estimates in hours, but sometimes days makes more sense.

For each line, develop an estimate. But before you start, we need to think more carefully about what an estimate really is. At GarageGames, we believe an estimate is a time and risk estimate. Let's take a second to really understand how GarageGames determines risk adjusted estimates, it's critically important.

Note: When estimating, we always use work effort which is the total work required to complete the job. Duration measures how much calendar time it will take from start to finish. For example a 16 work effort hours can have a duration of 8 hours if two people are working on the task.

Risk Adjusted Estimates

Let examine two estimates: Task A is a non-risky job you've done before. Task B is a risky job you've done before. Assume for a second that I've asked the engineer to give me an estimate for each and in both cases, they say two weeks.

Beware: In this case, two weeks does not equal two weeks.

More often than not, undiscovered work will cause job B to take longer than expected. It's likely that two weeks REALLY means: "two weeks if everything goes the way I hope it will." It's probably more accurate to assume that the job will take two to four weeks. Since we don't really understand all ramifications of a risky task until we do it, we really can't be very accurate. Some people pad the estimate; we prefer something a bit more pragmatic, descriptive, and standardized.

With every estimate, GarageGames uses confidence to determine a risk adjusted estimate. So if we include confidence numbers, the two estimates might look more like this:

Task A: 2 Weeks, Confidence: 9

Task B: 2 Weeks, Confidence: 4

To calculate the risk adjusted estimate, GarageGames uses the following function:

Risk Adjusted Estimate = Estimate + (Estimate*(10-Confidence)/9)

Task A: Risk Adjusted Estimate: 2.2 Weeks

Task B: Risk Adjusted Estimate: 3.3 Weeks

Having the confidence value represented is useful for more than just determining risk adjusted estimates. As stated earlier, finding ways to reduce cost for a project is something that we openly embrace. Low confidence numbers are expensive and clients are motivated to help you increase confidence numbers to reduce cost. Before writing a single line of code, you can reduce risk the following ways:

Cutting a Feature -- "We didn't need it anyway... It's not that important."

Constraining a Feature -- "What if it did half of what we were hoping for."

Defining a Feature Better -- "Oh, I didn't understand what that meant... That's not so risky now."

Rapid Prototyping - Allowing time for rapid prototyping early in the project (and then updating the estimate ex post facto) --"Eureka!"

This is a great time for leads to share estimates with those who need to actually implement the task. It's hard to have developers who are vested in delivering a project if you didn't consult them initially. They are usually quite motivated to help. It's during this phase that implementers will ask you, "What does this task actually mean?" You will need to work with them and the customer to define what the implementation will or won't do. Save this definition, it will be important later.

Now that you have all the estimates, confidence numbers, and risk adjusted estimates, you should be able to provide your first estimate of when the project will be complete. Add up the risk adjusted estimates and divide it by full-time employee team size (two half-time employees equal one).

This isn't a very accurate estimate because it assumes a perfect world where everyone can do every job perfectly parallel -- hardly the case. But at least you will have some data to help you understand the very best case very high-level ball park figures. If someone wants a task done in 6 months and your estimate says you'll be done in seven months, you'll know early that something needs to change.

Same goes for the risk level. Average the confidence numbers. If the confidence numbers are too low (below, say, a 6), then maybe it's time to suggest either 1) moving to a time and material contract or 2) doing a small fixed bid contract up front to rapid prototype in an effort to bring up confidence numbers.

Keep in mind that while you are working on estimates, you will be defining the project or asking questions that will lead to the definition of the product. Write them down in small one or two page descriptions and save them. You will use them to develop the Statement of Work.

The final step is to share it with your customer. Show them where the risk is high and tell them why. Discuss opportunities to save money by reducing risk.

Build the Statement of Work (SOW)

To some extent, you've probably developed the majority of the content for the Statement of Work. The process of decomposing implementations, estimates, and calculating risk has probably generated a decent amount of documentation between you, the developers, and the customer. It's time to organize that documentation into a draft Statement of Work. A good statement of work will define assumptions, clarify features, define milestones (and ideally the tests of completion). Of course, the customer will have details that are important to them as well. If scope changes during this process, make sure you go back and update the ROM estimate.

One particular tip that I think is really helpful, if you haven't done so already, it to define as many functional inputs and output specifications as possible. To us, that usually means GUI wireframes, APIs, and network protocols; today you may also want to define gestures, voice, etc. You can't define application implementations until you can understand and describe the inputs and outputs of the application.

It's also important to agree on quality. If possible, find work samples that reflect the quality bar you expect to achieve. Setting expectations is paramount and setting quality expectations is one of the more difficult tasks to accomplish since quality is not nearly as quantitative as other more technical specifications.

Determine Final Quote & Final Statement of Work

You are almost there. The last step will generate the final version of documentation required to perform a waterfall project correctly. You've answered what (SOW) and how much (ROM Estimate) but you still haven't modeled all of the complexity to answer when. And in order to do that, you need to have your final answer of the last "W" which is who.

For the final answer, we use Microsoft Project. Many people cringe when I mention Microsoft Project, but once you learn how to use it correctly for software projects, it's quite useful. The key is to add the "work" column (Right click on the header and select "Insert Column") and enter your estimates there. Don't enter work under the duration column; it will drive you crazy if you start using more than one resource to complete a task.

Finally, if you find yourself using dependencies to keep things from going crazy you have done something wrong. Dependences should only be used to represent a real work dependency -- not as a tool to force project into submission. If you want a task to begin or end sooner, use priority. If you want a set of work, like a milestone, to begin after another set of work, like the next milestone, then use dependencies at the milestone level. Add resources and use leveling liberally. If you aren't hitting "level resources" a lot, you've done something wrong and Project will be more difficult to use than it should. The strong majority of people that dislike Project probably aren't using it correctly.

With everything in project, you now have a reasonably accurate set of dates to share with your customer. By filling out the resource section, you also have a clear understanding of who will do the work.

Start the SOW with the customer's high-level goals. If you project no longer meets those goals... something went wrong along the way.

Conclusion

You have finally answered What, How, When, Who, and How Much. You've accounted for risk, which is the area most project managers fail to consider when building estimates. Those elements are everything you need to perform a FFP contract. Moving forward, if you prefer to run your team in an agile format, you can take your tasks in project and move them into the Agile project management system of your choice. At GarageGames, we use Jira and GreenHopper. I prefer to use Microsoft Project as my backlog and enter tasks during sprint building time while others at GarageGames like to enter the tasks from project directly into Jira.

Now to the final point: In the beginning of the document I mentioned that I thought it wasn't possible to make something truly waterfall. And what I meant was that all software projects require some flexibility. You may need to make changes on the fly. Discuss with your customer if there is wiggle room in dates or features in cases where things are slipping. It's not realistic to thing that nothing will change, so be sure to leave some room for... well...agility.

Read more about:

Features

About the Author(s)

Eric Preisz

Blogger

Eric Preisz is the CEO of GarageGames and has been involved with engine development for over thirteen years. Prior to GarageGames, he was a Department Chair at Full Sail University and the managing partner of a serious games consulting firm. He is passionately interested in performance optimization and co-authored the book Video Game Optimization. He worked for companies and organizations including Disney, NASA, and the U.S. Department of Defense and has given presentations at Microsoft Research, GDC, and a Congress Caucasus.

Daily news, dev blogs, and stories from Game Developer straight to your inbox

You May Also Like