Planning and the Unknown Unknowns

Last November, I gave a talk where I described the story of our video game Ultimate Chicken Horse from early conception to release and beyond. My goal was to amalgamate the answers I’ve given to people when they asked about different parts of the story: “how did you get funding? How did you get your game on PlayStation? How did you find your partners to start your company?”, etc. The format was to stop the story at each point where I felt I learned a lesson, and share those lessons with the audience.

One of the lessons that I drew from the chronological story of this journey was to try to think of everything. It seems obvious of course, but it seems like you couldn’t possibly know what you weren’t thinking of, because obviously you weren’t thinking of those things.  So given that “try to think of everything” is a bit hard to act on, I now use a new phrase: plan for the unknown unknowns.

When you don’t plan, you end up with half-built things.

This phrase is taken from a book by psychologist Daniel Kahneman called Thinking: Fast and Slow, in a section where he describes the planning fallacy. The planning fallacy is used to describe plans and forecasts that “are unrealistically close to best-case scenarios” and “could be improved by consulting statistics of similar cases”. The result, of course, is that plans take longer than planned, costs go way over budget, people get unhappy, products under-perform, and everybody loses. Sound familiar? Game development, and surely most other fields of work, are plagued by the planning fallacy.

I’m going to focus today on three proposed ways to mitigate the planning fallacy:

Establishing a Baseline

Kahneman and his partner Amos Tversky did a lot of research into many topics in behavioural economics from a psychology perspective, and eventually won a Nobel prize for that research in 2002. After coining the term planning fallacy, they offered a solution of how to solve it by establishing a baseline.

The idea is to avoid the common tendency to neglect the statistics of similar cases to yours. To use game development as an example: “Most MMOs take four years to make with a team of our size, but we have industry veterans and we’re really well organized, so it will only take us two years”. That’s obviously not a very forward thinking mentality, but it happens to all of us whether we realize it or not, for small tasks and large ones alike.

This idea was eventually formalized and given the name reference class forecasting,and it works in the following way:

You might only realize that you’re the little duck after establishing your baseline.

Often, you may find, that you might have to stray away from the baseline prediction by increasing it, not decreasing it. What are your resources like? Does your team have experience with this kind of project? Are there new technologies that can quicken the process? Are there new technologies which need to be learned, which may slow it down?

It sounds so obvious to look at statistics from our surroundings, and yet we all forget to do it and rarely catch ourselves forgetting.

Taking an Outside View

The next thing to do is to take an outside view, and to step back from our situation. We naturally take an inside view by focusing on what we have, what we’re doing, and the experiences we’ve had with regards to the situation. We extrapolate from what we know, we reason using small bits of data, and we get caught up in emotion when making decisions about ourselves and our plans.

The easiest way to get around this, and the way that has worked for me and for my company in the past, is to ask others. Find people who will give you straight up, no-bullshit feedback about your plans and ask you the tough questions that you’ve likely been ignoring.

Asking others means you’re no longer forecasting based on information in front of you, and gives you a more complete picture.

Planning for the Unknown Unknowns

Remember the example about the MMO that was only going to take your studio two years? We already discussed that it’s not the smartest thing to assume that it will take you less time. But most people will take an extra step and actually plan; they will thinking of all of the things they can, take an outside view, ask others, and even look at comparative projects to make their estimates. Once they’ve done all of this, they will sum up all of the things they think they need to do, assign times to them, plan using current and future expected resources, etc. After this whole process, they’re still left with a little over two years in their plan for this MMO project.

This is because they didn’t think of the unknown unknowns. These are the things that can come up mid-project: bureaucracy (and boy do we know about that one in game development), illnesses, divorces,  technical delays, dependencies on contractors, change of personnel, people moving, etc. etc. etc. As I mentioned in the introduction, you can’t know what you don’t know, so it’s hard to plan for it.

This is why we add contingency in our budgets, and why we should add a hell of a lot of contingency time to our plans when starting projects or agreeing to deadlines. We also try our best not to promise anything before it’s ready; many companies (and individuals) run into problems when they can’t meet deadlines for deliverables, but often these dates are self-imposed and do not need to be so fixed. There are, of course, cases where a client is dependent on you, or a project needs to reach a certain milestone because of timing with a season or sale, but I’ve seen many self-imposed deadlines set up by the “suits” for no apparent reason, and this can cause unnecessary stress and perceived failure due to missing those deadlines.

Beyond asking other people what unknowns you might run into (as was the case when taking an outside view), you can also study other projects or companies and see what kind of issues they ran into. Even if you don’t expect to run into the same exact issues, there’s a good chance that it will inspire you to think of potential issues for your own situation.

So first, think of everything you and all of your peers can think of, and then plan for the things you haven’t yet thought of.

Why is this Important?

This is relevant to anyone in a management position or anyone who is making decisions about planning.  In video game development, every single project I’ve ever heard of in the history of games has been late. If it wasn’t late, it was shipped at a way lower quality than it should have. I’m not sure if other industries are as notorious for delays, but I imagine it’s a common issue across the board.

While none of these suggestions are hard science or give you concrete steps to take to ensure success, they should help guide you in a way that can help prevent failure. As I read this section of Kahneman’s book, I realized the direct application that this psychology could have to the game development world and thought I would share, so I hope you enjoyed reading.

—

Thinking Fast and Slow, by Daniel Kahneman
Note: the idea of “unknown unknowns” is originally attributed to Donald Rumsfeld, as Kahneman states in his book.

Daniel Kahneman: Beware the Inside View