Crunch: does it help and how do you know?

This article is a continuation of the series from the Game Outcomes Project and extends the discussion around crunch. Here we dig into the question of why teams crunch and how we can measure the actual impact of crunching.

Eric Byron, Blogger

July 29, 2016

13 Min Read

Crunch: does it help and how do you know?

In the great debate amongst the games industry regarding crunch the one fundamental questions that doesn’t seem to be answered is: does crunch actually help a game development project be more successful?

In our original survey and articles, the Game Outcomes Project measured game project success using an aggregate score of 4 factors:

Return on investment
Critical acclaim
Internal satisfaction
Project delays

From our 2014 survey results, our fourth article concluded that teams that reported having crunched during the project were generally less successful than teams that reported little or no crunch. So why do so many still swear by crunch as a necessary part of game development?

Did we ask the wrong questions or make false assumptions about what success is? There are many in the games industry that believe working long hours is an inevitable part of the creative process and those who are passionate cannot and should not be constrained to working 40 hours a week. On the other hand, there is a lot of scientific evidence regarding the ill effects on productivity when working long hours. More importantly, there seems to be complete void of any real data to validate that those working long hours are actually as productive and creative as they would be working less hours.

Alex St. John, in response to an interview with Kate Edwards, recently described those wanting to avoid crunch as having a “wage slave” attitude and basically said we should feel privileged to work 80 hours a week. You can read his blog article here:

http://venturebeat.com/2016/04/16/game-developers-must-avoid-the-wage-slave-attitude/

We need less opinions and more data!

We’ve been talking with leaders in the games industry trying to gather more evidence to support or dispel the theory that crunch is a necessary and beneficial practice when making games or other creative endeavors. One problem we have run into right away is a problem with defining crunch itself. There are those who believe crunch should be defined as extended periods of mandatory overtime: 3 or more weeks of 60+ hours each week. Evan Robinson, author of the article “Why Crunch Mode Doesn't Work: 6 Lessons”, is of the opinion that anything beyond 40 hours per week is crunching. Dave Swanson, a director of engineering for Electronic Arts, commented that what people call crunch now is nothing compared to the hours he worked when he started in game development in the early days of Madden Football at the Tiburon Studio in Orlando, Florida.

We also want to acknowledge before going further with this discussion that there is a distinction between voluntary and mandatory overtime. Within our survey there was a statistical difference, in terms of project outcomes, when respondents indicated that overtime was voluntary. Voluntary overtime is not surprisingly less harmful but there are still studies that suggest, any time we work long hours, fatigue effects the quality of our work. This seems to be especially true for “knowledge workers” who have to make decisions or solve complex problems. In game development this may translate to more bugs or poor technical decisions that result in performance issues or generally poor quality or stability of the game. We’ve included some links at the bottom of the article where you can go review some of the research that’s been done. Don’t just take our word for it.

Let’s talk a little bit about why game teams crunch. Seppo Hellava, Co-Founder and Creative Director at Wonderspark, made a direct correlation to ship dates. He was proud to tell us that his teams don’t crunch because they do not set hard deadlines. They do this by working in short iterations with frequent small releases. This minimizes the risk of introducing a major problem with any release, allows them to quickly change course if they learn something new; either from development or from those playing the game. Because there are no hard deadlines, they can manage the workload and focus on just the next iteration, not a long feature list with pressure to get some major release out for a marketing window.

Of course, this doesn’t work for many game studios with pressure from publishers or marketing to ship a game for Christmas or release within a very short window to hit some specific target market or hit some revenue goal in a particular fiscal period. It can be extremely hard to avoid some degree of crunch when faced with a hard deadline and too much work left to do with limited time to do it. Joe Hoff, Director of Software Engineering at InContact and a former Development Director at EA Sports, put it this way:

“Crunch is caused by designers trying to shove 10 pounds of shit in a 5-pound bag. They will always insist on more features than can possibly be done within the available capacity.”

Yes, there are those that believe crunch is the result of poor management and a willingness to over commit, promising features that will force the team to work overtime to deliver them. Counter point: Warren Spector is famously quoted as saying:

“Crunch will always exist in studios that strive for quality. Look, I'm sure there have been games made without crunch. I've never worked on one or led one, but I'm sure examples exist.”

We talked to Kate Edwards, the Executive Director of the International Game Developers Association (IGDA), about crunch in the games industry. Kate is passionate on this subject and the IGDA recently announced an initiative called the Crunch Comp Initiative. The IGDA has been studying quality of life in the games industry for years, running an annual Developer Satisfaction Survey (DSS). In 2015 the DSS respondents reported the following:

“62% indicated that their job involved crunch time; 58% said they were in crunch more than twice in the last two years; and 61% said that crunch time is expected at their workplace.”

In our interview, Kate also stated, “crunch is almost entirely avoidable, even in creative works, because it goes back to that creative vision and that creative vision includes a certain agreed upon feature set. Someone has to draw a very hard and difficult line and say, ‘that is a great idea but we are not going to do it in this version’.”

She commented that in the DSS, respondents often blame crunch on inexperienced management. When she lectures she has a slide that says: “Crunch time: In every other industry it’s called it ‘poor project management’”.

We talked a little bit about the conflict between creative leadership and project management, who are trying to hold that line and say “no more scope” but the creative leadership is fighting to add more features (recall Joe Hoff’s quote above). Who wins the struggle between creative leadership and the passion to make the game the best it can be and project management, who know that scope creep can cause the team to crunch?

Kate commented that we have to make sure that when considering new scope, that could cause a team to have to crunch, that leadership has to really recognize the impact on people’s lives. She also reminded us that many of these companies don’t compensate folks for working later so it should not just be assumed that because the new feature will be awesome that people should be willing to work extra hours to make it happen. She also suggested that the decisions are often made by managers who aren’t going to work long hours with the staff they are expecting to crunch.

The 2015 DSS found that as many as 37% of developers are not compensated for extra hours they work.

And the debate goes on… Many teams crunch and many leaders make a conscious decision regarding scope versus capacity, believing that crunching will allow them to publish a better, more successful game. So how do we prove one way or another if crunch is beneficial or harmful to most game development projects? We need data. We have a proposal for a way that you can do your own data gathering so you know what the real affect is of working long hours on your project.

We believe that productivity can be measured as combination of three measurable factors:

Volume
Efficiency
Quality

Here’s how this works, and it isn’t rocket science.

Volume

Volume simply measures the amount of work being done during some period of time. You can use whatever unit of measure makes sense for you. It might be story points, work items, bugs fixed, man days of estimated work… You almost surely are already collecting this data point.

Efficiency

Efficiency is simply the amount of effort required to deliver the volume of work you are measuring. You may already collect this data too. For example, if your team delivers 20 story points with 5 guys working 40 hours in a one-week sprint your efficiency rating looks like this.

5 * 40 = 200 man days of work time divided by 20 units = 10 hours per unit. You have an efficiency rating of 10.

This assumes that all 20 story points were delivered with no defects or work remaining. This is not very realistic which is why we also need to measure quality to really understand the true productivity.

Quality

The easiest way to add this component to the calculation is to measure the amount of time that members of the team spend on bug fixing or rework. You don’t have to be precise in this measurement but if you ask folks to honestly ball park the amount of time they spend each day on bug fixing you can get a good indication of how much effort goes into resolving issues.

Using the same example from above with 5 guys working 40 hours in a one-week sprint, let’s say we identify that they each spent about 4 hours fixing bugs. That is 20 hours total or about 10% of the total time spent on bug fixing.

We subtract the bug fixing percentage and calculate the “productivity” rating. (10 – 10%) = 9.

Now we have a bench mark. In a good one-week sprint, with 5 guys we should hit a productivity rating of 9. If you measure this over time you will expect to see some fluctuation, hopefully with productivity increasing as the team gets more experienced and learns to work together more effectively.

The trick is to see what happens when people start working longer hours. We expect to see an initial increase in the volume of work completed with no significant change in the overall productivity rating. The question you need to answer is: what happens after 2 or 3 weeks in a row of working longer hours? Research in other industries has suggested that by the third week you will see all three metrics (volume, efficiency and quality) decline. Meaning, as people get tired they get less efficient and they will generate poorer quality work, be less innovative and make poorer decisions.

If you believe that crunch is okay and a natural way of working in creative endeavors than we implore you to measure productivity in this way and see for yourself what is really happening. Research suggests that after as little as 3 weeks of working 60 hours per week individuals may actually produce less than if they had worked 40 hours each week during that same period. Don’t you want to know if crunch really works? Measure it and find out! Even better; measure it and then share your aggregate data with the Game Outcomes Project and the IGDA so we can share what you’ve learned with the broader industry.

Conclusions:

We believe, from both the Game Outcomes Project survey and from the IGDA DSS, that most people working in the games industry have some period of crunch each year. We concluded from the Game Outcomes Project survey that for those that participated in the survey; teams that crunch have less successful projects than those that don’t.

From the personal experiences of members of the Game Outcomes Project team, and industry leaders we have spoken with, there appears to be a correlation between fixed ship dates and issues with managing scope, that contribute strongly to the need to crunch. Teams that can avoid hard deadlines or can manage scope well are less likely to rely on crunch to get their game released.

For those teams that expect to crunch, and intentionally commit to scope that they know will require crunch probably have no real data to know if crunch actually gives them the boost in productivity they want. It is also possible that crunch causes harm to employees that results in physical and mental health issues, but that’s a topic for another article.

It is possible to measure productivity and, if your teams crunch, you should want to know if the practice is effective or not. Maybe, “you can’t handle the truth!” Stop just believing that we should all expect, and be happy to work 80 hours a week and find out if that is really the best way to be successful.

Question we still need to address:

How long does it take to recover from crunching? If you work 80 hours for 1 or 2 weeks (or months) and then reduce your hours back to something closer to 40 hours, how long before you have recovered?

What if we never have a regular schedule and simply work 60 hours per week all the time? If we don’t have a baseline metric of productivity at 40 hours per week, how do we know if 60 is more or less productive?

How important is diversity of activities to creativity? There are some that would argue that people who work too many hours lose their ability to innovate and need outside activities to stimulate their creativity. How do we know when that is a factor for our staff?

We are here to help and to learn with you. We want to improve the industry that we work in and are all passionate about.

The Game Outcomes Project is an independent industry/academic partnership. It is driven entirely by intellectual curiosity. Our mission is to survey a large number of game developers and statistically analyze the connections between game development team culture and project outcomes.

The Game Outcomes Project team currently includes Paul Tozour, Lucien Parsons, Zhenghua “Z” Yang, NDark Teng, Eric Byron, Ben Weber, Karen Buro, Joe Frontiera, and Carlos Puertas.

Leaders who contributors to this article included: Seppo Helava, Joe Hoff, Clinton Keith, Keith Fuller, Dave Swanson, Andre Thomas, and Kate Edwards.

References:

IGDA resources on quality of life

http://www.gameqol.org/igda-qol-survey/

IGDA Developer Satisfaction Survey

http://www.igda.org/?page=surveys

Interview with Kate Edwards

http://venturebeat.com/2016/03/20/why-crunch-time-is-still-a-problem-in-the-video-game-industry/

Clinton Keith’s blog - “Death March Crunches: 10 Causes and Solutions”

http://www.gamasutra.com/blogs/ClintonKeith/20160422/271137/

Danial Cook article on productivity

http://www.lostgarden.com/2008/09/rules-of-productivity-presentation.html