Q&A: Producer Dante Falcone on avoiding crunch in live game dev

In this brief interview, experienced producer Dante Falcone (Turbine, ZeniMax Online, Oxide Games) shares advice on how to plan and schedule your next live game to avoid overworking your team!

Can you tell me a bit more about your background and experience with live ops ("games as a service") in game dev?

I have a bit more than 12 years experience in the games industry; most of that has been live-service based games. I got my start in QA at Turbine on Lord of the Rings Online during its beta, where I helped manage the beta tester feedback and interactions, as well as both development and live content testing from initial beta to launch and release of the first live expansion.

Then I moved onto another QA role at Neversoft on Guitar Hero 3, the live services part of that was a focus on the multiplayer mode for the game. I then worked at an early stage startup co-founded by John Romero on a MMORPG, where I transitioned to production of live operations and art outsource management, which included a lot of focus testing and in-game metrics analysis - unfortunately the game was not released.

I then was a producer on The Elder Scrolls Online at ZeniMax Online Studios, where I helped several departments in both development and platform - but mainly focused on managing 3d character development and outsource schedules as well as day-to-day for the initial launch and expansion content.

I then had a cool opportunity to work at the director level for an early startup, Super Evil Megacorp, on the game Vainglory - where I was in charge of live services and quality assurance, then later also full production of the live game - this included creating a QA team, putting together beta test programs, daily triage of live status with customer service, scheduling development and release of content on fixed cycles and many more things including software engineering. I now work as a software engineer on some really cool stuff I can't talk about yet at Oxide Games.

Have you ever crunched (however you define it, though we typically start at "working more than 40 hours a week because you were asked, expected to, or afraid of the consequences") on a live game? If so, why?

Yes, I have experienced crunch - I'm sure most developers have. There are many reasons it can happen - many are in our control and some are not. The most common reason for crunch I've seen is an un-achievable schedule - usually driven by legitimate business needs such as having a positive income for the month, delivering on promises to players, or meeting critical business performance goals. Sometimes it can happen because we can't hire people fast enough too. But most of the time I believe it can be avoided with proper planning and project management.

Have you ever worked for a significant amount of time on a live game without crunching? Did you or anyone else feel any pressure (from within or without) to work harder on the game, and if so, why?

Yes, the majority of my time in live game development has had no crunch at all. At every good business, there is always pressure to get the best results possible, which is balanced by good project management reality checks.

The toughest situations were for early startups, because what you do every week is critical to keeping the business alive and you don't necessarily have a full staff.

What's the key to sustainable, effective live ops game dev without crunch, in your opinion? 

The most important thing is realistic scheduling. Whenever I schedule anything I make sure there is 30% slack time for unexpected stuff - because there will always be things you can't predict - ranging from bugs and late deliveries to all-hands-needed live issues. Along with this, I stress a regular timed release cycle, such as delivering content to live every few weeks - when you do this over time you will get better at knowing what you can and cannot do in that cycle, as well as get better at executing.

Also, please please please, have live content development complete at least a week (or more) before releasing it to live. Down-to-the-wire releases are prone to live deployment issues and live bugs which lead to team crunch and larger project issues. The best live-service games I worked on for quality of life had expansion content completed months before releasing to live, and when it was released it was well tested and refined. That means if you are creating something like an MMO, then the first expansion should already be complete when launching the initial game - otherwise you risk always being in emergency mode after launch.

I've also found that having people dedicated to the live platform (IT/Servers/Live Engineering) has been extremely important for any large scale game. You never know what may happen day to day in a live environment. Its great to run scale tests and bot tests etc. but often a real live deployment has unique variables that need smart people to handle the servers and databases in real time. If the same people are also tasked with important time sensitive feature development, often something will slip.

What specific production practices, tools, systems, etc. would you recommend to other game makers working on live games?

I love agile development. Scrum is great for new feature development and anything that needs figuring out and a lot of collaboration. I prefer no more than 4 week cycles, because we are not very good at planning and breaking down tasks that are larger than that. But I often prefer Kanban for content pipelines and support tasks. Anything that is a repeat process with the same estimate is great with Kanban, such as creating a new game character from concept through abilities, or even creating levels/zones. Kanban makes you focus on cycle time - the time it takes on average to get a 'type' of thing done. So after a few cycles you just know it as a fact that start to release of a new hero character will take 5 weeks, for example. Kanban for support teams is also good because it allows you to quickly adjust planning daily and consume many small tasks with little to no wasted planning time. I like to run both side-by-side. Art, content, gameplay and live services are on Kanban. Engine, systems and feature development are on Scrum.
 
As far as tools, I'm actually not huge on one set of things over another. Whatever works, works. Often team members have opinions about what they like and don't like, so it comes down to talking with the team and figuring out what is best. However, when a team is first learning agile I always recommend starting with physical boards and cards because it helps engagement and learning, as well as collaboration in the stand-up. I've found much better team happiness and results when starting with physical whether later transitioning to digital or not.
 
Also don't bother spending a lot of time in Microsoft Project - the plan will change often and tools like this will just have you clicking the schedule and changing dependencies all day instead of actually adding value. I tend to just use excel or google-sheets and make quick block schedules one milestone as a time at 'deliverable' detail (not task-level). I only plan out 1+ years for doing long-term checks and assessing budgets - long term schedules are never actual schedules.