[For its latest feature, Gamasutra presents an extracted chapter from Clinton Keith's book Agile Game Development with Scrum, in which the veteran developer and Scrum consultant explains the nuts and bolts of Agile. In this chapter, he discusses integrating teams with Scrum processes, including best practices and potential pitfalls.]

I've worked on creating various products, from the F-22 fighters to games, for more than 20 years. The highlights of my career are clearly marked in my mind by the project teams I was working with. These teams were more consequential to enjoyment and productivity than the company or project we were working for at the time.

Working with the project team on the first Midtown Madness game was a highlight. The team was largely composed of developers who had never worked on a game before. Microsoft, our publisher, and the studio we worked at, Angel Studios, left us largely alone to develop the game. As a result, many of the smaller details of the game were left to us to discover. We were never far away from being canceled either...in some cases hours away. We had to prove ourselves.

What emerged was a team with a shared vision, a sense of ownership, and pride. We worked hard on the game. For example, we started a LAN party with the game every day at 6 p.m., and at 8 p.m. we met in a conference room to talk about improving the experience.

I'd often have an idea pop into my head during the night, and I'd rush back in during the early hours of the day to try it, often finding teammates who had arrived earlier or had even spent the night working on their own idea.

Although we spent long hours on the game, it seemed more like a hobby we were passionate about than a job, but it never felt like a "crunch."

The game we shipped was a success, but the real reward was the experience of working with that team. Much of the chemistry of that team is a mystery to me. There doesn't seem to be a formula for how such teams can be created, but I've found that it's quite easy to prevent such teams from forming. Scrum's focus is on allowing such teams to form, if possible, and nurturing them to grow.

This chapter will explore some of the basic Scrum principles and practices that support such teams and how large projects of more than 100 people can use these practices and allow individuals on teams to still have a shared vision, sense of ownership, and pride. It will also explore various team structures that have been formed on large game projects.

This chapter describes the central role of teams in Scrum, the role of leadership, and how small Scrum teams scale for large projects.

Great Teams

Great teams are one of the most influential factors for creating a successful game. Great teams are also the most difficult teams to foster. They cannot be created through the application of rules or practices alone. Studio and project leadership are required to facilitate them.

Great teams share the following characteristics:

Follow a shared vision and purpose: Everyone on the team understands the goal of what they are working on.

Complement other team members' skills: Team members depend on each other to achieve their goals by applying their unique skills to a shared goal.

Exhibit open and safe communication: Team members feel safe to communicate anything to one another.

Share decision making, responsibility, and accountability: The team succeeds or fails together, not as individuals. Everyone earns their spot on the team daily. There is no room for titles or egos.

Have fun together: They spend time together and enjoy each other's company. They care for one another.

Deliver value: Great teams take pride in their work and deliver high value consistently.

Demonstrate shared commitment: Great teams have a unified cause. When one member has a problem, the entire team will pitch in to help them out. As a result, great teams deliver value because they focus on the whole rather on their own parts. Great teams are committed to their goals. They'll go the "extra mile" to achieve a goal that they believe in.

Scrum creates a framework, through its practices and roles, to support these teams. They require facilitation and support of leadership and management to evolve. Great teams are uncommon. They create experiences -- like the one I mentioned in the chapter introduction -- that people strive to be a part of over their entire career.

When baking a cake, a few ingredients are needed before you start. If you are missing any of these, such as eggs, flour, and so on, you can't make a cake. However, just how these ingredients are prepared together and baked into the cake is the main difference between a memorable wedding cake and something that might taste like it that came from an Easy-Bake oven.

Leadership and talent are the required ingredients for a great game, but like the cake, how these ingredients are brought together, such as in a team, is the main determinant of the quality of the game. Scrum doesn't provide the ingredients for great teams but helps them "mix and bake" what's there to achieve that goal.

A Scrum Approach to Teams

Scrum creates conditions that enable such teams to achieve greatness through its practices and principles:

Cross-discipline teams: Enables teams to deliver features and mechanics that have clear value to customers and stakeholders

Self-management: Enables teams to select the amount of work they can commit to every sprint and complete that work through whatever means they find appropriate

Self-organization: Enables teams to have a degree of authority and responsibility to select their membership

True leadership: Provides leadership focused on mentoring and facilitation to free the best performance possible from the team

The rest of this section will examine the principles and practices in greater detail.

Experience

"At the heart of scrum is the interaction of the team. A daily meeting around the task board is interactive, vibrant, collaborative, visual, and tactile. It is a visual way of showing the goal the team is striving toward and the progress they are making. They, each and every member of the team, are peers.

"They own the goal. It's a team effort. They gather around the board to align themselves with each other, to honor others' contribution to the effort, and to course-correct when they are missing the mark. They argue, discuss, share, learn, continually improve, celebrate, boost each other up, and create solutions.

"There is another thing that Scrum does for the team: It creates transparency. Since Scrum depends on collaboration and continual forward progress, problems are addressed by the team as they crop up instead of dealing with them later or covering the problem under a layer of 'spin.'

"A structured, militant environment will never create a team. A team works together toward a shared goal. A group works together toward a goal given to them. Scrum is messy and noisy. It lives, it breathes, it stretches, it morphs, and it expands. Interaction is the heart of the team. The heart of Scrum is the team."

-- Shelly Warmuth, freelance writer and game designer

Cross-Discipline Teams

When the various documents are written and schedules are created, the priorities of each discipline's schedules don't often mesh. Programmers often read the design document and architect a number of systems based on the goals established in the document. Complexity and risk prioritize this work, not feature value.

For example, if the design identifies characters that walk on walls, then they architect that requirement into the character system. This requires a great deal of work to alter the physics system and the camera system. The programmers consider these changes as high priority, since they affect core systems at a fundamental level. As a result, they begin working on these changes from the start. The problem is that the "walking on walls" feature may not be very important to the designers. The feature may even be dropped when it is seen.

This lack of synchronized prioritization between disciplines leads to delays in building knowledge about the game: knowledge that comes only from using each mechanic in a working game.

Scrum requires a synchronization of the disciplines every sprint. This forces change in how each developer works daily regardless of their discipline. A cross-discipline team uses value to explore a solution, which addresses the needs for technology, design, and animation.

This drives changes in the way each discipline works to avoid one discipline getting too far ahead of the others, such as creating speculative architectures.

Programmers on a Scrum team may eventually adopt test-driven development practices, discussed in Chapter 10, "Agile Technology," to enable value-prioritized development without the cost of late changes that up-front architectures attempt to avoid.

Cross-discipline Scrum teams minimize the delays and costs that are incurred by large discipline-focused hierarchies. Team members share the same goal and therefore the same priorities, which encourage collaboration. Practices such as the daily scrum reinforce a team's commitment to the sprint goal and to solving the problems that ordinarily would "fall between the cracks" between the disciplines on a daily basis.

Self-Management

Scrum addresses the problems of communication on large teams not by adding management layers but by dividing the project staff into small teams. Scrum teams are usually composed of five to nine cross-disciplined developers who take on major game features and create vertical slices of those features every sprint. Teams take on an increasing level of self-management by doing the following:

Team self-management doesn't happen overnight. It requires mentoring and practice to achieve. It requires trust to be built between management and the teams and the clear definition of the dividing line of responsibilities.

Self-Organization

Team self-organization is the most challenging practice for teams trying to achieve self-management. Self-organizing teams select their own members whom they believe can help them achieve the best results.

The benefits of self-organization are an essential part of a self-managing team. When teams "own" their membership, then they treat team commitments with a great deal more ownership. When people are assigned to a team built by management, it's something that they have no control over, and, as we've seen, a lack of control prevents full commitment.

Teams are allowed to change their membership between sprints, but most often they only make changes before the first sprint of a new release. Shortly after a release plan is discussed with the project staff, they negotiate among themselves to exchange members. The teams take the following into account when self-organizing:

What are the release goals and initial release plan? Sometimes release goals require a complete reorganization of the teams. For example, a game that has focused on single-player mechanics might work on online gameplay during the next release. This might require a new distribution of skills.

What disciplines and skills are required to implement the release goals? For example, if a team that has been implementing a shooting mechanic is now being called upon to create AI characters that shoot back, they need to bring an AI programmer on board. If some simple changes need to be made to the codebase, perhaps a junior programmer is the best fit.

What are the priorities of the release goals? If teams compete for people, the higher-priority release goal determines where people go. For example, if both a shooting and driving team need AI and there is only one experienced AI programmer, the higher-priority goal determines that the AI programmer first goes to the shooting mechanic team.

Do the team members have chemistry? Although it is often ignored, the chemistry of people working together on a team has as much impact as any of the practices. For example, teams often benefit from having one outspoken member but suffer from having two. Both might be equally talented, but together they just don't work well. Teams recognize this and should be able to control their membership to avoid it.

Like many other Scrum practices, a team isn't expected to master self-organization from the start. Most studios starting Scrum avoid this practice at first and slowly over time allow the team a greater degree of influence over who is added and removed from the team.

Eventually the team will make the decisions on their own, with leadership influencing their decisions and providing help with conflicts and problems that exceed the team's authority. The rewards are profound. Self-organizing teams deliver unequalled levels of performance and enjoy their work far more than conventionally managed teams (Takeuchi and Nonaka 1986).

When people first hear about self-organization, they are skeptical. It reminds them of painful childhood memories of not being chosen for a sports team. Inexperienced teams treat this practice as a popularity contest. They will need the assistance of management to help them make the best choices (see Chapter 16, "Launching Scrum"). Experienced agile teams, which understand team commitment, end up making better choices that benefit their velocity.

Occasionally, teams "self-organize people off" between sprints. The team removes poor performers or poor team players (see the sidebar "When Someone Is Kicked Off a Team"). This is necessary to allow teams to truly take ownership and commit to their work.

When Someone Is Kicked Off a Team

It's pretty rare for a team to unanimously eject someone from their team. Usually the person ejected has ignored months of team and leadership feedback about their poor performance or teamwork. However, being ejected from a team cannot be ignored. It is a strong statement from a group of your peers.

After this occurs, another team willing to take this person has to be found. They have to be made aware of the issues that led to their ejection from the last team. Teams cannot be forced to accept people they don't want.

If it's the first time this person has been ejected and several other teams are around, the person will be able to join another team easily. Most of time this person corrects the issues and becomes a valuable member of another team, or the person simply finds a team where the chemistry is better.

On rare occasions, they don't work well on the next team either. After a few ejections, it's common to find that no other team will accept this person. It then becomes management's duty to release this person from the company. Sometimes the person gets the message and leaves before this happens.

I've only seen this happen a few times. It's unfortunate, but it makes a statement to the teams: They control their destiny. They are responsible and have the necessary authority to make changes to improve their performance. When this authority doesn't exist, it doesn't allow a team to achieve its potential. Teams that can't self-organize will feel that they are stuck with the members on their team and there is nothing they can do about it. They feel helpless to make the change they need and don't make the same level of commitment they possibly could.

When you see the performance of teams that achieves their potential, you stop questioning the value of these practices. It does take a leap of faith to allow them into your studios. Unfortunately, some don't reach this point and don't see the potential of great teams realized.

Team Size

Scrum literature recommends teams have sizes of seven to nine members (Schwaber 2004). This is based on studies (Steiner 1972) and firsthand experience that shows that team productivity peaks at these sizes.

A challenge for agile game development is to build cross-discipline teams that don't exceed this range. For some teams, the number of disciplines desired to achieve a goal can be large. For example, a level prototyping team may need the following:

This is a 12-member team that begins to exhibit some of the problems seen on larger teams. For example, some members are more likely than others to not speak up and be heard. This inhibits the team from raising their performance through commitment and shared accountability.

Another problem with larger teams is that subteams or cliques tend to form. I was on a team such as the previous one. The designers and artists formed separate cabals that raised communication barriers. Whenever I visited one of these cliques, they would criticize the other.

These criticisms weren't shared between the two factions, so problems lingered. This had a major impact on the quality of the prototype levels and the speed at which they were created. ScrumMaster intervention eventually resolved this, but a smaller team would have self-corrected this problem sooner.

A team like this might consider separating into two teams with smaller goals that "stage" the development of prototype levels, but that introduces dependency and accountability issues. I encourage teams to try different arrangements for a few sprints, and if that doesn't address the problems, they can reform into a larger team again.

Some studios have used teams of three to five people in size and report that it worked very well.

Drawing the Line

At High Moon Studios, we established some "laws" that were meant to describe the practices and rules that projects and their teams had the authority over and others that they did not. We referred to these laws as the "state laws" that defined project and team authority and "federal laws" that defined decisions that were made for them.

For international readers, in the United States federal laws govern the entire country, while state laws govern those within a state. If the two conflict, federal laws take precedence.

One example of a federal law was the use of a studiowide engine and pipeline technology. Studio management didn't want the teams creating significantly different engine and pipelines for their own games. We wanted the benefits that came from individuals understanding a shared technology as they moved from one project to another. An example of a state law was how the project organized themselves into individual teams and implemented items from the product backlog.

Leadership

When I was a child, my father decided to teach me to swim, as his father had taught him, by tossing me into a lake where the water was over my head. After watching the bubbles come up for half a minute, he dove in and pulled me out. I didn't learn to swim that day. In fact, I learned to avoid even trying for the rest of the summer.

With my children, we have adopted a more gradual approach of coaching them to take on greater swimming challenges from a few seconds of dog paddling through the point where they can swim back and forth across an Olympic-sized swimming pool.

Leadership in an agile organization has a similar challenge. Agile organizations need to grow leadership at every level but find the approach between micromanaging teams and throwing them in over their head that will bring about success. Both of those extremes will lead to failure.

Project Leadership

The responsibilities of project leaders (lead programmers, lead artists, lead designers, and so on) may change as teams adopt agile:

Design and planning: Leads still define the design (gameplay, technical, concept, and so on) for their discipline in concert with the other disciplines and oversee how the design is implemented.

Resource allocation: Leads will estimate how many people in their discipline are needed on the project, what areas they will work on, and approximately when they will work on them, but these will only be estimates. The teams will slowly take over the responsibility of identifying areas of need on a per-sprint and even release basis.

Task creation and management: Leads no longer break down work into tasks that they estimate, assign, and track. Teams manage this themselves. Leads still participate in sprint planning and helping members of their discipline improve their task identification and estimating skills.

Review and promotion: Although leads may continue to review every member of their discipline on a regular, usually annual, basis, the performance of the team becomes a more important part of their the information for the review (see the "Reviews" section).

Mentoring: Leads work with less experienced developers to improve their productivity. The lead role shifts from managing primarily through project management tools to one where they "go and see" what is occurring with each developer as frequently as possible (see the "Mentoring" section).

Team self-management challenges the lead role definition. It's difficult for many leads to give up making detailed decisions for teams on a daily basis and allow them to risk failure, even for smaller challenges. However, the benefits of growing self-management behaviors become apparent as some of the more mundane management duties of a lead, such as task creation, estimation, and tracking, are taken over by the team. For example, a project staff of 80 developers generates and tracks approximately 1,600 tasks during a four-week sprint.[1] This is an overwhelming volume of detail for any lead group to manage and draws their time away from the more valuable leadership duties of their role.

Mentoring

The most important role of the lead is to mentor developers to improve how they work. An example is when lead programmers pair with associate programmers to teach them how to improve their coding and design practices.

Junior programmers often implement simulation solutions that are far too expensive in the CPU resources they consume. I recall one new programmer who was tasked with implementing a wind effect. They started implementing a complex fluid dynamic engine that used 90% of the CPU time to simulate thousands of air particles. A lead programmer intervened and showed them a few tricks to introduce a good effect in a few hours that required hardly any CPU time.

Scrum creates opportunities for leads to continue working on games and lead by way of example instead of through a spreadsheet. Rather than spending half their day with a tool creating and tracking tasks, they interact with people working one on one.

[1] 10 teams × 8 people × one task per day × 20 days per sprint

Reviews

Another critical role of leadership is to provide career support for the developers in their discipline. In companies that employ a matrix management structure, this takes the form of a yearly management and salary review. This reinforces discipline-centric performance over team-centric performance.

For example, if an artist is evaluated on how productive they were creating assets over the past year, then they focus on faster asset creation to improve this metric. As a result, when a teammate interrupts the artist about a game problem, it reduces the number of assets they create; the artist then tries to isolate him or herself to reduce these interruptions for the benefit of a better review. This is not a good cycle.

Leads in agile environments have introduced frequent team-based peer reviews to supplement, if not replace, the yearly review process. This allows feedback about teamwork and cross-discipline collaboration to be introduced. Individual lead roles for each discipline will be described in greater detail in coming chapters.

Director Roles

The game industry is filled with director roles, such as art directors, technical directors, and so on. Usually these roles are given to members of a discipline who show the greatest level of craftsmanship but who also have authority over the work, rather than a group of people.

Often these roles exist to oversee and approve or disapprove of the work being done in their area. Scrum teams need to adjust their practices to meet the needs of these roles, such as described in Chapter 11, "Agile Art and Audio."

Game Teams and Collaboration

Game development requires a high level of collaboration between diverse disciplines. For example, a character in a game has animation, physics, textures, models, sounds, and behaviors that need to work seamlessly together to produce the whole. A single discipline cannot accomplish all of these functions alone. They need to work closely with other disciplines to create features.

Unfortunately, as team sizes grow, disciplines tend to pool together. This delays integration of their work and leads to many problems. Scrum focuses development on frequently integrated features that drive close cross-discipline collaboration. Daily cross-discipline collaboration leads developers to think of themselves as game developers first and as programmers, designers, arts, QA, producers, and so on, second.

In this section, we'll examine various team structures that are used by agile game teams to promote collaboration across the project and across disciplines.

Collaboration, Interrupted

There is no shortage of ways in which companies try to build morale with large cross-discipline teams. I've been involved in a few of these potentially dangerous exercises myself in the past.

I still recall the day that a team-building exercise nearly maimed me. It was on a paintball field located in high chaparral land east of San Diego. I was lying flat on my back, nearly out of ammunition, while nearly 30 electrical engineers were trying to shoot me.

I was a young software engineer working for a military avionics company. During my career in the defense industry, I witnessed the animosity between electrical engineers and software engineers. To the electrical engineers, we lacked true engineering discipline and were overpaid. They often considered our code as a "necessary evil." We saw the electrical engineers as elitist in attitude and outdated in their technical philosophy.

Personally I believed the electrical engineers hated us because we were often the heroes of a project. The software we wrote often worked around the flaws in their hardware that threatened a project in its final hours.

It started with a division into two teams. Naturally, one team consisted of the electrical engineers, and the other consisted of us software engineers.

I won't lie and say we were better fighters that day. We weren't. I won't make excuses and accuse them of cheating, although they brought some suspicious-looking tools with them. The plain fact was that the software team lost most games, and we lost them badly.

I faced the greatest challenge during the last game of the day. We were playing an elimination match in a small plywood "village." The goal of the game was for one team to eliminate every player on the opposing team to win.

It was another bloodbath for the software team. We were quickly decimated. I survived by hiding, with several other programmers, on the roof of a plywood hut. The partial cover of three walls protected us, and the only way to enter the roof was through a hole in the floor from the room below.

One by one, my roof mates were killed off in heroic displays of gallantry and ignorance of the value of cover. I was content to hunch low and survive.

Suddenly the referees blew their whistles signaling the end of the game. They believed that all the software engineers were dead! I jumped up, roaring in defiance at what I hoped was one or two remaining enemies.

My roar was quickly choked when I saw that nearly the entire enemy team of 30 electrical engineers was still alive. Endless seconds seemed to pass as we considered each other. Then one of the referees announced, "Game on!" and blew his whistle. I will never forget the sight of all those electrical engineers, shouting with gleeful nerd rage, running toward me as I ducked back into cover.

I held out for a while. I even managed to kill a few of the enemy engineers. I would love to say that I was a hero that day, but it was not to be. Someone eventually shot me. The electrical engineers completed their victory over us, and we went back to work with renewed feelings of antagonism.

Feature Teams

Features teams are cross-discipline teams that develop core game features. For example, the small cross-disciplined team shown in Figure 8.1 could take full responsibility for a driving mechanic.

A major benefit of feature teams is the sense of ownership they experience. Participating in the full development of a few mechanics is far more satisfying to most developers than participating part-time on many. For many developers, this gives a greater sense of accomplishment.

Figure 8.1: An example driving mechanics team

A feature team should have everyone they need to build the mechanics. In practice, this is difficult to accomplish. Sometimes teams need to share disciplines in short supply. An example of this is an effects (FX) artist who is utilized 25% of the time by any single team. Often this person is shared among multiple teams.

Functional Teams

Functional teams are composed of developers who are mostly of the same discipline yet still work on a key feature. Although less common than feature teams, functional teams have their benefits.

One example is a platform team comprised of mostly programmers. These programmers are experts in optimizing performance for a particular platform such as the PlayStation 3 (PS3) or Xbox 360. Concentrating these individuals on a single team focuses effort on challenging problems that are wasteful for feature teams to solve on their own. For example, if the PS3 programmers are spread across multiple teams, then their efforts creating a working PS3 build are diluted.

Functional teams are typically used only for foundational or infrastructure work. Using functional teams for higher-level, cross-discipline work often leads to solutions that are more suitable for the discipline than the game.

An example of a poorly conceived functional team was one formed to create the character AI for a game. This team consisted of AI programmers who wanted to create the best-architected AI possible. Unfortunately, their efforts led to AI functionality that was handed off to other teams that neither understood how the AI worked nor benefitted from many of the features designed into it. Ultimately this team broke up, and the AI programmers were scattered around the project to implement AI needed by various teams.

Production Teams

Production teams are cross-discipline teams that are used for game development projects that have a production phase. These teams have a more defined pipeline of work for creating content and applying some of the lean and kanban practices described in Chapter 7, "Video Game Project Planning."

Production teams may exchange members as needed with other production teams to maintain a steady flow of asset creation. Production teams often form from feature teams as a mechanic transitions from pre-production to production. For example, most of the programmers might leave the level pre-production team as it enters production, to be replaced with more modelers, texture artists, and audio designers.

Shared Infrastructure Teams

Shared infrastructure teams provide shared support services such as engine development and cinematic and audio services that multiple games rely on. These teams are dominated by one discipline, such as composers on the audio team, but also have other disciplines on the team, such as programmers, to support their pipelines and tools.

A frequently asked question is how shared infrastructure (SI) teams should organize themselves in an agile project environment. Since they support multiple teams, they receive requests for features that cannot be as easily prioritized as they are for a single team.

This can create confusion and conflict between the SI team and their "customers," the games that depend upon them.

There are a number of valuable practices for these teams:

Tool Teams

A tool team consists of a number of tool creators (programmers, technical artists, QA) whose customers are users of a common tool set and pipeline.

Like a shared infrastructure team, a tool team often supports multiple projects and has their own product backlog.

Tool teams have the added benefit of releasing tools to customers who are in the same building, not just stakeholders who represent them. Having tool users who can participate in backlog definition, prioritization, planning, and reviews is a major benefit to the tool developers. Tool development can be even more exploratory than game development and benefits greatly from using an agile approach.

Pool Teams

A pool team is a collection of developers from a single discipline. Unlike other teams, they don't have their own sprint goals. They exist to support teams that do. Examples of this are a pool of animators that could support a feature team that needs a large number of animations in a single sprint.

Another benefit of a pool team is to provide a service center for art production. This is also referred to as in sourcing. Environment artists and animators are in greater demand during production, and pool teams help level the resource requirements in a larger development studio.

Pool teams require more planning and management during a release to ensure that they are fully utilized. Pool teams are more commonly used in late pre-production or production.

Integration Teams

It's challenging to share a common vision with larger projects with more than 40 developers. The vision among separate teams can drift, even with a hierarchy of product owners. As a result, some projects have "integration teams" that integrate mechanics, developed initially by feature teams, into a unified experience.

These teams are similar in structure to feature teams. The difference is that they are responsible for the overall theme of the game. For example, in an action-racing game, eventually the core team takes over the driving mechanics from the team that developed it once it is mature enough. From this point forward, they modify and maintain both mechanics to seamlessly work together.

Objection: Pool Teams and Core Teams -- These Don't Sound Very "Scrum" at All 

Ideally, every team should be a functional team, but the sheer number of specialties required for game development leads to pool and core teams. Scrum is more about teams finding ways to perform best rather than "following the rules out of the book."

Scaling and Distributing Scrum

Although the ideal Scrum teams size is five to nine people, modern game development projects typically require more developers. Sometimes the developers are separated across multiple locations. Scrum has practices that allow it to scale and distribute multiple Scrum teams.

This section will examine the practices commonly used to scale up Scrum: holding the Scrum of Scrums meeting, creating a hierarchy of product owners, aligning sprint dates, creating communities of practice, and avoiding dependencies. It will conclude with discussing the challenges and solutions of distributed development.

The Problem with Large Teams

Project staff sizes for many console and PC games have grown steadily from the "good old days" of the lone developer who designed, coded, illustrated, scored, and tested the game on their own to project team sizes that now often exceed 100 people. Unfortunately, the effectiveness of a 100-person project is usually not 100 times that of a one-person project. This loss of effectiveness has many sources, the main one being the overhead of communication.

Consider the combinations of any two people who may need to communicate on a project. These "lines of communication" grow much faster than the number of people on the project (see Figure 8.2). For example, a project with 100 people has 4,950 possible lines of communication between members.[2] This is far too many for everyone on the team to grasp in order to know who to talk to when a question or issue arises. As a result, hierarchies of management were created to oversee this complexity.

Figure 8.2: Team size and lines of communication

Such hierarchies create barriers between any two people who need to solve problems quickly. For example, if an animator needs a fix in the animation code, they have to send a request for their change to their lead.

That lead, in turn, passes the request to a project manager. The project manager sends the request to the lead programmer who then communicates the change to a programmer to make the change. This entire process depends highly on every link in the communication chain working quickly and on the information being passed accurately. Unfortunately, this doesn't usually happen.

[2] The formula for lines of communication is n*(n – 1)/2, where n is the number of people on the project.

The Scrum of Scrums

The central practice for scaling Scrum is the Scrum of Scrums meeting. Figure 8.3 shows how a larger project could divide into subteams and how each team sends a member of their team to the Scrum of Scrums.

Figure 8.3: The Scrum of Scrums meeting

This meeting enables one or more representatives from every team to gather to inform other teams about their progress and impediments. Who attends the meeting often depends on what needs to be reported. It's very effective for identifying shared or potential problems that one team can solve for all the others.

For example, an engine technology team often works with multiple teams to improve the engine technology. Changes to this technology often create impediments for teams when rolled out because of unforeseen bugs.

Imminent changes to the shared technology are described at the Scrum of Scrums so any resulting problems are quickly identified and resolved. In this case, a technical lead from the shared technology team attends the meeting to report the pending changes.

The Scrum of Scrums is different from a team's daily scrum meetings:

It doesn't have to be daily: It can be weekly or semiweekly. The group that meets should decide on the best frequency for it.

It is for problem solving: The meeting is not timeboxed to 15 minutes. Potential solutions are addressed in the meeting. This meeting may be the only time when these individuals meet during the week, and the problems they discuss have larger impact on the project.

The questions are different: The meeting starts with everyone answering slightly different questions:

It's important to discuss a whole team's progress and not the progress of each individual on each team when answering the first two questions at the Scrum of Scrums. Otherwise, the meeting takes far longer!

The Scrum of Scrums doesn't have a product backlog, but it creates a short backlog of shared impediments that are addressed at every meeting. An example of a shared impediment is when the one FX artist for the studio is out sick and it impacts multiple teams. Many impediments that are identified take days to resolve, so tracking them is beneficial.

A ScrumMaster for the Scrum of Scrums isn't necessary, but teams often assign one of their members to the role to help keep the meeting on track.

A Hierarchy of Product Owners

The demand for a product owner's time on game teams can be greater than the demand for product owners in other industries. The reason is that teams are challenged with knowing if the "fun" they are creating is the "fun" the product owner wanted. Questions such as how "bouncy" a physics response should be or how "snappy" an animation transition needs to be are subjective and may need daily feedback from the person who owns the vision for the game: the product owner.

For large projects, with a dozen or more teams, this creates a problem. The product owner's time becomes spread too thin, and they cannot effectively maintain a shared vision for the game across all teams. Without a shared vision, each mechanic will drift from the original vision as it evolves, and the game becomes less consistent and appealing.

An effective practice is for the product owner for a large project to delegate some ownership. One way of doing this is to establish a hierarchy of product owners. A lead product owner guides the project, and each core mechanic has a product owner. Figure 8.4 shows an example of such a product owner hierarchy.

The lead product owner oversees the two product owners who work with one or two Scrum teams. The lead product owner continues to work with teams directly, such as the user interface team, but they delegate "product owner as pig" responsibility to the teams that have their own product owner.

Each product owner works with their teams during the sprint, helping them plan the sprint and working with them daily to ensure that they achieve the sprint goal. For example, as a product owner on a team implementing a driving mechanic, my role included educating the team about the shared vision for the mechanic. This often required conversations about the balance between a "sim" vs. "arcade" feel for the controls, vehicle physics, and the environment.

Figure 8.4: An example product owner hierarchy

The product owners need to ensure that there is a shared vision for the entire project. This includes frequent meetings among them all, including the Scrum of Scrums meetings, to address any questions about the game's vision.

Product owners take direction from the lead product owner between sprints and release planning. There is often a difference of opinion on the best path to achieve release goals between the product owners. The insight of a team's product owner is invaluable in finding the best path, but the lead product owner is responsible for safeguarding a consistent vision for the game across all mechanics and features.

A product owner team creates a shared vision on a large project and ensures consistency of vision everywhere.

Integration teams are a type of team that helps ensure a shared vision on a large project is maintained as the features and mechanics are integrated.

Aligning Sprint Dates

Separate Scrum teams working on a project may align their sprint planning and review dates or have independent schedules. Figure 8.5 shows the difference between the two dispositions.

For teams with independent schedules, there are some benefits. The biggest one is that each team doesn't have to vie for time with the product owner. For multiple teams with aligned dates, it can be challenging to schedule the product owner's time, especially for planning the next sprint.

Nonetheless, it's usually best to align the sprints (Cohn 2009). The benefits to the game are as follows:

Teams can exchange members: Following the sprint review, nobody has a commitment to any sprint goal, so it is easy for teams to trade members for the next sprint.

An integrated view of the game is encouraged: Teams with the same sprint review date can integrate all work into a single build so that the entire game is reviewed. This encourages more cross-team collaboration and an integrated view of the game.

A hierarchy of product owners on larger teams eliminates the problem of the lead product owner being spread too thin when teams using synchronized sprints need to plan the next sprint.

Figure 8.5: Independent and synchronized sprints

Communities of Practice

Another challenge created by large Scrum projects is the potential loss of communication caused by the separation of discipline, or functional expertise, across multiple cross-discipline teams. For example, if all the graphics programmers are spread across multiple teams, what is to prevent them from solving the same graphics problems in different ways?

We had this problem when all the AI programmers were split up across three feature teams. Each team implemented a unique AI state machine. One team implemented a script-based system, another implemented a C++-based system, and the third team developed one that was manipulated with a graphical interface.

The solution is to establish communities of practice that can share knowledge and eliminate the duplication of effort. Figure 8.6 shows how the AI programmers from across multiple Scrum teams can form an AI community of practice.

Each community can decide how frequently they need to meet and address the issues they are facing. The AI community might discuss common solutions they could each implement.

The ScrumMasters can form a community to share improvements to their team's practices. The designers could form a community to complain about everyone else.[3]

Communities of practice do not have their own sprint goals or assign work outside their own teams. Their only purpose is to share information.

Figure 8.6: An AI community of practice

Avoiding Dependencies

Interteam dependencies inside a sprint can prevent teams from achieving their goals. Consider a team whose sprint goal is to implement a wall-climbing mechanic but has to rely on another team to provide the animation.

Because of the separation of teams and goals, it's likely that the mechanic team will hand off their work to the animation team near the end of the sprint rather than collaborating daily. At best, this limits the number of iterations that can occur with the mechanic. At worst, the goals that the animation team has for their own sprint might prevent them from handing back the wall-climbing animation in time.

When projects begin using Scrum, these dependencies are quite common, and they are the source of many impediments and sprint failures. Over time, teams change their membership to reduce dependencies and establish other practices to prevent their impact.

Changing membership to create more self-contained, cross-discipline teams is the easiest solution. If the team implementing mechanics needs full-time animation work throughout the sprint, having an animator join them is best.

In many cases, there isn't enough work to justify a specialist joining one team full-time. In these situations, teams can share specialists within a sprint or trade them between sprints. Doing this requires a bit more planning and foresight to avoid overlapping demands for a specialist's time. There are two places where this is done: at release planning meetings and at lookahead meetings.

Release Planning

In release planning, teams identify potential sprint goals for the next several sprints. Using these goals, they identify sprints where part-time specialists or a concentration of disciplines (such as a bunch of texture artists) might be needed. Often these will uncover conflicting needs among teams.

The best way to resolve these is to raise or lower the priorities of stories creating the conflicts. For example, if two teams require the same FX artist full-time during the same sprint, then the product owner changes the priority of one of the stories requiring FX work enough to shift the sprint for one team to remove the overlap.

Lookahead Planning

Release planning does not identify specific goals for more than several sprints because change is more likely. As a result, regular lookahead planning meetings are held during the release to update the goals for approaching sprints.

Lookahead planning takes an hour or two during each sprint. It can be combined with prioritization meetings. It identifies changes to team membership that may be necessary and any pending conflicts that the product owner and teams need to navigate around.

Borrowing an Audio Engineer

Our team was working on a driving mechanic. We were able to implement much of the simple audio ourselves. However, the sprint goal of adding complex audio behavior for the drivetrain was approaching. This required engine sounds that would be realistic throughout the entire RPM range and blend between gears.

This was a difficult problem to solve, especially since our vehicle was licensed, and the drivetrain physics (revolutions per minute [RPM] and torque curve) didn't match those of the actual vehicle. We asked the audio pool team if we could borrow one of their audio engineers for the sprint. We moved our goals around a bit with the product owner to accommodate a sprint where that team could free him up. This worked out well for both teams.

Problems occur with little warning on a day-to-day basis that require a specialist on another team to help out. For example, one of our projects had one UI scripter that could implement UI changes rapidly. Almost every day he was requested to help another team for an hour or so. Because of the demand for his time, his team would allow him to commit to only half the available hours during a sprint.

Requests like these can be handled in the Scrum of Scrums meeting described earlier. Whether or not a specialist can help another team within a sprint lies with the current team to which the specialist belongs.

Scrum doesn't solve the problem of specialists who become bottlenecks, but it makes such problems transparent and therefore easier to solve. In the case of the UI scripter, the solution might be to hire more people who can script or cross-train others to be able to write UI scripts. The ideal solution depends on the project and the studio's needs.

Distributed Teams

To reduce costs and help balance staffing demands, studios frequently distribute the development of games. With this model, teams distributed across two or more locations develop core mechanics and features of a game in parallel. This is different from outsourcing, which typically focuses on distributing certain types of production work, such as asset creation or technical support.

The challenges of distributed development are mainly those of communication, which are much more likely to impact distributed teams.

This section examines the challenges that face distributed teams and some of the agile tools available to overcome them.

Challenges

Three common challenges affect distributed teams:

They lack a shared vision: It's more common for distributed teams to experience their visions "drifting apart" because of physical separation. This divergence leads to conflicting or incompatible efforts from the teams.

They have a less collaboration: Physically separated teams cannot collaborate as closely as colocated teams. If the differences in time zones is great enough, a single question can take a day to answer.

Iteration and dependencies can destroy the benefits: The potential savings in cost for distributed teams is easily lost when time and effort is wasted through iteration delays and dependencies between teams.

[3] One design group I knew actually did this…they were able to keep it mostly constructive!

Agile Tools

Many of the agile practices discussed help distributed teams overcome these challenges. They give teams the opportunity to maintain a shared vision, increase collaboration, and avoid going down separate paths.

Scrum Teams Align with Location. Usually each distributed team is a separate Scrum team. There are occasions when it's beneficial to have members of a Scrum team distributed to share knowledge and create bonds between locations (Cohn 2009), but for the most part having each Scrum team colocated is best.

When each Scrum team is colocated, they can more effectively collaborate on a shared sprint goal. Such teams need a local product owner but should find ways to hold sprint review and planning meetings with other teams and the lead product owner either in person or through avideo-conferencing system.

Shared Scrum of Scrums. A video- or phone-conferenced Scrum of Scrums meeting is essential for distributed teams. These don't have to occur every day, but they should be held at least once a week.

If teams are spread across many time zones, the time of the conference call should not be fixed so as to impose a constant burden on one team more than the others. The meeting time is changed on a regular basis so that attendees from different locations have to come in early or stay late.

Shared Release Planning. It's critical for a release plan to be developed and shared to the greatest degree possible among the teams. Often this will include flying one or more members from each location to a central place where the meeting is held. It's often a necessary cost that cannot be avoided with distributed teams. You either spend money on airfares and hotels or spend much more on the costs incurred by divergent goals.

There are many ways to organize depending on the how the teams are distributed. The sidebar "A Global Game Development Story" tells the story of how one company organizes their releases across three continents and sixteen time zones.

Improved Sharing of Builds, Assets, and Code. A problem with large projects is how to share the large number of changes that occur. Frequent small changes can perpetually break the build, and bulk changes committed weeks apart can bring teams to a halt for days at a time.

With colocated teams, this problem is bad enough. With distributed teams, defects passed across in shared builds, assets, and code can be disastrous. When a single question can take a day for an answer, tracking down a problem and the necessary expertise to solve it consumes days rather than hours or minutes. Extra care must be taken to protect distributed teams from external defects. This requires a focus on improved commit practices and testing described in Chapter 9, "Faster Iterations."

Processes and Tools. Agile methods value "individuals and interactions over processes and tools." A wider team distribution can impede interactions and place more weight on processes and tools. Distributed teams often use more tools to track and display sprint progress and product backlogs and share knowledge. Wikis and other documentation tools are beneficial and are used more with colocated teams.

A Global Game Development Story

CCP games, the developer and publisher of EVE Online, was founded in 1997 with development split between studios in Reykjavik, Iceland; Atlanta, United States; and Shanghai, China. It has grown to have more than 400 employees and to host more than 300,000 active subscribers in a single online world.

In the fall of 2008, CCP undertook the development of its tenth expansion pack called Apocrypha. Apocrypha was the most ambitious expansion of the EVE universe. It added major technical features and significantly extended the size of the EVE world. The goal of the company was to release the expansion pack within six months following a four-month development cycle (see Figure 8.7).

Figure 8.7: The Apocrypha timeline

This ambitious goal required CCP's worldwide development studios to work in parallel. Features and content developed simultaneously across three continents had to seamlessly come together to achieve their goal. Normally this would be the introduction to a disaster story, but CCP pulled it off. CCP is a longtime adopter of agile methods, specifically Scrum.

In the case of Apocrypha, with more than 120 developers in 13 Scrum teams spread across three continents, 9 product owners were required. These product owners took their direction for the game from a project management group in Iceland.

Release Planning

The stakeholders identified two release cycles of development to release the expansion pack in March 2009. Ideally, release planning should gather every developer, but it was nearly impossible to collect 120 people from around the world in one location, so CCP had to create some innovative practices to perform release planning.

Apocrypha release planning meetings took place over a single 12-hour period. A high-definition video conferencing network facilitated this meeting and allowed every developer to take part.

Since most of the development staff, including the project management group, was in Reykjavík, the meeting was focused there. It started at 9 a.m. Because of the time difference, it was far too early for the developers in Atlanta to attend. It was the end of the day for the developers in Shanghai, so they and the Reykjavík group met first. Hours later the Shanghai team would leave, and the Atlanta team would join to discuss the release (see Figure 8.8).

Figure 8.8: Overlapping meeting times

You can see a time-lapse video of this amazing meeting in Reykjavik on YouTube.

It's important to note that the time zone limitations and the limitations of video conferencing allowed only two teams to meet at one time. In this case, Atlanta and Shanghai could not attend together. To avoid potential problems, two developers from Shanghai flew to Reykjavík to participate at the core meeting and to represent the Shanghai teams.

Each pair of locations would discuss the release goals and begin breaking them down into smaller goals that were reasonably sized to fit into a sprint. The conferencing network enabled multiple simultaneous meetings to occur. This was necessary since many questions raised during this process required a great deal of live conversation among the entire staff or individual teams.

The goal of the release planning meeting was to generate a set of potential sprint goals for each of the 13 teams that would fulfill the release BHAGS. This wouldn't be the last time that the teams would communicate. Daily issues were communicated, and a build of the game was demonstrated every sprint, which were two weeks long. Changes to the release plan were made every sprint to adjust for the realities of development.

After two releases, Apocrypha was ready to do final testing and polishing, and on March 10, 2009, it shipped on schedule.

Solving the Problems

A globally distributed development team can be successful in creating a high-quality product within budget and within the schedule limits. The raw ingredients for this success can be summarized in several points:

Local vision and ownership: Scrum enables individual cross-disciplined teams to take ownership of their sprint goal and achieve it independently. Having a product owner on-site who is responsible for the vision of what the team is creating is essential.

Iterative development methodology: Creating an integrated, potentially shippable version of the game every two weeks from work done around the world forces problems to the surface and demonstrates real progress. Without this, critical problems can remain submerged until late in the project, when it is too late to avoid delays and cost overruns.

High-bandwidth communication: Communication on large colocated teams is difficult enough. On large distributed teams, it can be the main challenge. Teams must have the tools to communicate effectively, such as a networked conferencing system, that are reliable and ubiquitous, as well as a methodology that creates transparency, such as Scrum.

Distributed teams will never be as effective as colocated teams, however. What is lost from daily face-to-face communication cannot be made up through conference calls. But it can come close enough to ensure that the teams are productive.

Summary

There is no single formula for creating the best teams on a game project. The teams and practices described here result from adapting the existing Scrum roles and team to improve how people work together and how a vision is unified and shared. Exploring roles and team structures is an ongoing process.

Teams must use the retrospective practice to find ways to improve how they work together and with the stakeholders of the game. By allowing teams to take ownership and authority over some of the daily aspects of their life, they are more likely to take responsibility for their work. This doesn't happen overnight. It takes years in many cases and will create head-on collisions with studio leadership culture. The results are worth the effort.

Scrum scales up to projects of any size but drives changes in team structure, the project owner role, and practices such as the Scrum of Scrums.

Scrum also drives changes across disciplines, focusing the team on changing how they work day to day to improve communication and the pace of iteration. It also drives changes in how each discipline works. The remainder of this part of the book will address these changes.

The goal is to create teams that can take more ownership of development and have greater purpose in what they do.

Additional Reading

DeMarco, T., and T. Lister. 1999. Peopleware: Productive Projects and Teams, Second Edition. New York: Dorset House Publishing.

Katzenbach, J. R., and D. K. Smith. 2003. The Wisdom of Teams: Creating the High-Performance Organization. Cambridge, MA: Harvard Business School Press.

Larman, C., and B. Vodde. 2009. Scaling Lean and Agile Development: Thinking and Organizational Tools for Large-Scale Scrum. Boston: Addison-Wesley.

Leffingwell, D. 2007. Scaling Software Agility: Best Practices for Large Enterprises. -Boston: Addison-Wesley.

Schwaber, K. 2007. The Enterprise and Scrum. Redmond, WA: Microsoft Press.

This chapter is an excerpt from the book, 'Agile Game Development with Scrum' authored by Clinton Keith, published by Pearson/Addison-Wesley Professional, May 2010, ISBN 0321618521, Copyright 2010 Clinton Keith; to see a full Table of Contents please visit this link.