7 min read
GDC China: Playfish On Metrics Vs. Intuition
At GDC China's Social Games Summit, Playfish product director Andrew Mo dived into the debate over metrics versus intuition, sharing examples of Playfish's successes in prioritizing the latter over the former.
The first session of GDC China's Social Games Summit in Shanghai was presented by Andrew Mo, director of product for Playfish's Beijing studio. His topic, intuition versus metrics, taps right into a hot area of debate in social game development. As Mo noted, social games is "really is a marriage between traditional games and web development", and "there are some complaints that the social game space is more and more becoming a numbers-driven game, about optimization, number-crunching, and tweaking, without having the same soul and spirit that game developers are very passionate about." Crucially, said Mo, "now that we have grown to a much larger audience... We find that having a grasp of the numbers and understand our customers' behavior is critical to succeed." However, "one of the biggest reasons for failure is using the wrong tool at the wrong time." Looking Outside Games: Two Tragic Examples Mo presented two examples from the non-games business world; before the BP oil spill, the company was highly focused on improving safety metric. "Initiatives [were] underway and money was being spent... but there were clear signs of problems" even before the spill. "Metrics may not have been the best tool to use in this example to make sure you are running a safe but very complex operation." On the other hand, Lehman Brothers "was exposed to a lot of risk that the data said it shouldn't be" because the CEO of this company valued intuition, and empowered his management to act on it. Building a Balanced Culture When formulating a strategy, said Mo, "the team also matters. Make sure there is a balance between creative and data. In terms of leadership, in [traditional] game development we look for a balance between the producer and game designer," who are both creative-led. Meanwhile, web development focuses on a product manager and a designer -- one led by metrics, the other creative. Social games is the same split, with a producer/designer and a product manager. And while you should have user testing as an integral part of development, said Mo, "it's really important to separate your user researcher" from the game design team, and only feed in data as required. The creative versus metrics balance of power is key, he said. "For us, we try to manage that by having the producer and the PM really sitting at the same level and no one person really reporting into the other, and the decisions really being mutually agreed-upon decisions." Intuition Best Practices In Mo's view, highly complex problems can't be solved with data; highly simple problems, on the other hand, should be tested. He recommended the book Blink by Malcolm Gladwell -- which explains "how do you think without thinking." When you develop in a purely metrics driven fashion, there comes a point where you reach a point where you can't improve. "Whatever moves you make and whatever tweaks you make to the product will see your numbers declining... you would typically optimize yourself to a point where you can't maximize yourself any longer... But if you take off your blinders and look at the big picture you would probably find a better point." For example, in Playfish's Pet Society, "one of our designers came up with the idea that it's a pets game -- why don't we add fishing in our game? And when they introduced fishing they really saw the DAU really take off." When it comes to intuition, said Mo, "if you haven't seen or you haven't experienced a lot of the genre you can't lead. Typically, we want to lean toward intuition for complex problems and not details. And for long term planning -- how do we get toward the next big bump in my business." Intuition is useful for "creating something brand new." But when it comes to making intuition-based decisions, "try to back it up with data wherever you can." On the other hand, intuition can lead you astray -- the book to read here is Predictably Irrational by Dan Ariely, said Mo. It will teach you "why intuition is wrong." For example, in the social games space, "just that term 'microtransactions' is misleading. When you think about it, most people would think about making an item extremely low cost. But surprisingly if you actually look at the numbers, the price ceiling you can have for items in your game could be very very high." And how do you determine that ceiling? "The best way to find that out is to have a controlled environment where you can track these things." Good and Bad: Metrics When it comes to metrics, the good is that "what gets measured usually gets done. If you give your team a weekly sales target... Usually you'll find that week on week they will improve that metric, because that's the thing they're focusing on day-in, day-out." "Good metrics will drive good behavior. If you're leading with a set of numbers, those will get done," said Mo. "And good metrics will help you make good decisions. They are very good for highly measurable environments with short feedback loops and not so much noise and false data clouding the mix." However, these can be weaknesses, too. "What gets measured gets done. If your game is just taking off and has a few thousand DAU... if you are telling a team to focus on its sales metrics, that's probably not the way to go. You're giving away a lot of focus and growth behind the title." Said Mo, bad metrics will actually enforce bad behavior, especially when the metrics are founded on bad data. And in noisy environments... "you'll find yourself spending more time preparing the metrics than taking action" It's also a concern that "metrics age really quickly... It's important to stay on top of these things."