Crisis Averted: How a Critical Bug Almost Wiped Patchwork’s Entire User Base

I think it’s safe to say that the worst day of my professional career was a certain Friday in March 2016, when I woke up to more than 150 angry emails from our players in my mailbox -- with new messages pouring in every two minutes, and a multitude of missed calls. This heart attack-inducing moment (combined with the worst external circumstances imaginable) led to 48 agonizing hours out of which we walked away with higher ratings, a stronger community -- and in the long run, an unbelievable boost in sales.

Definite sign that something is very wrong.

But in order to put this nightmarish situation into context, I have to rewind a bit: I’m part of a four-man team called DIGIDICED. We founded the company in September 2015 with the goal of pooling our respective industry experience together in order to bring high quality, cross-platform multiplayer board game implementations to life. Our first product, Le Havre: The Inland Port, had been released on iOS, Android and Windows Phones in November 2015. This euro board game designed by Uwe Rosenberg about harbor management was performing well, considering the small nature of the IP -- and we had just been informed that with it we had reached 3rd place in the Unity Developer Contest 2016.

Le Havre: The Inland Port

Our second game, Patchwork, had been released at the end of February 2016. It’s a much more accessible game (from the same designer) that combines Tetris-shaped tile-laying mechanics with stone cold economics. It was performing much better than expected. We received a few high profile mentions in the press, had been featured as a hot new release on multiple iOS markets, and were very happy with the resonance we were getting on Twitter and in reviews.

Patchwork

Then came the aforementioned disaster: In an effort to curb piracy, we implemented a few methods to spot a pirate copy and kindly ask the user to buy it. It was a harmless pop-up that didn’t prevent you from playing the game. We were hoping to convert more than 50% of the “pirates” to paying customers. In addition, we added a timer in the APKs we sent out to the press. A month in, it would display a message that the “trial period” was over and ask the user to purchase a copy. Due to getting caught up in the stress surrounding the release, we neglected removed that fatal line of code. Due to its time-based nature, the fault remained undetected all the way into the release candidate.

Accurate depiction of our reaction to the disaster!

So on that fateful day in March, the time bomb in our game had exploded in our face and proceeded to kick out all players from the app across all platforms while telling paying customers: “Please buy Patchwork. Your timed trial period is over” -- fresh customers included. (We were getting around 200 sales a day during that period.)

After recovering from the initial panic attack -- trying to wake up a team already dealing with damage control since midnight -- I rushed to the office to fix this blunder ASAP and help “manage the damage.” Needless to say, this felt like the longest subway ride of my life.

The culprit was quickly found in the code just under an ALL CAPS tag -- a bold and deep blue comment line: //TODO: REMOVE BEFORE RELEASE. I removed the faulty code and built a new version -- and we uploaded it as fast as we could to all our platforms. As fate would have it, Apple had an issue with its upload server that day -- and although we had been granted an expedited review (as if there was any better reason than ours), the server issue and the weekend effectively meant that it took 48 hours for the patch to arrive on iOS. I have to hand it to the support team at Apple, though -- whom I spent most of that weekend harassing over the phone: They recognized the emergency and did everything in their power to help the update go live as soon as possible.

During that weekend, our little four-man team transformed into a full-time support crew. We replied to every single email and every single tweet -- on Reddit and on Facebook. And this is where, to our surprise, we turned this quagmire into the single best community-building effort we had ever done up to that point. The messages we received were understandably outraged and a few outright insulting. But by remaining calm, friendly -- and most importantly, honest -- to our players, we received a second wave of messages that were completely outside of what we expected. Almost all our users were impressed by the speed of our response, and some were surprised to even receive a reply at all. With a few exceptions, all the 1-star ratings that came pouring in during that period had been changed to 5-star ratings -- and many promises were made to buy all products ever developed by us :)

Accurate description of how it felt to read those replies

A week after this issue, we uploaded another free small update containing a pack of new profile pictures that users were wishing for (like cats!). By that time, the community was very understanding, our sales had risen a bit, and the upgraded ratings meant that our overall score was higher than before the crisis. Needless to say, replying to all user messages individually, honestly, and in a timely fashion has become a core value of our team ever since :)

If you want to try out our games for yourselves, here are the links:

Le Havre: The Inland Port

iOS: https://itunes.apple.com/us/app/le-havre-the-inland-port/id1044671796?ls=1&mt=8
Android: https://play.google.com/store/apps/details?id=com.digidiced.lhbrelease
Windows Phone: http://windowsphone.com/s?appid=e2aebd5a-a174-44e0-9884-9589bdf65978

Patchwork

iOS: https://itunes.apple.com/us/app/patchwork-the-game/id1075851197?l=de&ls=1&mt=8
Android: https://play.google.com/store/apps/details?id=com.digidiced.pxwrelease
Windows Phone, Windows 8 & 10: https://www.microsoft.com/en-us/store/games/patchwork-the-game/9nblggh58k2z