Two guys made an MMO: The Growtopia Postmortem

So, being the master of game marketing that I am, I've carefully been totally silent here on my blog about my latest game, despite being released five months ago.

The game now regularly hits 3,000 players online at once and has been financially a huge success.  (At least by my humble standards!)  It's been a top 1000 grossing US App Store game for the past four months and doing similarly well on Android - all without being featured and zero paid advertising.

11-9-2014 Update:  We've grown!  We now regularly hit 40,000+ players online, have 5+ million users and around 300K daily active users.  80+ million worlds have been created by users.

7-18-2019 Update: The game continued to grow, it actually got a bit too big for us to handle.  We sold it to Ubisoft on Feb 17th, 2017.  At that time we were at over 500 million user created worlds and were regularly hitting 70K+ concurrent online users.  Today Ubisoft has released the game on consoles (Xbox, PS4 & Switch) and I couldn't be happier.  While we were not involved with the ports I still feel like that's our baby growing up!

(back to the 2013 blogpost)

Today I'm finally sitting down to document how this project took shape as well as the trials and tribulations.

September 8th, 2012: Six pictures

After my highly competitive multiplayer online game Tanked was more or less finished,  I began thinking about a new game.  Something that would capitalize on the networking client/server experience I'd gained but apply it to a fresh idea with simpler controls that would be more accessible.

Despite Tanked being 3D, I decided to do the game in 2D - faster development and we could target even very crappy mobile devices.

With Tanked, I felt my fatigue in "doing it all, alone" was the cause of my non-existent marketing efforts (I've STILL never done a youtube video for Tanked?  REALLY?!?!), as well as reluctance to add the final layers of polish.  I needed a partner, someone to provide a mutual motivation as well as handle the artistic needs of the project.

So with that in mind, I presented six mock-up screenshots to long-time IRC pal and fellow indie, Mike Hommel, aka Hamumu to entice him to come aboard.

Here are the original shots (created using stolen sprite rips mostly) for your viewing displeasure (I'm no artist!):

We changed and improved the ideas represented in these mockups over time, but stayed focused on the general goals.

He agreed to give it a shot.  Like so many indie collaborations, I suspect he was worried we'd never reach beta. Hell, me too!

I think I predicted 4-6 months to finish it.  This is usually where I would chuckle at my foolish naivety and admit it took years longer than expected but... not this time?!

The reason things went so smoothly?  Well, I guess probably because we had very few "unknowns".

In the past, we'd both already made 2d tile based platformers, collision code, networked games, mysql based projects and websites, and done cross platform development.  Using my Proton SDK insured we could run on eight platforms (nearly) out of the box.

We figured the only real unknown was what our max player limit would be and how things would scale up.  I had guessed conservatively, 600 players online would be our max.  Luckily, I was wrong, the original server actually was able to handle much more.

September 15th, 2012: Networking and collision are functional

The above shot shows four instances running around in the same world.  All graphics are just DrawRects for now.  The collision is incredibly simple and as a result gives us very few problems later as we tweak it for special tiles and items later.  The game is written in C++ with GL/GLES.

I think one reason this project went so fast is I was able to fight the urge to go overboard on everything like I usually do.  There are no fancy 2d physics.  I mean hey, we're not making Limbo here. It would be overkill.  I think it helps that I had already done the box2d physics platformer thing in a previous project so I didn't really feel like I had something to prove, if that makes sense.

I've never actually met Mike in person, he's in California and I'm in Japan.  But if anything, that probably helped - we work on the code base (he programs too!) in (roughly) day/night shifts which help avoid svn code conflicts.

We use IRC and Google Docs to communicate - spreadsheets for tracking hours and financial stuff, and the drawing board to share screenshots and ideas.   The best part is we have a full history of everything we've done, that's where I'm getting most of the screenshots for this article!

September 21st, 2012: GUI designs

The GUI starts to take shape via Hamumu's mockups.

October 18th, 2012: Doors, signs, chat, inventory, and one kind of lock working

In roughly five weeks, the game looks pretty similar to its current form.

November 30th, 2012:  The game is stealth-released on Android as a free beta after three months of development

We were unable to use the original name we wanted (Buildo) as someone else was sort of already using it.  After a VERY painful few days we finally agree on Growtopia.  I really couldn't get anything done during that period, strange that stress of finding a game name could grind everything to a halt like that.  I was never totally satisfied with the name, but it's very searchable so that's a perk.  The domain name was taken so we had to live with growtopiagame.com.

IAP is disabled initially as we expected to reset the universe a few times. (We ended up never having to do that)

There were no "World Locks" yet, so people tended to clump together in little towns, locking their areas with the smaller lock types.

Despite the total stealth release (I didn't even mention the beta on my website) the player base quickly grows.  It's the perfect design for PR slackers like us,  very viral in nature due to the social aspects.

January 9th, 2013: The game is released on iOS, and the word beta is removed from the Google Play description

The release!  We get a boost from Touch Arcade, Pocket Gamer and IndieGames who all post our teaser video.  (we sent it to them and a few other sites, that was pretty much it for our marketing campaign)

We stress out as we watch the users online grow.  200 users, 300 users.. 600 users online at once!  Will the server die?  We upgrade our server three times in two weeks to handle the increase, wondering if after the initial surge we'll need to go back to the cheaper one.  Servint, our data center, did a great job of quickly migrating us from server to server as needed.

Life after release, where we are now

Like a newborn, we found the game screaming for our non-stop attention.  We find ourselves constantly putting out fires and dealing with issues. 600 hundred forum posts a day to read, hundreds of daily support emails.

There is no finish line, there is no "done" - we've basically worked full time on the game since its release.

Our game has an extremely sensitive virtual economy that could be decimated in only hours by a rampant bug.

A single server crash can cause an hour of lost work per player online.  A single day roll-back on a busy day now would mean instantly vaporizing SEVEN HUMAN YEARS worth of effort.

These things weighed very heavily on me in the beginning.  I had trouble sleeping and would check the server throughout the night to make sure it was still running correctly.  After a while though, you sort of reach a certain level of numbness/comfort with it all.

I guess I'll forgo the usual "what went right/wrong" and just illuminate the most important/damaging events and how we dealt with them.

Our first penis

Well, we knew it had to happen.  I appeared in their world and gave them five minutes to remove it, but secretly I was sort of proud, it's sort of a developer achievement to have someone miss-use your creative tools like this.

As our player count grew, we found our player discipline tools inadequate.  Over time we beefed them up.

The problem with tape

Instead of simply muting people, we thought it would be much cooler to visually duct-tape their mouth shut.  When they attempt to speak, only muffled noises come out.

The end result wasn't what we expected.  Being duct-taped quickly became the goal. When players saw us, they'd mob us yelling obscenities, hoping to be taped.  We had to make a new rule: "If you try to get taped on purpose, we'll just ban you" to keep it under control.

The item duping

Due to a bug a patch introduced, it became possible to log on with two instances of the same player for a short while if you did it at EXACTLY the perfect time.  Within hours a group of Asian players had figured it out and were duping like mad.  I remember using google translate on Growtopia related Korean message boards trying to figure out the method used.

Knowing it was happening, but not understanding the modus operandi was VERY stressful, we had to spy and watch them doing it.  We figured it out and patched the server.  We hand deleted the ill-gotten gains as best we could (we didn't want to roll back!), but it quickly becomes impossible to track as items are traded.  For a while, we had the server report anyone with > 100 world locks and we'd just assume they were holding duped items and punish the account.  Nowadays that wouldn't work because many legitimate players have earned that much wealth through normal gameplay.

The bedrock bug

One day fairly early in Growtopia's lifetime (I think it was still in the wide beta on android) we made a horrible mistake with user security levels which let anybody destroy bedrock and white doors.  (Normally a player can't destroy these tiles, as means player can fall out of the world at the bottom, and can't enter worlds the normal way)

Twenty minutes had passed before I'd noticed the frenzied broadcasts that excitedly shared the bug - not only were people destroying normally impenetrable blocks, those blocks were giving seeds, which would grow into trees that would eventually yield more of the forbidden blocks, letting the user place them!

This bug really didn't cause too much damage, but we were running around fixing holes in levels and replacing missing entrance doors for weeks.  We also added a filter that would remove those items from peoples inventories.. but I wouldn't be surprised if there is still a bedrock tree growing somewhere out there and you do still see the occasional bedrock piece missing from older worlds.

The IAP hackers - "Someone is giving away world locks"

Like most developers,  I've grown used to the idea knowing that people are pirating my games.  I don't stress too much, it doesn't ruin the experience of others for the most part.

But if thousands of dollars worth of gems are stolen via fraudulent means, it has a real impact on us as it is pumped into the game economy.

We'd watch would-be thieves dump their contraband in remote locations.  We'd wait (invisibly) and see who came to pick it up, track down the persons main hideout and ban them all.  Eventually we upgraded our server to do full IAP receipt checking, but life had a certain cops 'n robbers taste to it for a while.

We still have issues with stolen credit cards and chargebacks, but it's now at manageable levels for the most part.

The Tapjoy hack

Tapjoy is a great way to let players watch a 20 second commercial, to buy gems using their time, and only if they choose to.  (They get 90 gems, we get 5 cents, or whatever)  However, due to me using it wrong (similar to IAP, I was using my code from previous single player games I worked on rather than custom stuff with full server verification) it was very vulnerable to client hacks, as illustrated in the image.

I wasted a lot of time writing code to detect hacks and such, when I should have just re-implemented Tapjoy the correct way, which I eventually did anyway.  It now runs flawlessly.

Scamtopia

The virtual economy made item trading and bartering very important.  The original release had no secure way to trade, only "drop trading".  Naturally scamming was prevalent. We told people "Never drop trade with people you don't know!"

We introduced a robust trading system as quick as we could.

The blueberry hack

Due to a server vulnerability, it was possible to hack a client to give you unlimited blueberries.  (Basically, when using it to eat, a special case happened where it didn't check if you actually had any left)

We were perplexed why a certain group of players had so many blueberries, it didn't make sense... someone eventually emailed us and explained the trick and we quickly patched it.  Thanks, sir.

The wall hack

It turns out the first thing that people do in a game like this is use a memory scanning utility (a "trainer") to locate important memory addresses (speed, X/Y position, etc) and modify them.  We realized wall hacking would be possible, but didn't think it was a big deal due to our lock/ownership systems. (If someone gets into your locked area, it doesn't really matter, the server wouldn't let them take anything)

However, players liked to drop items around their worlds, which presented juicy targets for wall hackers.   We quickly combated this with clever server + client hacks which worked pretty well, but in the end we removed all doubt by adding full A* path finding on every single player movement.  (this wasn't possible for speed reasons until the V2 server upgrade.. more on that later)

The black day - the rollback

But it wasn't the hackers or credit card fraud that did the worst damage.  It was us!

On February 23th our worst nightmare became a reality.  We'd inadvertently made some changes that caused certain very cheap items to give out a high number of gems when "recycled", this created an infinite money loop.   As soon as it was noticed, someone "broadcasted" it which meant every single person online knew how to earn unlimited gems.   The entire economy was trashed in only minutes.

We had to roll back almost 24 hours worth.  (We do automated backups daily.. and sometimes I do extra backups before a serious patch, but today I didn't for some reason)

That weekend was dubbed "Apology Weekend" and we worked quickly to add perks and gems bonuses to try to make it up to the players.  If you ever see a "Rollback Plaque", it was earned by someone on that weekend.  We wrote a program to restore all gem purchases that had been wiped, with double the gems.

This was a wake-up call to more seriously test things before patching.  It's tough trying to surprise your audience with something new (we're very secretive) and yet wanting to properly test it.  We now do have a fully functional beta test server that we can route normal players into.

That day was easily the absolute worst for me personally.  I'm  happy to say we haven't had another roll-back yet.

Too many users

In mid April we hit 2,000 concurrent users on weekends and our server began to buckle.  Round trip to punch something could take a full second and people were constantly being disconnected.  We added live profiling support and narrowed down the slow down to world creation and enet packet sorting.

We decided I would write a "V2 server" upgrade in a separate svn branch that would take advantage of multiple cores;  it made no sense that our hardware had 16 cores and 32 threads but our entire Growtopia server process was run in a single thread.

Meanwhile, Mike would handle running active game and adding items and preparing to move them over to the new server.

We had a lot riding on the V2 server, would it really solve our problems or fail miserably?  It's very difficult to fake 2,000 users, so we released a client upgrade that could smoothly switch between the old server and the new beta one.

When things looked good, we just pushed a button and switched the real database over to the new server, if it died, we could just switch them back.  Anyway, it worked beautifully and we haven't outgrown it yet.

Here's how I explained it to users on our Facebook page: (it was actually only a software change, we didn't change hardware.  But this was the result as far as latency and gameplay)

The freemium dilemma

After railing against slimy abuses of the freemium model myself, I vowed to do it 'right'.  Here is what we did:

A few times we saw someone making daily purchases that seemed strangely high and emailed their Google purchasing account email directly to verify that everything was ok. (We thought it might be a kid using a parents' device)  I don't think we ever got a reply.  I think Apple and Google should be doing the emailing really, they have more and better information than we do.  We will disable IAP on any account by request. (the option won't even show up in the store in that case)

I think overall we "did it fairly right", but I still have moral reservations about this model and know that we're exploiting certain kinds of vulnerable individuals due to the nature of open-ended purchases, even if you don't need them.  In a single player game I don't think it would ever be justified to extract $100 in IAP, but in an MMO where we have players with 1,000+ hours logged and who purchase presents for others, I think it isn't totally unreasonable for an informed adult to spend that much.

But what is the line?

This is game where a kid can do a /broadcast and instantly be visited by fifty or more REAL people hanging on his every word because a prize is going to be given away.  He or she can run any kind of game within Growtopia and have complete control over these guests, ordering them around and banning at will.   I'm proud we created mechanics that allow something this amazing but also humbled by the damage this power could do to someone vulnerable.  My own son (he's nine) has spent $40 of his own money in Growtopia over the 200+ hours he's played.  He's given most of it away to his "friends".

If he's learned a real friend doesn't require a World Lock,  maybe it was a worthy investment.

Giving away too much

A problem with our single currency is you can make around $3 of IAP per hour farming in the game.  Our richest player has never spent a dime, and has $1000 usd worth of gems.  I'm sort of proud of that on one hand, it illustrates our game is REALLY free and not a trick.  But on the other hand, I don't know what he should spend his 1,000,000+ gems on.  If we add a 100,000 gem "Vanity item" in the store, eight year olds are also going to be pressured to use IAP to buy that essentially worthless item to keep up with the Jones'.  (Worthless as far as functionality, but .. well, the value as trading power due to the rarity and cost is very real in a multiplayer society like this one)

It will be a challenge to figure all this out.

Email support system woes

Originally we handled support emails by replying and bcc'ing eachother.  This .. doesn't scale up very well.  Eventually we setup with desk.com and now we can give much better service.  When a customer forgets to tell us his GrowID, we can just look through his history and grab it.

The future

It's not yet written, John!

All I know for sure is I chose the perfect person to collaborate with and our combined efforts have definitely resulted in superpowers.  Llike when spandex wearing heroes stand close together they can shoot larger balls of lightning.  It's like that but totally different.

In the five months since Growtopia's release we've had 3.5 million+ worlds created by nearly 400,000 users.  Last Saturday we set the record with 65,000+ hours logged in a single day.

I don't see any shortage of ways to improve the heart of Growtopia, which is about giving useful tools to the player to let him or her be creative with both strangers and friends; but each addition and tweak is now taking increasingly long to test and safely deploy due to the size and scope of the existing user base.  We can't make any mistakes now, especially with huge additions going in this weekend.  No rollbacks.

-Seth

Growtopia was created by Seth A. Robinson and Mike Hommel.