To enable Contraption Maker to run without requiring Steam, Spotkin developed a server using Google App Engine. This allowed us to build a scalable backend that handles authentication, workshop uploads, and build distributions.
Here is what the workshop looks like in the game. It allows for logged-in users to upload and download levels with a screenshot and the level, just like in the Steam Workshop.
As the game user base grows, we want to track our costs and see where we should be looking for optimizations as we use the various services that App Engine provides. Using a Google managed virtual machine, this is actually pretty easy. I will describe all the steps in detail, but here is the end result: a chart showing the cost of each service in App Engine that we use on a daily basis. As you can see in the graph, right now it is a little under two dollars a day.
The first step is to set up a Google Storage Bucket for the billing data. You do this in the Google Cloud Console. I called our bucket "sws-billing".
Now go into the Billing settings for the billing account you want to monitor. Click the export tab and enable export of the billing data, selecting the bucket you previously created. I also selected to export the data as JSON to make it easy to parse.
Now for the final infrastructure step. Create a Google VM Instance. I chose Ubuntu as the operating system and gave it a static IP. The most important step for me was to make sure to enable both read and write access to Google Storage. My plan was to move each processed JSON file into a "processed" directory to make sure I never processed the same file more than once. Originally I created the VM with read-only access to Google Storage. You cannot modify these permissions once you have created the VM. I had to delete that VM, preserve the disk, and then create a new VM using the same disk. There are certainly ways to avoid needing write access to the bucket - you could keep track of the JSON files you have processed in a non-cloud storage directory on the VM, for example.
I chose InfluxDB + Grafana as the way to store and graph the data. There are many other options you might want to consider (RRD and Graphite are two others I looked at). I won't cover the installation of these tools, but it was very straightforward. Just follow the instructions at the corresponding sites: InfluxDB:
Just click the ssh button and the console launches:
What is really nice about using a Google Managed VM is that you automatically have access to the Google services you need (in this case, Google storage). So right away, I can list the billing reports in the bucket with this simple command:
gsutil ls gs://sws-billing
And copying those files to a local directory is simply this command:
gsutil -m cp gs://sws-billing/*.json ./sws-billing
Now I had all the pieces necessary to start graphing the billing data. We use Rake / Ruby for a lot of our build processes, so I decided to just write a simple Rake task that could run every night in a cron job to parse the billing data. The steps would be:
- Remove the local copies of the billing files
- Copy any billing files in sws-billing to a local directory
- Parse the billing file and send data points to influxdb
- Move the json files in sws-billing to sws-billing/processed
Here is the complete Rakefile
And here is the crontab entry
It took me a few tries to get the graph to look right. I used the project id in the appengine billing data to separate out the cost on a project basis. We have a separate project for our development server, for example. Here are the configuration settings I used.
We have found Google App Engine to be an incredible resource for our server infrastructure. I hope to talk more about it in a future blog post. In the meantime, if you are thinking of using Google App Engine for your project or already using it, I hope you find this blog post useful!