informa
5 MIN READ
Featured Blog

Scaling Dedicated Game Servers with Kubernetes: Part 2 – Managing CPU and Memory

Second part of a series on managing and scaling Dedicated Game Servers with open source projects Kubernetes and Docker. In this part we cover how you can use these tools to track, manage and limit the CPU and Memory utilisation of your game server.

Originally posted on compoundtheory.com.

This is part two of a fivefour-part series on scaling game servers with Kubernetes.

In the part 1 of this series, we looked at how we could take our dedicated game server, containerise it with Docker, and then host and manage it on Kubernetes, which was an excellent start. However since our Kubernetes clusters are generally of a fixed size we could potentially run out of available capacity to run all the game server containers we need to match all the players who want to play our game - and that would be a very bad thing.

There are lots of options for scaling up and down Kubernetes clusters, and we will go deeper into a custom Kubernetes node scaler in future posts. First we have to work out one very important thing: How much CPU and memory does my game server actually take up?

Without this knowledge, there is no way we can match up the CPU and/or memory utilisation of our game server with the available resources in our Kubernetes cluster, and therefore know how many game servers can be run within a cluster of a given size. We also won’t be able to calculate how many nodes to add when the number of players in our game increases (as it naturally will 😉) and we want to make sure the cluster is large enough to accommodate all of them. We will also want to scale down our cluster if traffic starts to drop to take advantage of cost savings by removing cluster nodes that aren’t being used.

Kubernetes Dashboard

The great thing is, visualisation of CPU and memory usage is another feature that Kubernetes provides out of the box. Kubernetes is bundled with a dashboard that gives us a nice visual representation of what is happening inside our cluster - such as listing Pods and Services, providing us with graphs of CPU, memory usage, and more.

(The Kubernetes dashboard!)

To securely connect to the Kubernetes dashboard, Kubernetes’ command line tool has the command:

kubectl proxy

Which when run, will return with:

Starting to serve on 127.0.0.1:8001

This creates a proxied connection to the Kubernetes master, (assuming you have permission to access the cluster) which hosts the Kubernetes cluster’s API and the dashboard that is displayed above.

If we now point our browser to http://localhost:8001/ui, we will see the dashboard.

Determine CPU and Memory Usage

You may have noticed that the dashboard gives us aggregate statistics of CPU and memory across the cluster, but it can also provide us with the same information at the Pod level as well! Therefore, all we need to do to determine how much CPU and memory our game server is using, is deploy a Pod containing our game server (which we set up in the previous post), do some load tests by running multiple game sessions on it, and have a look at the provided graphs.

I did this for the Paddle Soccer game, and you can see one of the results below:

In the test above, this simple dedicated game server peaks its usage at around 0.08 CPU cores and just over 34 Megabytes memory. This was the maximum usage we saw when doing load tests on the dedicated game server, so we’ll draw a line in the sand here and say that this is the upper limit of what our server utilises, add some buffer, and plan accordingly.

Restricting CPU and Memory Usage

One of the very useful capabilities of software containers such as Docker, is that it has the ability to put constraints on both CPU and memory usage of the running container, and therefore the processes within it.  Kubernetes exposes this to us through its Pod configuration, which means that we can explicitly ensure that CPU and memory usage won’t go over a certain threshold and adversely impact other game servers running on the same node. Given how sensitive game servers can be to CPU fluctuations, this is a very useful thing!

Therefore, if we want to limit the CPU usage of the Pod that contains our game server, we can do so by updating our previous Pod definition:

apiVersion: v1
kind: Pod
metadata:
  generateName: "game-"
spec:
  hostNetwork: true
  restartPolicy: Never
  containers:
    - name: soccer-server
      image: gcr.io/soccer/soccer-server:0.1
      env:
        - name: SESSION_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
	resources: # This section limits the cpu usage
	  limits:
	    cpu: "0.1"

It’s worth noting that in my sample code, I use the Kubernetes API to provide a configuration identical to the one above, but the yaml version is easier to understand, and it is the format we’ve been using throughout this series.

In the above example we have set a limit of “0.1”, which ensures that our game server only ever has access to one tenth of a CPU on the node that it is running on. We did this by adding the resources section with a corresponding limits and cpu section in our yaml for the game server container definition.

I chose to set a maximum CPU usage of 0.1 to give some padding to the game server 0.08 core usage result we saw above, while still letting me fit 10 game servers per core on each Kubernetes cluster node, which should scale well for our needs.

We could also do similar limits with memory usage, but for the sake of simplicity we’re going to just be limiting CPU usage, and ultimately using just CPU for our scaling metrics as well. If you want to learn more, the “Managing Compute Resources for Containers” documentation is also an excellent guide on these Kubernetes features.

Next Steps

Now that we have a known CPU usage and limits on our game servers, we will combine this in our next posts with the capacity of nodes that Kubernetes API reports, to enable scaling up and scaling down the cluster size in relation to the number of game servers requested on the cluster.

In the meantime, as with the previous post - I welcome questions and comments here, or reach out to me via Twitter. You can see my presentation at GDC this year on this topic as well as check out the code in GitHub, which is still being actively worked on!

All posts in this series:

  1. Containerising and Deploying
  2. Managing CPU and Memory
  3. Scaling Up Nodes
  4. Scaling Down Nodes
  5. Running Globally

Latest Jobs

Xbox Game Studios

Redmond, Washington
10.5.22
Technical Lighting Artist

Innogames

Hamburg, Germany
10.5.22
Game Designer - Elvenar

Six Foot

Houston, TX
10.3.22
Six Foot Director, Player Relations

Hometopia Inc.

Remote
10.7.22
Lead Engineer
More Jobs   

CONNECT WITH US

Explore the
Subscribe to
Follow us

Game Developer Job Board

Game Developer Newsletter

@gamedevdotcom

Explore the

Game Developer Job Board

Browse open positions across the game industry or recruit new talent for your studio

Browse
Subscribe to

Game Developer Newsletter

Get daily Game Developer top stories every morning straight into your inbox

Subscribe
Follow us

@gamedevdotcom

Follow us @gamedevdotcom to stay up-to-date with the latest news & insider information about events & more