In-depth: Leaderboard benchmarking

[In this reprinted #altdevblogaday in-depth piece, Agora Games's David Czarnecki demonstrates how tweaking basic "knobs" can bring about significant speed improvements in your leaderboards.] I recently got a new MacBook Pro, and I wanted to do some benchmarking. Naturally I turned to the leaderboard library I maintain, which already had a base set of benchmarks that I could compare the results to on my new computer. You may also recall I've written about leaderboards before on #AltDevBlogADay. In this blog post I'll discuss two "knobs" I tweaked to get more performance out of the leaderboard library, the underlying interpreter and an underlying client library. While there was definitely a speed improvement when switching the underlying interpreter to a more recent version, there was a more significant speed improvement when switching the underlying client library. Tweaking Knobs The benchmarks were run on a 2.5 GHz Intel Core i7 MacBook Pro with 8 GB of memory under OS X Lion 10.7.2 using Redis 2.4.5. They measured the total amount of time to rank 10 million people in a leaderboard using sequential scores, total amount of time to rank 10 million people in a leaderboard using random scores and the average time to retrieve an arbitrary page from the leaderboard. The basic benchmarking code can be found in the README on the leaderboard project page. Switching Ruby Interpreters Below are the benchmark results from switching between different Ruby interpreters, Ruby 1.8.7, Ruby 1.9.2 and Ruby 1.9.3. Values are in seconds. Ruby 1.8.7

Time to rank 10 million people in a leaderboard (sequential scores): 794.157420158386

Time to rank 10 million people in a leaderboard (random scores): 849.301838159561

Average time to retrieve an arbitrary page from the leaderboard (50,000 requests): 0.00165219999999999

Ruby 1.9.2

Time to rank 10 million people in a leaderboard (sequential scores): 656.387526

Time to rank 10 million people in a leaderboard (random scores): 748.955826

Average time to retrieve an arbitrary page from the leaderboard (50,000 requests): 0.0011321999999999895

Ruby 1.9.3

Time to rank 10 million people in a leaderboard (sequential scores): 651.057383

Time to rank 10 million people in a leaderboard (random scores): 719.157958

Average time to retrieve an arbitrary page from the leaderboard (50,000 requests): 0.001079199999999996

Summary

Ruby 1.8.7 to Ruby 1.9.3 (sequential scores): 18% improvement

Ruby 1.8.7 to Ruby 1.9.3 (random scores): 15% improvement

Ruby 1.8.7 to Ruby 1.9.3 (average page request): 34% improvement

Switching Redis Client Library I decided not to benchmark Ruby 1.8.7 or Ruby 1.9.2 in tweaking the client driver "knob". Below are the benchmark results using Ruby 1.9.3 with the hiredis-rb client library. Values are in seconds. Ruby 1.9.3 and hiredis

Time to rank 10 million people in a leaderboard (sequential scores): 472.544572

Time to rank 10 million people in a leaderboard (random scores): 549.911350

Average time to retrieve an arbitrary page from the leaderboard (50,000 requests): 0.0003803999999999928

Summary

Ruby 1.8.7 to Ruby 1.9.3/hiredis (sequential scores): 40% improvement

Ruby 1.8.7 to Ruby 1.9.3/hiredis (random scores): 35% improvement

Ruby 1.8.7 to Ruby 1.9.3/hiredis (average page request): 76% improvement

Ruby 1.9.3 to Ruby 1.9.3/hiredis (sequential scores): 27% improvement

Ruby 1.9.3 to Ruby 1.9.3/hiredis (random scores): 23% improvement

Ruby 1.9.3 to Ruby 1.9.3/hiredis (average page request): 64% improvement

Wrap-Up It seems every aspect of video game technology is tweaked to maximize performance, whether it be frames per second or requests per second. Fiddling with very basic "knobs" here, I was able to get decent improvements in writing to and reading from a leaderboard of non-trivial size. Further speed improvements could potentially be realized by using a socket-based connection versus a TCP-based connection as well as faster hardware optimized for a server environment versus an off-the-shelf laptop. Thanks for indulging my leaderboard nerdery yet again. You can find more hilarity over on my Twitter account, @CzarneckiD. [This piece was reprinted from #AltDevBlogADay, a shared blog initiative started by @mike_acton devoted to giving game developers of all disciplines a place to motivate each other to write regularly about their personal game development passions.]