Analysis: What VGChartz Does (And Doesn't) Do For The Game Biz

Recently, there's been a significantly greater profile for the video game chart compilation site, VGChartz. As well as beginning to contact major news sites on a regular basis to disseminate its news, the site was also the subject of a positive article on O'Reilly Radar from Robert Passarella, comparing its open data dissemination method favorably with The NPD Group, the generally agreed 'canonical' source for North American game charts. Indeed, as the Wikipedia page for VGChartz notes, Forbes, Fortune, The New York Post, and The New York Times have all referenced the site. And since it's been more aggressively marketed by the site's creators - especially regarding the 'holy grail' of global sales figure comparisons, its references in the news are rapidly increasing in frequency. But how is the site actually compiled, and is it a good source for reasonably reputable news websites such as Gamasutra to be citing? Thus far, we have referenced VGChartz data twice - once with regard to Xbox Live Arcade game sales, and more recently because Michael Pachter has started to cite the data in his NPD game sales previews. The second citation provoked a number of reader queries about the veracity of the data, so we embarked on some detailed analysis of VGChartz, and followed it up with a long series of emails with the site's creator, Brett Walton. How Charts Are Compiled So, let's start with the basics. Most worldwide game charts (of which NPD in North America, Media Create and Enterbrain in Japan, and Chart-Track in England are the most prominent) are compiled by extrapolation from sales figures provided by retailers. Thus, there's no third-party that uses regular access to publisher data on sales to make the charts - all of them take in data from major retailers, and then calculate sales from there. This service is sold to major publishers and financial firms for a monthly or yearly subscription, allowing publishers to see how well their competitors are doing. Limited amounts of the data is made available to the public, generally in terms of a top 10, 20, or 50 either weekly or monthly, depending on the territory. Obviously, if the pay service was sufficiently 'off' from the internal numbers the publisher was getting, they would not want to pay for it. But what VGChartz claims to do is a weekly estimate of every game published in every territory - including Japan, North America, and the particularly difficult to estimate Europe (there's no subscription-based pan-European chart right now, due to the fragmented nature of the market) - and offer all of that data for free. Which is extremely impressive, let's face it, because it enables them to provide real-time updated information for the entire market. But how accurate is it? If I was, say, writing a story for the New York Times, what proof do I have that the 'correct' numbers are displayed on the site? Obviously, as mentioned above, all sales figures are by necessity estimates, and that's the crux of the issue - we'll get back to that later. But I asked Brett Walton his methodology, and he gave me the following, quite impressive answer: "The methodology we use for all of our charts in all regions is the same and our data is arrived at by a combination of the following: - Sampled direct sell-through data - Industry knowledge and experience - applying past trends in terms of marketshares, regional breakdowns, casual vs hardcore and so on - plenty of statistical analysis, regression calculations, market projections - Contact with industry figures - buy-side analysts (such as Pachter / Divnich), sell-side analysts who work with us on specific products / projects, manufacturers who work with us to project sales of their key titles - Retail checks - we have a team who talk to stores and estimate shipment figures for low-stock and hard to find items which we struggle to track with our normal data samples. Exactly how we get from these various sources of data to final figures differs from game to game and console to console and our exact methodologies are confidential for obvious reasons." Essentially, Walton is saying that he uses a number of high quality factors to produce his estimates, but can't mention any of the retail sources, or companies that VGChartz works with. Well, fair enough. But did you realize that VGChartz estimates can retroactively change by 100% or more based on 'official' chart results? Iron Man & Retrofitting One of the most unexpected results in the recent NPD charts for May was the appearance of the poorly reviewed Iron Man game for PlayStation 2 in the Top 10 of the charts, with 130,000 copies sold. Thinking about it carefully, with the movie rocketing to unexpected success during the month, it would make sense that the game would sell well. But it's not the kind of game that you're likely to estimate in the Top 10 - and indeed VGChartz did not, estimating 53,000 units in sales, according to VGChartz staffers. But what's surprising is that Iron Man for PlayStation 2 has been adjusted in its official VGChartz page so that its first four weeks of sales (encompassing May) add up to 111,000 units. Clearly, these numbers have been changed after NPD debuted, showing a couple of things. Firstly, if you were a journalist, you could have cited VGChartz as saying Iron Man was a flop on PS2, selling half as many units - when NPD vibrantly disagrees. In addition, and more interestingly, it shows that VGChartz trusts NPD over their own prediction data by retroactively changing things to better match. Apparently, this has happened before, because in a FAQ about North American VGChartz numbers, Brett Walton addresses this precise subject: "Do we adjust our data? Not as such. Do we adjust our methods then? Yes - which will of course alter some data. On what basis? If we believe that a particular data set differs significantly from other sources of data (data released into the public domain by tracking firms, manufacturers, analysts) then we do re-check our data and make adjustments to the methods / scaling factors used. This happens on a fairly infrequent basis - less often than we adjust due to internal data changes - and is something that every tracking firm and analyst does. I personally have no issues with "benchmarking" our data from time to time against other sources of data - as long as it has been made public." In other words, if they are sufficiently out, then VGChartz will retrofit their results - either weekly or monthly - to conform to the more 'official' data. But they won't credit those firms as the source of the retrofitting - they'll just bump their numbers around without saying why on the site. As a result, we get to what VGChartz actually is - a strange mixture of a prediction market (as consensus prediction site TheSimExchange is) and a retroactive, but non-credited reflection of charts that have historically been known for having more concrete data. Where's The Beef? OK, so you might say - and a lot of VGChartz' forumgoers do - what's the problem with that? If VGChartz gets close enough, and can adjust if it's too far off when top-end data comes out, then why would there be a problem? Well, because you then have a moving target for checking/reporting purposes, and particularly because there's a high probability that VGChartz figures will be significantly wrong for those titles on the lower end of sales - those that lurk outside the top of the charts. In other words, for those high-selling titles, VGChartz is checking against public data, and they will change their estimates if they are majorly off. Most of the time, they are quite close compared to the worldwide charts. That's because VGChartz is - like services such as The SimExchange - using common sense, Internet buzz, real-time data such as Amazon.com and analyst commentary to synthesize a sensible estimate. But in covering all games, they are doing readers a disservice, because it's clear from the Iron Man example that they simply do not have the direct sale retail contacts to extrapolate unexpected but nonetheless true results. And if a title spikes but is outside public data, VGChartz will never catch it. And the amount of concrete data available to VGChartz is low - as is freely admitted in a recent interview, VGChartz had 2-3% of the North American market as a sample at the time, whereas by estimate, NPD might have 60-65%. If this 2-3% was clean and canonical, this might not matter - but how do you explain the big Iron Man discrepancy, if so? Wouldn't VGChartz' retail sources have picked it up too? So, let's take a step back and concentrate on some games that have sold in significant numbers, but have never made it into the Top 20 in North America for a significant time. One good example is the Ben 10 series of games from D3 Publisher. VGChartz has the series listed at 590,000 sold worldwide to date. But when Gamasutra interviewed D3's Yoji Takenaka last week, he specifically said: "Ben 10 is selling well over a million units right now, since last Christmas." So sure, Takenaka could be conflating shipped with sold - making the number closer to the estimate. But that's an awfully large discrepancy - one that most people won't care about because it's not a prominent or critically acclaimed game, and there's no way to refute VGChartz on it, but a discrepancy nonetheless. Unfortunately, we don't have lifetime NPD data for this set of titles - but in researching this story, we spoke to a third party who had access to NPD lifetime to date sales that are not normally disclosed to the public. We picked two titles released for one of the next-gen consoles over the previous year, neither of which had been in the public NPD charts for more than a month, leaving VGChartz to make estimates based on their own sources on their selling curve over time. Well, somewhat spectacularly, in both cases, NPD and VGChartz disagreed by about 100%. In one case, VGChartz was citing 300,000 sales, whereas NPD had the game at 150,000 units. And in the other case, it was inversed - NPD had the game at around 200,000, but VGChartz had it at 100,000. If VGChartz knew of this discrepancy, would they have retroactively changed their data? Probably so, given the Iron Man example. And this is essentially the problem - that with very limited access to retail numbers, especially over time, the downward curve of a game's sales becomes essentially a guessing game for VGChartz, whereas services like Media Create and NPD merge in greater real sales data to calculate their curve at much higher levels. [Here's one more public datapoint, this time referencing NPD, but uncorrected by VGChartz, since I presume they didn't notice it or consider it important enough. Variety recently revealed that Brash's Alvin & The Chipmunks game had sold 286,000 copies since launch, according to NPD. VGChartz has the combined SKUs listed at just 110,000 units.] Conclusion Let's be clear. I think the concept behind VGChartz is a wonderful one - freely available data to let everyone see how well games are selling. And it's absolutely true that all data is an estimate - not even major services such as Media Create and NPD get it exactly right. But VGChartz is staffed by amateurs working in their spare time to estimate sales, and while they are perfectly smart, they are much closer to the SimExchange model of estimation than the Media Create method. What I'd like to see is some clear labeling of what is estimated data, and what is extrapolated or changed from companies that have greater access to retail sales. And not only does VGChartz have no intention of doing this, it is starting to claim major scoops based on data which, in some cases, estimates entire territories without any real data. In particular, the site widely and loudly disseminated to the media its worldwide Day 1 Metal Gear Solid 4 sales, explaining: "VGChartz can exclusively reveal that first day sales of Metal Gear Solid 4, released on June 12th 2008 in most major markets worldwide, were an impressive 1.3 million units." The headline actually originally read 1.5 million, but was changed by a not insignificant 200,000 units after publication. Even more surprisingly, the figure debuted just 48 hours after the launch of the game - not a lot of time to compile data from retail sources. I asked Brett Walton about the change, and why this figure was not advertised a little more prominently as an estimate, given the short amount of time to get real data, and he explained: "It was based on first day Japan sales, first day America sales, and from that projecting for Europe / others which we didn't get direct day 1 for. We projected Europe would be ~20% higher than America given the larger install base and based on previous game releases, but it turned out at 430k for the week vs 510k for America - whereas we estimated it at more like 600k given America and Japan figures." Firstly, Walton freely admits the numbers were based on zero actual data for the entire European market, just pure extrapolation. It's also very unclear how far the estimates for launch were based on real retail data for Japan and North America. It's a reasonable figure, of course, because the VGChartz folks are smart people. But it's not a real figure. It's a educated guesstimate, and it's much more of an estimate than the subsequent Chart Track data for the UK, for example. Walton clarified due to my complaints: "So yes, maybe we should be clearer with the word estimate, especially in early PR and this has been reflected in comments back to the guy who wrote the story. From now on we will label day 1 sales as preliminary for that very reason." But that doesn't really change the main problem with the site. There's a place for a resource like VGChartz, but it'd be a site that clearly labels the source of its estimates (whether it be Chart-Track, NPD, Media Create - even if some of those sources have poor data dissemination and a fractious relationship with the media) and then labels which are its own estimates based on its own industry knowledge and whatever channel checks it has. But if I was a writer or analyst trying to extrapolate significant information from the resource, especially regarding those titles which don't chart regularly, given the major discrepancies with other figures shown here, I would not recommend it.