Valve wasn't 'really sure what to do' following Assassin's Creed Unity review surge

Earlier this year, Valve updated Steam’s review policy to account for surges in off-topic reviews, or the "review bombs" that Steam users often lob at a game to tank its score as an act of protest for something unrelated to the game itself.

The system first took effect when a wave of negative reviews hit Borderlands games after it was announced that Borderlands 3 would be an Epic Games Store exclusive, but did not trigger when a large volume of positive reviews hit Assassin’s Creed Unity weeks later to praise the game’s recreation of the Notre Dame cathedral.

Now, Valve has published a blog post that explains why its systems didn’t classify that behavior as “off-topic reviews," both based on data and the larger context surrounding the situation

It’s a bit of a late-breaking verdict since it’s been about a month since those positive review bombs first hit, but Valve’s post addresses some of the complexities surrounding ‘review bombs’ and its previous rulings about the practice.

Ubisoft opted to give Assassin’s Creed Unity away for free on its Uplay store following the Notre Dame fire as a way to give players a way to explore Ubisoft’s in-game recreation of the cathedral, and donated $500,000 to the restoration effort as well. Valve says that the promotion ultimately led to more players buying and playing the game on Steam as well as the positive reviews began to flow in.

Looking at just the data-side of the reviews, Valve says that the numbers didn’t have the same pattern as the negative, off-topic review bombs its new review system is set up to address. Its data showed a jump in players alongside an increase in reviews, making it look like a game that has seen a sale or update, without considering the added context of the Notre Dame fire.

In the case of past off-topic decisions, Valve says the context behind the review bombs were “largely divorced from the game itself,” something that it says isn’t entirely true for Unity’s review surge. The post notes that not all reactionary reviews are considered off-topic: “for instance, news that the live team of a multiplayer online game has been laid off is a context change that seems useful to have reflected in the review score for prospective buyers.”

“In this case, the Notre Dame tragedy has made it so that [Assassin's Creed Unity] happens to now include the world's best virtual recreation of the undamaged monument,” reads the post. “That's a context change that could be increasing the value players are getting from the game, so perhaps the game really is better than it was before? Or maybe that's unrelated, and it's actually players feeling good about Ubisoft's significant donation to rebuilding the monument?”

In short, Valve admits “we’re not really sure what to do here” but opted to leave the scores for Assassin’s Creed Unity as they were (though it notes that removing those recent reviews would only drop Unity’s score by 1.3 percent.) The company’s full musings on the topic, including a brief FAQ on the decision, can be found in on the Steam Blog.