informa
/
Production
Featured Blog

The New ESRB Rating Algorithm

Understanding the pros and cons of the new "automated" ESRB rating system.

The news media or whatever reporting on the ESRB's new "automated"/"machine" rating system is  confusing and probably inaccurate.  But I don't think that it is really their fault.  The press release by the ESRB is a bit cloudy, hyped, and open to mis-interpretation.

Having now done several years of empirical research on violence in video games, I think that I should comment on the new and old ESRB rating system as I understand it.  You can read about one of our studies in the Laboratory for Automation Psychology and Decision Processes.

The new process involves an online questionnaire developed by the ESRB. At the time of this writing, I have not seen the questionnaire, so I don't how long it is, what the questions are, the options, etc. 

I can only speculate about its contents and hope that the ESRB hired some really good professionals develop it using psychometric methods.  Whatever, some of the structural properties of the questionnaire can be inferred.  I should also note that I have more than 10 years of research experience with online questionnaires, 30 years with psychometric methods and statistics.

So as I understand it, the game developer assigns one of it's employees or perhaps a team familiar with the new game to fill out the ESRB questionnaire.  The answers to the questionnaire are run through a program an aggregates them and spits out a categorical rating (e.g., "E", "T", "M") and assignee descriptors (e.g., "Mild Cartoon Violence", "Strong Language", etc.). 

To know how this works, you've got to know the algorithm (formula) that they have developed and will be using. The formula, hopefully, was the result of analyzing a large sample of games with given ESRB ratings. They would have had individuals familiar with the games answer the questionnaire that they developed. Then they used a statistical clustering analysis, such as Discriminant Analysis, to approximate the best fitting formula.
 
Now here are the problems: (1) The ratings will be no accurate than previous ratings since the previous ratings were used to derive the formula. (2) Game developers can easily outwit or circumvent the system by figuring out how to fill out the questionnaire to get the rating that they want.

Ultimately, the ESRB has pretty much handed over the job of rating games to the developers and abdicated their responsibility in the name of efficiency and cost savings. But what else can they do with the number of games that are being produced every year.

The ESRB promises to check up on the ratings by having a human evaluate the game after the rating has been posted and the game released.  The result is that the backlog in the human rating process will no longer delay the game rating and hold up the release of the game.  Rather it will delay any corrective action.  Below is my diagram of the old and new timeline as I understand it.

Timeline of ESRB Process

However, I believe that this is, or could be, an inherently a better system.  The game developers know their game better than anyone.  In the old system anyway, they were the one's who generated the DVD of incidents critical to the rating (e.g., language, violence, sex). 

The reason that I note that it could even better is because the new system has some potential tweaks that could improve it's accuracy and reliability.What might be added is that in the development of a game, literally thousands of hours can go into the play testing of the game.

 Developers will probably want to sample these testers for the answers to the ESRB's questionnaire. Thus, they can access the wisdom of the crowd rather than one person filling out the questionnaire. But there can be is a problem.  There is a strong correlation between type of gamer and acceptance of violence in a game. Heavy gamers tend to rate more violent acts as acceptable than do soccer moms and senators.  So it is crucial to have a representative range of people filling out the questionnaire.

A second advantage of the new system is that after releasing the game, feedback from the community can be used to refine the algorithm.  Essentially it can shift over time to accommodate the changing moral code of the society and perhaps even for specific societies and regions.  Thus, if different states were to enact different rules for rating video games, the ESRB could do it.

So we shall see just how this rating game plays out!

Latest Jobs

Sucker Punch Productions

Bellevue, Washington
08.27.21
Combat Designer

Xbox Graphics

Redmond, Washington
08.27.21
Senior Software Engineer: GPU Compilers

Insomniac Games

Burbank, California
08.27.21
Systems Designer

Deep Silver Volition

Champaign, Illinois
08.27.21
Senior Environment Artist
More Jobs   

CONNECT WITH US

Register for a
Subscribe to
Follow us

Game Developer Account

Game Developer Newsletter

@gamedevdotcom

Register for a

Game Developer Account

Gain full access to resources (events, white paper, webinars, reports, etc)
Single sign-on to all Informa products

Register
Subscribe to

Game Developer Newsletter

Get daily Game Developer top stories every morning straight into your inbox

Subscribe
Follow us

@gamedevdotcom

Follow us @gamedevdotcom to stay up-to-date with the latest news & insider information about events & more