The power of review scores: how critics have more control over sales than we think
EEDAR is a video game research and consulting group that looks at a crazy amount of data in video games. They employ researchers who play games and take notes of many details: is the game third or first-person? How many weapons does it have? What kind of multiplayer? An incredible amount of quantitative data, including partner data such as sales information, market spend, and consumer awareness, is recorded for each game. Game publishers pay for access to this information in order to gain data about what features impact sales, or to look at what is being offered in competing games when planning a new project.
For better or worse, EEDAR is changing the way games are planned and designed with this data set. I sat down with Geoffrey Zatkin, the Chief Operating Officer of EEDAR, to use his data to learn about something we’ve all wondered about: Just how powerful are game reviews? After running the numbers the answer was surprising, and the data to back it up is powerful.
Game reviews can make or break a game.
The power of data
“If you say here are five ideas for games, which two do you think will earn the most money, that’s actually an easy question for us. With that we help companies lose less money,” Zatkin told me. Companies paying for access to the database are able to compare their upcoming projects with games that have already hit the market and see what works, what doesn’t, what helps sales, and what wastes money. If you’d like to compare the sales of games that let you customize guns vs. games that don’t, EEDAR can help you do that.
Zatkin and EEDAR collect as many as 100 or more reviews per game, and they have their own system to come up with an aggregate review score. He claims they usually come within a point or two of Metacritic. “We use reviews as a proxy for how well a game was executed, because quality is a very hard thing to measure. At the end of the day a review is a pretty good proxy for the execution and quality of a game. Right now if you mapped out unit sales to review score, you will see there is an exponential power curve going on,” he said. To put it very simply, games with high review scores sell much better than games with low review scores.
Zatkin put together a chart that showed sales against review scores, and the image is shocking. If your game is scoring an average of a 90 or above in a game review, you’re going to sell three times as many copies. Take a look at the data, with the aggregate score on the X-Axis and game sales on the Y-Axis. The first chart includes the 360, 3DS, Game Boy Advance, GameCube, Nintendo DS, PSP, PS2, PS3, Wii, and Xbox, and the data includes every game released for those systems up through December 2011.
The second chart shows games for the 360, PS3, and Wii from launch to December 2011. The higher the game’s score, the more units it’s going to sell at retail.
We’re assuming that reviews are rational, which may be a dangerous assumption to make. There is always the idea that critics are bribed, or they’re influenced by ad buys in the publications, or by gifts sent by the publisher. That sort of thing may help move one or two scores at a few publications, but Zatkin said that sort of influence is washed away by the sheer amount of scores and games they look at to create these charts. There simply aren’t enough marketing dollars in the world to move the needle when it comes to numbers that look at this many games across this many systems. Buttering up writers may work for a few games in a few circumstances, but not enough to change the data systemically.
“People buy good games, that’s been true this generation and the sixth generation of games. It’s not going away,” Zatkin said. This data set is helpful when looking backwards and finding trends, but there are so many games and systems accounted for that outliers are hard to spot. EEDAR takes into account much more data than just score vs. sales when trying to predict the future impact of upcoming games, including genre, features, and supported platforms. That’s the service they sell to clients however, we’re just interested in what reviews mean to games. And it turns out reviews mean a great deal.
Correlation vs. Causation
The question is whether good games review well and would have sold anyway, or whether good reviews actively fuel sales. Do reviews indicate good sales in the future, or do they help drive those sales? “Both, actually. It’s kind of awesome,” Zatkin explained. It’s an indicator and a driver of sales. “People like good games, people talk about them, you definitely get more buzz. There are more articles written about it. People review good games more.” The higher a game is reviewed, overall, the more people are going to review it. When a reviewer sees a game they like at E3, they’re going to ask for a copy to review. They’re going to write more previews and run more interviews. Coverage will increase.
EEDAR conducted a study with the Guildhall at SMU where they invited people who have never seen Plants vs. Zombies to come in and play the game. One group was shown reviews that rated the game highly, one group wasn’t shown any reviews, and the third group was shown reviews that gave the game a low score. The results were amazing.
When asked to score the game, the group that had read positive reviews, which gave the game a score of 90, themselves rated the game as an 85. The group without an review influence rated it a 79, while the group that had been shown the low review score of 61 rated the game a stunningly low 71 out of 100. By simply showing review scores to players before they tried the game, EEDAR and SMU were able to create a 14 point shift in the perception of the game.
“That’s called the anchoring effect, where people get a preconceived notion in their head. In this case we were doing it explicitly with reviews,” Zatkin explained. They also tried to find out how these preconceived notions changed buying habits. They offered the students $10 cash at the end of the study, or a copy of the game. 38 percent of the players shown the high scores took the game, while only 17 percent of the players who were shown the lower scores took the copy of the game. By anchoring their perception about the game’s quality before they played it, EEDAR and SMU were able to change the perceived value of the game to players. That’s the power of a review.
This works for word of mouth as well. Seventy-nine percent of the players who read no reviews said they would recommend the game to their friends, while 91 percent of the players shown good reviews said they would recommend it. Only 65 percent of the players who read the bad reviews said they would recommend it to a friend.
Game reviews are more powerful than you may think. A good series of reviews from major publications can help drive sales, and a game with reviews that average to a score of 90 or higher are likely to sell three times as many copies as a game with a score in the 80s. Players who read positive reviews assign a higher value to games, even after playing them and forming their own opinions. Higher reviews influence positive word of mouth, and recommendations given to friends. Games that go on to score highly in reviews are covered more. “They just read reviews of the game, and then they went and played the game, and we asked them about this. And it was striking. I have a psych degree and I didn’t expect to see this strong of a result,” Zatkin said.
I asked if they were able to estimate review scores and sales based on the number of stories written about games, and the tone of those stories. Zatkin said they’re working on it, and they now have a team that reads a wide variety of gaming publications in order to analyze the number of stories, previews, interviews, and even mentions of games in the press. It’s going to take a while, but they could be able to predict game review scores and possibly even sales performance based on the number of stories written about each game prior to release.
What this data means
Good reviews sell games. Bad reviews hurt games. At least those are the broad findings when you look at a large amount of data. The truth is companies like Capcom can do much to mitigate the effects of low reviews with marketing. “Awareness goes a long way. Marketing spend is actually a higher factor on sales than game quality. If you haven’t heard of the game, you’re not going to buy it,” Zatkin explained.
The low review scores will likely hurt Resident Evil 6’s chances at retail, but Capcom has the money to push the game and the brand awareness to get people interested. Most games aren’t that lucky however, and the numbers speak for themselves. If you get killed in the reviews, you have a reduced chance for retail success. Looking at retail sales as a whole, reviews line up with sales performance in a striking, direct way.
Reviews are powerful things, and perhaps, from a purely business perspective, bonuses based on Metacritic scores are more rational than we thought. When a high score has this much power to shape the sales and success of games, why not reward teams that can deliver high scores, which are shown to lead to higher sales? It’s a cynical point of view, but the numbers speak for themselves. When reviewers sit down to craft that number, they’re wielding an incredible amount of power.
More business coverage: