• Hey, guest user. Hope you're enjoying NeoGAF! Have you considered registering for an account? Come join us and add your take to the daily discourse.

Numbered Reviews Must End

viveks86

Member
Made a small edit.

Haha. Though I agree, I don't think console warriors are the only problem. It could be anyone. Imagine if half life 3 gets an 8.5 (I'm not sure which is more unlikely, the game or the score). Don't you think there will be a unanimous uproar against reviewers? The issue here has to do with misguided passion, be it platform, genre, developer or game.

Also, game publishers are over reliant on these scores as well. People get fired, projects suspended if a game isn't the critics' darling. That's just ridiculous. I can see the metacritic score influencing sales a bit, but that can't and shouldn't be given so much significance.
 

Jea Song

Did the right thing
Remember when EGM magazine had like 3 people review a game? They average score would be the total score. I liked that style of review. We need 3 people's option on a game.. Not just one person.
 
I don't have as big a problem with the numbers as I do the inconsistencies with the numbers. An 8 is only an 8 if the average review score is 5-6! Otherwise the 8 is the new average and therefore meaningless!
 

McSpidey

Member
Whatever ranking system you use someone will find a way to convert it to a number for comparison. We're pattern finders.

Despite this I fully support the death of numbered reviews in favour of a recommend/excitement rating. This keeps the ranking as subjective as the content.
 

NervousXtian

Thought Emoji Movie was good. Take that as you will.
I think game "reviews/critics" should go away period. The system as is is very broken. Really both the new consoles are set up for this. Go to the share area, look for the game you are interested in, and watch for a bit... watch a few different people play. That will give you all you need if you are on the fence. Game "journalists/critics" can go extinct... and should have been put out of their misery long ago.

Journalists should do what journalists do. They should be reporting on the industry, the studios, the people, the culture, the players, and the adventures they have in game.
Comments like "it's too last gen" are not relevant to the industry or the players. All we need is game studios making games just to appease the critics... hence all of the multiplayer tack-ons added to games last gen. I hope this trend does not continue... it's a waste of everyone's time just so that the game won't lose critic points because it didn't have a multiplayer element.

Let's Plays spoil games.. also unless you watch the whole thing how do you make an opinion of if the whole things holds up.

I'm glad it's an option for those who want it.. but to say the standard review system is broken because YouTube & Twitch exist is absurd.

We have that today: it's called reading more than one review.

So much ether in this here thread. Love it!
 

Vice

Member
Remember when EGM magazine had like 3 people review a game? They average score would be the total score. I liked that style of review. We need 3 people's option on a game.. Not just one person.
Metacritic does that on a wider scale. . .
 

r3ddvil

Member
I like the movie review system of generally 1 to 4 stars and the whole scale gets used. I especially hate breakdown and "averaged of score criteria" style reviews, I could care less about the graphics, if the game is fun (and I don't care how pretty a lousy playing game is)

Has anyone seen a movie review with breakdown numbered scores for acting, directing, story, and special effects? While characteristics will lend themselves to final verdict, in the end it's about how everything comes together, and you dock a bit if there is something really bad that stands out to lessen the experience, and bump a bit if something sticks out as exceptional in an otherwise mediocre experience.
 
A 100-point scale is stupid. Believe it or not, I actually prefer the old GamePro scale:



There are certain things you can give objective scores for. For example, a game that manages 1080p@60fps should get a better score in that category than a different version of the same game running at half the frame rate, or you may give a game that stutters to single digits a low score in that category. Things like audio and gameplay may still get high scores in this scenario, though, which is perfectly fine. There's too much stuff happening in games at once to lump everything into one score, imho.

This really is the only way to rate a game. We need to take a Technology Review standpoint where everything is broken down into parts, like how a review for a Cellphone breaks it down into Design, Camera, Display, Call Quality, ect. A score for games should be broken down into Scores for graphics, sound, gameplay, ect. In addition, numbers should never be combined into an overall score. Keep everything partitioned.

Until everyone does this, Numbers are completely subjective.
 

MC Safety

Member
Not in the way where they would play off each other in the reviews. The last person could disagree with the first, make comments on something the 2nd one said.

Most of those reviews were very short, and any interplay came at the expense of criticism and analysis.

I used to write for EGM when it was still published by Ziff. I want to say the small reviews were about 80 words long. It's hard to do much of anything with that limited word count.
 
This really is the only way to rate a game. We need to take a Technology Review standpoint where everything is broken down into parts, like how a review for a Cellphone breaks it down into Design, Camera, Display, Call Quality, ect. A score for games should be broken down into Scores for graphics, sound, gameplay, ect. In addition, numbers should never be combined into an overall score. Keep everything partitioned.

Until everyone does this, Numbers are completely subjective.

And those individual numbers would still be just as subjective. This doesn't resolve anything.

Also, I hate the idea of games be reviewed as products instead of as an artistic whole. Films aren't broken down into separate Script, Acting, Special Effects scores, and I don't think games should either. It's about how it all comes together; rating the individual pieces outside of that context provides no additional value.
 

viveks86

Member
Not in the way where they would play off each other in the reviews. The last person could disagree with the first, make comments on something the 2nd one said.

That is a fair point, but this raises other issues:

1) What do you do if all of them have opposite opinions, which is quite common in this medium? The average score ends up being even more meaningless than the individual scores.

2) This implies that more than one person needs to be paid to review the game, thereby increasing costs. The more it costs the website to do this, the more they transfer that cost to the reader. Enter subscription charges and/or more advertisements (as if we don't have enough already).

3) Committing more resources to review the game and play off each other implies a significant amount of time spent on the review process itself, which makes reviewing games before release difficult. With increasing consumer expectations for early reviews, this just adds pressure on everybody and could affect the quality of the reviews.

Basically, there are down sides to any method. People just need to make the best of whichever they choose.
 

Dire

Member
Hear, hear! Copy/pasting from another thread...

Yeah, it's one big toxic cycle that will only end when gamers stop caring enough to make threads like this. Bonuses are tied to Metacritic scores because Metacritic scores affect sales, and Metacritic scores affect sales because gamers obsess over whether the newest AAA game just got 8s or 9s, as if a game's quality can be quantified like a piece of meat......

This is ironic and frustrating. You're implying a causal relationship. Eg - gamers buy games because of a high metacritic score. What if good games tend to sell more, and good games tend to also have above average review scores?

This isn't just about semantics. These are two far different possibilities and it cuts to the core of the supposed problem, which I do agree is a problem, but for different reasons (more on that in a minute). If metacritic scores simply correlate with the quality of a game as do sales, then getting rid of metacritic scores would do absolutely nothing except remove a convenient resource for players. Now on why I actually agree it's a problem, but for different reasons. Publishers have also assumed a causal relationship and they're starting to develop games based explicitly around achieving metacritic scores. The games media and game players don't exactly see eye to eye. When publishers start going overboard and doing things like scrapping concepts based on mock reviews and the like then they're no longer making games for players, but games for the games media. This is something that may potentially hurt the correlation between "good games" (from a players perspective) and above average review scores. And that is a problem.

If publishers believe that there is a causal relationship between reviews and sales then getting rid of scores wouldn't actually do that much. They'd still engage in the same sort of behavior aimed and targeting the games media with their games. And for that matter the games media rarely works as an actual media outlet and instead are more of a PR branch. Here is IGN's lead paragraph for their Titanfall review:

After my first few rounds of Titanfall, hearing the “Your titan is now ready” notification began to induce a Pavlovian adrenaline-rush response. I still catch myself looking up to the sky as I press down on the D-pad to call it in, because watching my 20-foot-tall robot exosuit fall onto the battlefield, seemingly from Heaven, is a glorious sight that I still see replaying when I close my eyes at night. It’s a signal that I’m about to transition from the liberating mobility of a jetpack-powered, wall-running soldier (called a pilot) to the ego-swelling walking tank that punches enemy players midair as they try to leap on its back and squashes AI-controlled minions with heavy metal feet. It’s more than a “Call of Duty with mechs” gimmick – Titanfall turns out to be an invigorating multiplayer first-person shooter that melds fresh mechanics with familiar ones, creating a new watercooler moment almost every time I play. I only wish there was more of it, and that it was easier to fight my friends.

That's splattered right below some giant audio/visual hype ad for the game that auto plays when you visit the review page. You don't need to see some number to know it's going to get an absurdly high score from them. Imagine if Ebert had wrote a review that began: "With the action packed sequences from the first moments my heart began to palpitate - I literally could not sit still in my chair. The excitement was palpable - the experience unprecedented.. blah blah blah" But he wouldn't because that's retardedly unprofessional. That, like the games media already long since has, goes well beyond the realm of objective critique and into the realm of fanboy PR. The problem isn't about the numbers, it's about assuming a casual relationship which is driving publisher influence on the media - both coercively and simply in terms of what they release. Ironically, you assumed the exact thing in your critique of the problem!
 

Eiolon

Member
The scale of 1-10 is too large. Anything below an 8 no one wants to buy. When something get's 7.9'd it's the end of the world for some.

I prefer the star system myself, though I think 5 stars is too much.

1 star = bad, don't rent, don't buy
2 stars = average (good things, bad things, worth a rental or bargin bin purchase)
3 stars = good (primarily an excellent game with very few flaws, worth a full price purchase if the type of game is your style)
4 stars = exceptional, does nearly everything right and you can only nit pick on most things. This would be the equivalent of the coveted 10.
 
Let's Plays spoil games.. also unless you watch the whole thing how do you make an opinion of if the whole things holds up.

I'm glad it's an option for those who want it.. but to say the standard review system is broken because YouTube & Twitch exist is absurd.

Why would you need to watch the entire game to decide? If you did then it's a good sign that the game isn't worth your money. How did you know if a game was good in a video arcade? You noticed the gathering, and you watched the game being played... if it looked like it was worth your quarter you stuck it on the screen seal. This will only make sense if you are old enough to remember arcades before NES.
 
Honestly, I think reviews are fine as is.

What needs to happen is for people to understand what they are, what an opinion is, and stop getting so invested in the number hit on at the end!

Scores exist as a way to quickly summarise a game review or organise it into a category (good, amazing, average, etc.), what happens is that people cling onto that number and then proceed to make a judgement regarding the entire game without paying attention to the actual review itself.
 

Dire

Member
...1) What do you do if all of [the reviewers] have opposite opinions, which is quite common in this medium? The average score ends up being even more meaningless than the individual scores....

Strongly disagree. Think rotten tomatoes.

Gattaca is viewed as a classic of enormous import by many. It received a 64/100

Titanic is the GTA of the movie industry. Huge budget, huge advertising, TITS, LOVE STORY, ACTION, primarily positive reception: 74/100

Avatar is the highest grossing film of all time: 83/100

Unanimous agreement is typically the sign that either you have a one in a million treat on your hands or that something has gone terribly wrong in the system. In video games I think it's obviously the latter almost always. GTA4 received a 98. That score is just stupid. Disagreement is the heart of consensus! If at least one person isn't disagreeing then your sample sucks or there are shenanigans at play. In the current state of video games - it's both.
 
I don't mind numbers, I just ignore them. I read the meat of the review then make my decisions based on that, I normally don't even look at the score. Still, I like formats without scores better, like how Totalbiscuit does it.
 
I like simple Yes or No system. Buy or Don't. Thumbs up, thumbs down. It makes sense for reviews, not just some completely arbitrary nonsensical number that doesn't even hold up as a relative rating against other games due to the fact most sites have multiple people with ranging opinions doing the reviews rather then one unified perspective.

That said, I don't really care about reviewer opinions anyways, I only watch video reviews occasionally to treat them like glorified gameplay trailers, or in the cases of people like Totalbiscuit to get info on features and how broken they might be.
 
Unanimous agreement is typically the sign that either you have a one in a million treat on your hands or that something has gone terribly wrong in the system. In video games I think it's obviously the latter almost always. GTA4 received a 98. That score is just stupid. Disagreement is the heart of consensus! If at least one person isn't disagreeing then your sample sucks or there are shenanigans at play. In the current state of video games - it's both.

It should be noted that MetaCritic has far from every review site on there--which is kind of a problem in and of itself. They selectively choose which sites to represent, which could also be a factor in the perceived consensus.
 

viveks86

Member
Strongly disagree. Think rotten tomatoes.

Gattaca is viewed as a classic of enormous import by many. It received a 64/100

Titanic is the GTA of the movie industry. Huge budget, huge advertising, TITS, LOVE STORY, ACTION, primarily positive reception: 74/100

Avatar is the highest grossing film of all time: 83/100

Unanimous agreement is typically the sign that either you have a one in a million treat on your hands or that something has gone terribly wrong in the system. In video games I think it's obviously the latter almost always. GTA4 received a 98. That score is just stupid. Disagreement is the heart of consensus! If at least one person isn't disagreeing then your sample sucks or there are shenanigans at play. In the current state of video games - it's both.

Hmmm? I totally agree with everything you said. You are taking my point out of context. I was specifically referring to the suggestion that having multiple people collaborate and give a single average score isn't any better than just one person giving a score. If one person gives a 0 and the other gives a 100, then those scores are useful to process separately. The average of 50, is meaningless. That's all I was trying to say. All your examples are spot on. They highlight exactly why people shouldn't take scores too seriously.
 
How about week prior to launch paid demo's for all games.

$1 for an hour of play. Then you can decide if you like it, or not. You really don't need someone else to tell you whether you like something or not. For those that played the beta of Titanfall do you think any of the reviews meant anything to them... playing the beta made the grade or didn't.
 
Firstly, I've never seen a reviewer give a game a 8.1 or 8.2. Those numbers only show up when averaging reviews from multiple reviewers. In that case it's fine if you want to know where the general consensus is on a game. In a single review a number isn't as important.
 
Indeed. Can you imagine if you bought every game they recommended that you play? That would be the majority of games...

Well but the point isn't to go strictly by the "yes/no" rating. I hope you don't just buy every 8.0 or better game just because that. The rating is just supposed to be the reviewers "sum" of the whole equation of pluses and minuses they discuss in the review, and only a guide post for you as you do your own math on that equation and come up with your own answer.

The benefit of a yes/no system is just that its much more easily explainable in a review. You can add up what you like and don't like and decide if it balances out for a yes or no. When you do some arbitrary number or stars or whatever theres no real structure to it. They are just pulling a "percentage goodness" out of their bums. I kind of like it, its good but not great, 8.3. What? A few places do it more strictly where perhaps like 5 categories take up 20 points each, add up to 100 for final score, but then your just making arbitrary point choices on individual assets and adding them together, same thing in the end. None of it makes sense, but they do it because its good for marketing, when people just see the score and not the review behind it.

I think if you actually read a review you should completely ignore its score. Those aren't really meant for you, they are meant for metacritic and box art.
 
Firstly, I've never seen a reviewer give a game a 8.1 or 8.2. Those numbers only show up when averaging reviews from multiple reviewers. In that case it's fine if you want to know where the general consensus is on a game. In a single review a number isn't as important.

IGN and a lot of other sites do it all the time.
 

kaching

"GAF's biggest wanker"
The 5 point scale is too few now that metacritic has such a hold on the industry.

There's 20% difference between each point, there's a very big difference between a 60%(3) and 80%(4).
And in each case they're doing games a favor by assuming it's the top of the segment on a percentile scale, rather than the bottom (41% or 61%). It's fine.

The use of a 5pt scale by any review outlet expressly implies they don't want to get too granular with review scores, so I don't see the problem in representing their scores in a rough order of magnitude on a percentile scale; it abides by the spirit and intent of their actual rating scale.
 
Well but the point isn't to go strictly by the "yes/no" rating.

Yep, thus rendering it even more meaningless.

The benefit of a yes/no system is just that its much more easily explainable in a review.

Only because it's sufficiently vague. I'm not an advocate at all of the 100-point scales, but take something like Giant Bomb or Joystiq's 5-point scale, which I find to be not only much more useful than a binary scale in how it more adequately summaries the reviewer's opinion, but is also just as easily justified via the review text.
 

whitehawk

Banned
It should be out of 5. With no decimals. Rev3Games knows what's up.

1 - Avoid at all costs
2 - Not very good, only buy if you're a big fan of the series/genre.
3 - Fun game. Nothing amazing, but you will likely enjoy it.
4 - Very solid game, recommended.
5 - Must-buy game.
 

emag

Member
Hmmm? I totally agree with everything you said. You are taking my point out of context. I was specifically referring to the suggestion that having multiple people collaborate and give a single average score isn't any better than just one person giving a score. If one person gives a 0 and the other gives a 100, then those scores are useful to process separately. The average of 50, is meaningless. That's all I was trying to say.

That's one reason Metacritic pulls from such a large base of reviewers. If ninety reviewers think a game is good and ten reviewers think it's horrible, the score will reflect the mostly positive rating. In practice, there are very few games where the distribution of scores is bimodal; in general, review scores will be clustered around a single peak with a few outliers.

All your examples are spot on. They highlight exactly why people shouldn't take scores too seriously.

All the movies mentioned were competent and scored well. The vast majority of moviegoers would agree that any one of those movies (scoring in the >60 range) is better than the movies that score in the <30 range.


How about week prior to launch paid demo's for all games.

$1 for an hour of play. Then you can decide if you like it, or not. You really don't need someone else to tell you whether you like something or not. For those that played the beta of Titanfall do you think any of the reviews meant anything to them... playing the beta made the grade or didn't.

There were over 100 Metacritic-rated games released in 2013 on the Xbox 360 alone. Would you be willing to spend $100 and 100 hours trying each out for yourself?

But that's the point! We don't want to talk down to readers or act as if our opinions are better than theirs in any way. That's one of the reasons we don't use scores. They're too definitive. We happen to be informed gamers and experienced writers, so hopefully we're good at capturing how a game feels to play, but nobody at Kotaku wants to pretend that our opinion is better or more valid than any one of our ~12m monthly readers. That's why we write as if we're talking to a friend. It's also why we share stories and information that other outlets might not -- part of our company's mandate is to tell the "real" story behind stories, rather than the puffed-up version.

But I won't ramble on about Kotaku - to get back to the main point here, yet another bullet-point on the very long list of reasons not to use review scores is that they're too authoritative. "This game is a 7." What does that mean, and who are you to even say it?

Kotaku does effectively use a explicit binary review scoring system (Yes/No) with the addition of simply ignoring games that it deems unworthy of reviewing. We can quibble over how broad the range of scoring should be, but to pretend that Kotaku doesn't use an "authoritative" or "definitive" voice because it uses a word to represent each of its two explicit scores is absurd.
 

ghst

thanks for the laugh
And after a big Nintendo game, and a big Microsoft game, and a big PC game, and...

No, there's no GAFbias in that regard. Just a lot of people who each, individually, are certain that a GAF hivemind is entirely up the arse of [insert] platform/brand.

almost all of the big anticipated PC releases from last year (rome 2, company of heroes 2, sim city, arma 3, x: rebirth) received a mix of luke warm and outright savage reviews. i don't remember a similar reaction in any of their review threads. in many cases (CoH 2 and sim city in particular), the audience were by far the loudest critics.

even the titanfall thread was markedly even handed considering the weight of that release.
 
What I don't understand is why the "number" effects you? Did you enjoy a game? what did you think? Based on what you gathered, if you care enough about the score it got, it's obvious you have enough interest in the product to warrant a purchase. So why should the number that might not reflect your opinion matter to you?

That's my whole thing. Its comical that people get worked up over a score and the OP in this sense is comical. The score is what it is. Society rates shit and people use this to help make a judgement. But if it's the only thing they are using to help make a judgement then that is their own ignorance.

Point is, what you think and feel about a product should be most important, not what others think about or how others quantify it.

even the titanfall thread was markedly even handed considering the weight of that release.

Oh please, it wasn't the titanfall official thread, but it was all the other threads that eventually got locked.
 
Numbers have to go. It would be one thing if there were some standard to which points are added and taken away, but it's almost completely random. Certain games have points taken off for this or that while other games aren't penalized for having the same issues. This happens not only across an individual publications, but even individual reviewers. Until there is some consistency with how games are reviewed individually and collectively, numbers have absolutely got to go.
 

NervousXtian

Thought Emoji Movie was good. Take that as you will.
Why would you need to watch the entire game to decide? If you did then it's a good sign that the game isn't worth your money. How did you know if a game was good in a video arcade? You noticed the gathering, and you watched the game being played... if it looked like it was worth your quarter you stuck it on the screen seal. This will only make sense if you are old enough to remember arcades before NES.

Yes, I am old enough to remember that. I didn't need to see all that much of most arcade games back in the day... if you watched them for a couple minutes you got everything you needed to know.

The nickle arcade was a multiple time a month journey for us as kids. A lot of games aren't like that anymore.

This is ironic and frustrating. You're implying a causal relationship. Eg - gamers buy games because of a high metacritic score. What if good games tend to sell more, and good games tend to also have above average review scores?

This isn't just about semantics. These are two far different possibilities and it cuts to the core of the supposed problem, which I do agree is a problem, but for different reasons (more on that in a minute). If metacritic scores simply correlate with the quality of a game as do sales, then getting rid of metacritic scores would do absolutely nothing except remove a convenient resource for players. Now on why I actually agree it's a problem, but for different reasons. Publishers have also assumed a causal relationship and they're starting to develop games based explicitly around achieving metacritic scores. The games media and game players don't exactly see eye to eye. When publishers start going overboard and doing things like scrapping concepts based on mock reviews and the like then they're no longer making games for players, but games for the games media. This is something that may potentially hurt the correlation between "good games" (from a players perspective) and above average review scores. And that is a problem.

If publishers believe that there is a causal relationship between reviews and sales then getting rid of scores wouldn't actually do that much. They'd still engage in the same sort of behavior aimed and targeting the games media with their games. And for that matter the games media rarely works as an actual media outlet and instead are more of a PR branch. Here is IGN's lead paragraph for their Titanfall review:



That's splattered right below some giant audio/visual hype ad for the game that auto plays when you visit the review page. You don't need to see some number to know it's going to get an absurdly high score from them. Imagine if Ebert had wrote a review that began: "With the action packed sequences from the first moments my heart began to palpitate - I literally could not sit still in my chair. The excitement was palpable - the experience unprecedented.. blah blah blah" But he wouldn't because that's retardedly unprofessional. That, like the games media already long since has, goes well beyond the realm of objective critique and into the realm of fanboy PR. The problem isn't about the numbers, it's about assuming a casual relationship which is driving publisher influence on the media - both coercively and simply in terms of what they release. Ironically, you assumed the exact thing in your critique of the problem!

Why exactly is that IGN paragraph PR? Pretty much captured my exact feelings while playing the game on my PC.

So automatically if someone likes something, they are just part of the PR machine?

Isn't it really just easier to believe that reviewers just like or dislike things, because they like or dislike things.. instead of some grand conspiracy theory?
 
It would be one thing if there were some standard to which points are added and taken away, but it's almost completely random..

Uhh, I don't think it works like that for most sites. It's not like "long load times = -1 pt." The score is a representation of a reviewer's overall opinion, hence why they often have a word that goes along with them saying exactly what the number represents.
 

viveks86

Member
That's one reason Metacritic pulls from such a large base of reviewers. If ninety reviewers think a game is good and ten reviewers think it's horrible, the score will reflect the mostly positive rating. In practice, there are very few games where the distribution of scores is bimodal; in general, review scores will be clustered around a single peak with a few outliers.

All the movies mentioned were competent and scored well. The vast majority of moviegoers would agree that any one of those movies (scoring in the >60 range) is better than the movies that score in the <30 range.

Yup. Agree with both your points. I don't see any issues with the way metacritic compiles these scores. My point was specific to individual sites making aggregates based on a couple of reviewers. The only issue I have had is the way those scores are perceived in the gaming industry. Like it's some kind of horse race and every point matters! It's totally bonkers.
 
Top Bottom