Serious flaws in the Google restaurant rating system

[Jason Perlow]

So, having moved to a new part of the country, in order to help us find new restaurants, my wife and I have been using Google Maps/Google Local as a basic restaurant search engine when we are looking for specific cuisines as well as well-rated restaurants.

Since acquiring ZAGAT, Google has "normalized" their ratings system in a similar fashion to ZAGAT, it has a point scale that is identical. However, the ZAGAT Survey reviews that appear on Google are compiled by ZAGAT using their known amalgamated methods of combining diner surveys and professional reviewers (as they have always been) whereas the Google restaurant reviews are 100 percent user-contributed.

So while in a search both types of reviews may appear right next to each other, a ZAGAT "27" and a Google "27" are very different.

The problem with the Google ratings system is that the ratings are assigned based on a mob-mentality type of approach.

There have been many criticisms of ZAGAT's system and the way it is weighted and what kinds of flaws it has -- historically, we've had some very big threads on this site and my co-founder, Steven Shaw, has written extensively about this.

But I have always felt that ZAGAT had at least enough sensibility and representative sampling imposed on it to make it a good initial point of reference and has a good level of base information to inform the diner about overall restaurant quality, whereas Google's system seems to make no sense whatsover and can empower a single reviewer, in some cases, to artifically inflate or knock down the rating of a restaurant, particularly if there aren't enough ratings submitted on a restaurant to give it a good representative sampling.

I will give you one particular example. This week, my wife and I went to visit two Thai restaurants in South Florida, both located within one mile of each other. The first is Pad Thai in Tamarac which has a 24 rating, and the second is Tobu Thai & Sushi in Coral Springs which until last night had a 27. Today it is a 25. I'll explain why in a moment.

The Google ratings system is composed of three data points, Food, Service, and Decor. Each of these data points is rated from 0 to 3. The overall restaurant rating is computed from an algorithm that essentially takes the total amount of user ratings in each of these areas and averages them.

So if you have a restaurant that only has a few reviews, but they are all exceptional (a 3 in each datapoint) you can easily crank the overall rating up to 27 or higher. And in ZAGAT parlance, a 27 is an exceptional, destination worthy restaurant at very high culinary standards. So you can see how easy it would be to confuse a ZAGAT 27 with a Google 27.

Comparably, a local Sushi or Thai restaurant would never end up being a ZAGAT 27, it would be considerably lower unless they were exceptional examples of the genre. For example, Jasmine Thai & Sushi in nearby Coconut Creek, is a ZAGAT 22. Which is a solid review for a local Thai/Sushi joint under the ZAGAT system.

Conversely, a few bad reviews (zeroes across the board) among only a few total reviews will also deflate a restaurant's rating.

I saw this happen in the case of Pad Thai, which I thought was a very solid, local and authentic Thai restaurant. Probably would also be a 22 or 23 on ZAGAT. One user had given the restaurant zeroes across the board, and I've captured this review for posterity just to illustrate how ludicrous it is that a single user have this much influence.

I flagged this and informed Google, but to offset this we gave this restaurant a deservedly good review. So far it has not moved the needle on the overall rating. I'm hoping they can remove this, because this is a totally unfair way to evaluate a place, especially if you haven't even eaten there.

The other restaurant we went to, Tobu Thai, was a total disappointment in every regard. You can read Rachel's review on the page I linked above. As a result of that review, it was knocked down from a 27 to a 25 overnight.

First, I think it is confusing for Google and ZAGAT to have identical numerical ratings, and they should consider giving Google a different way to identify those reviews. Second, it's obvious that mob mentality in either direction or a low representative sample is not helpful in determining the qualitative aspects of a restaurant.

My opinion is that Google needs "Super Reviewers" or people who have the experience to make informed judgements to supplant the regular reviews, sort of the way Amazon has trusted product reviewers who have a history of reviewing books and items for sale, so prospective diners can make better decisions regardless of what numerical rating the algorithm has given, or so the algorithm can make appropriate judgements/weightings.

Not only should a single reviewer's feedback be flagged as helpful or informative. etc (and I think the reviewer should get negative imput as well if there's a clear misjudgement or lack of expertise, something Google does not do now -- you can only rate a review as "Helpful" or "Not Helpful") but it should also go on their record to allow Google to build a history of confidence, or conversely, indications of bad advice for that particular reviewer. Good reviewers should get gold stars, or plusses, or whatever, just like resellers do on eBay. Bad or seriously uninformed reviewers should get bad marks.

I'd appreciate anyone's thoughts on this.

Edited March 31, 2013 by Jason Perlow (log)

[weinoo]

My thoughts are that whatever the ratings are, most of them are from amateur reviewers and they are as worthless as yelp.

[ChrisTaylor]

There's a similar issue with Urbanspoon, probably the most popular system for Australians (well, at least in the big cities). Altho' the 'rate this review as helpful/unhelpful' system doesn't seem to solve the problem. A review can be ranked as unhelpful simply because it expresses an unpopular opinion (i.e. this place isn't the bee's knees, it's merely good), even if that opinion is justified and fair. A review is not unhelpful because you disagree with it. That review is unhelpful. Another example of unhelpful is, say, 'I didn't check the prices for the degustation menu before I went but, gee, $150 for a few bite-sized morsels is really expensive! I could get a burger and fries for $7 at McDonald's!' Like not being compared with like: i.e. comparing the perceived value for money against, say, other popular but expensive restaurants would be fairer and more helpful. I think that the 'everyone's a critic' nature of these sites is the cause of the flaws. You can introduce any mechanic you like but there you have it. Even Urbanspoon, which tries to get around the problems tied to sliding scales (just look at the breakdown of IMDb scores for any popular film: plenty of people will give it a 10 if they enjoyed it but a 0 if they disliked it even a little or, shit, will give a book they enjoyed 0/5 stars on Amazon because the courier left the parcel in the rain) by having a love/like/dislike scale is flawed. Oh, this restaurant didn't give me stuff I don't deserve (like your guy wanting the early bird menu too late in the evening), that's 0 stars. Down voting that makes sense but I've also seen rational comments get down voted because, I assume, some people dislike hearing that a place that's getting a lot of love from hipster bloggers isn't actually that great.

Don't get me wrong. I think that Urbanspoon is a nice place to get a rough guide of whether a place is going to be any good. If I'm passing through an unfamiliar area and want an affordable lunch I'll rely, pretty much, on Urbanspoon rankings (after a quick glance at the actual reviews to see whether the complaints/praise is fair and balanced). That said, I'm not convinced they're much worse than professional reviews. In Melbourne and Sydney, The Good Food Guide (and a couple of sister publications) comes out every year. It's tied in with The Age and Sydney Morning Herald metro dailies. And mostly it's an okay guide to what's nice ... but at the same time it's very obvious that the reviewers have their little darlings. Some restaurants get a very high ranking when public opinion--whether you judge from people that have been there or all those amateur review sites--has turned on that place many years ago (or was never in favour of it to begin with). Some new restaurants opened by certain owners seem to go from nothing to a high ranking almost straight away. I can understand a newspaper giving a glowing review to a restaurant on day one but for an annual guide to pick up a place in its first year? I like to think of a 'three hat' (think 'three Michelin stars') as having staying power. Give it a year of consistent quality, at least. There's also, unlike Urbanspoon, a very strong bias towards certain cuisines or parts of the city.

In short, I tend to temper Urbanspoon (or similar) scores with professional reviews and the other way round. And just as I think about this owner/restaurant's history with the publication in the case of anything put out by Fairfax, in the case of Urbanspoon I skim through a dozen or so reviews. I've found that looking at the score, no matter what systems are in place to make the score fairer, is never enough. This goes for reviews by everyday punters as much as it does professionals.

Edited March 31, 2013 by ChrisTaylor (log)

[gfweb]

The problem with all of these crowd-sourced numerical reviews is that the reviewers have a huge selection bias. People who like Applebees will dine there and review it positively. Applebees can be reviewed as positively as the French Laundry, yet there is no comparison in reality.

I guess I trust Tripadvisor the most, because at least there is a written review from which more can be learned.

[Syzygies]

If a hedge fund were using Google's raw data to predict Zagat ratings, and was placing bets representing serious "skin in the game", they'd apply statistics to Google's raw data in a very different way.

Remember the Netflix Prize? At least Netflix tries to figure out reviewers in common with you, to predict individually what you'll like.

I'm just back from Brunswick, ME where I used Tripadvisor's ordered list to pick a couple of great dinners. One needed a model for why each entry was positioned where it was, having conversations exactly like this thread. Such analysis works, looking beyond the aggregate scores.

[Dave Hatfield]

Agree that Trip Advisor offers the best chance of getting somewhat accurate reviews. Yes, its crowdsourcing, but one can read the individual reviews to see why the reviewers marked a place high or low.

Reading a few of these will give a pretty good idea of whether or not the reviewers know what they're talking aBOUT OR NOT.

[huiray]

Yet one can also read the individual reviews (all of them are posted) on Google. Yes, they're there. They're on the "more info" page for any of the restaurants shown on the main google map page. It's the same as reading the individual reviews on TripAdvisor or Yelp [why hasn't this one been mentioned yet? :-) ] or Urbanspoon or other sites and judging the worth of the reviews. Google also links to these other reviews at the bottom of the "more info" page.

As for TripAdvisor, hasn't the site had (and still has) serious problems? It's also supposedly better for hotel info and patchy for restaurant info.

https://www.google.com/search?q=tripadvisor+problems&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a

http://www.telegraph.co.uk/travel/travelnews/9160472/TripAdvisor-under-fire-over-fraud-detection.html

(...and Yelp has issues of its own too...besides the usual mutterings about the competence of the amateur reviewers, just like other sites...

http://www.eastbayexpress.com/oakland/yelp-extortion-allegations-stack-up/Content?oid=1176984 )

[Edward J]

Yelp IMHO, places more emphasis on the reviewer's "credibility", than on actual truth. Should a restaurant get a good review by a yelper with little or no previous history, that review will be pulled down, but should it get a poor review from an established yelper that didn't even eat there, that review will stay for years. Logic being, that a yelper with a history is to be believed, never mind that the content of that review was terrible.

This system (Yelp) can not be compared to Zagat or even Google

[gfweb]

Tripadvisor is anal about reviewer identity, makes you swear that you've eaten/stayed at the place you review etc. I think its the most reliable of this unreliable group.

[Baselerd]

The problem is double-fold. First, people tend to think in extremes, giving restaurants (or anything) a full score or very low. Moderate scores are seldom used.

Additionally - and here's where I think the larger problem lays - the scores are not relative to each other. Someone living in a suburban area may give the best neighborhood sports bar full points, whereas they might never dine at a Michelin star restaurant. Obviously the quality of the sports bar would not live up to the high-20's score.

This isn't just a Google reviews issue, but also Yelp, Urbanspoon, etc. It may make it more confusing due to the integration into the Zagat ratings.

[ChrisTaylor]

Yeah, there's that, and I've sometimes been ... not amazed, but interested to see a little suburban cheap and cheerful joint getting the same score--or even a superior score--to somewhere like Attica (highly regarded by local guides. on the San Pellegrino list, etc). And then I step back. Now, don't get me wrong: I like Attica a lot. I like that kind of restaurant. But there's a certain ... snobbishness, if that's the right word, to saying, oh, it's in a different game altogether to the local sports bar or cheap dumpling house or whatever. Yeah, I know, part of the higher price tag involves higher quality ingredients, (probably) more labour and so on ... I get that. I get that they're offering something that's a bit more than a meal. But it's like a newspaper or magazine or, say, a website like GoodReads having two different scales or a scale that, through some magic, adjusts for how valid or artistic something else: one for your JM Coetzee/Vlad Nabokov/Ian McEwan-type guys and another, totally different, less valid scale for King and Clancy and Rowling and all the rest. And maybe it's because I'm from the suburbs, the outer suburbs at that, but I really don't like that idea at all.

To some extent, I guess the person reading the Urbanspoon/TripAdvisor/et al scores does that themselves. I don't think a 5 star sportsbar and 5 star dumpling house are offering the same thing. I am not necessarily going to enjoy a 5 star sportsbar more than a 4 star Italian restaurant. The way I read those scores is, I guess, that if I want to go to a sportsbar than this one here, which has a 5 star rating, is very good at what it does. I don't need the paper/book/website to make a different scale (i.e. 5 drunk college kids out of a possible 5 drunk college kids instead of 5 stars) for me. But based on, yeah, the opinion of people that like places like this, it's good. It's irrelevant whether the score is higher or lower than that of a different kind of restaurant, as chances are a lot of the people that eat dumplings might never set foot in a sportsbar or, if they do, might judge it by the standards of dumpling houses. I like the idea that both a fine dining restaurant and sportsbar can get five star ratings. If the sportsbar is turning out excellent steaks and such and ticking every single box for what makes an excellent sportsbar, why should its rating be limited by the fact that it's a sportsbar? It's not up to a publisher to make that distinction by actually having a different scale. It's up to the reader to look at it as such.

Edited April 2, 2013 by ChrisTaylor (log)

[gfweb]

The Zagat/Google problem is that one cannot always tell what sort of a place is being reviewed. Is it a bistro that does mainly lunch business or Les Halles in NYC? Written reviews help with this.

[huiray]

The Zagat/Google problem is that one cannot always tell what sort of a place is being reviewed. Is it a bistro that does mainly lunch business or Les Halles in NYC? Written reviews help with this.

As far as I can tell, the Google listings for a place WILL tell you what sort of place it is . You just need to go to the "more info" page. You would be on that page anyway if you are reading the individual reviews, wouldn't you? Under the name of the place would be the address; and under that the number of dollar signs (how expensive the place is) plus the hours of operation and what sort/type of place it is; and under that is usually a brief further description of the place within quotation marks. Even without the quotation-marks description just the number of dollar signs and the type of place it is should tell you the designated "level" of the place, if it is that casual cheap lunch bistro or a high-falutin' place.

If one does come across an entry devoid of all this info (I haven't seen one like that so far) then surely it becomes incumbent on one to call the place or google the place using the general google search engine?

In any case, when I use Google maps to locate restaurants in a town or city or wherever, I don't usually just just type in "restaurants" in the search box. I tend to try to narrow them down by typing in "Chinese restaurants" or "sportsbars" or "brewery" or whatever I feel like having or eating that day or night.

[huiray]

Yeah, there's that, and I've sometimes been ... not amazed, but interested to see a little suburban cheap and cheerful joint getting the same score--or even a superior score--to somewhere like Attica (highly regarded by local guides. on the San Pellegrino list, etc). And then I step back. Now, don't get me wrong: I like Attica a lot. I like that kind of restaurant. But there's a certain ... snobbishness, if that's the right word, to saying, oh, it's in a different game altogether to the local sports bar or cheap dumpling house or whatever. Yeah, I know, part of the higher price tag involves higher quality ingredients, (probably) more labour and so on ... I get that. I get that they're offering something that's a bit more than a meal. But it's like a newspaper or magazine or, say, a website like GoodReads having two different scales or a scale that, through some magic, adjusts for how valid or artistic something else: one for your JM Coetzee/Vlad Nabokov/Ian McEwan-type guys and another, totally different, less valid scale for King and Clancy and Rowling and all the rest. And maybe it's because I'm from the suburbs, the outer suburbs at that, but I really don't like that idea at all.

To some extent, I guess the person reading the Urbanspoon/TripAdvisor/et al scores does that themselves. I don't think a 5 star sportsbar and 5 star dumpling house are offering the same thing. I am not necessarily going to enjoy a 5 star sportsbar more than a 4 star Italian restaurant. The way I read those scores is, I guess, that if I want to go to a sportsbar than this one here, which has a 5 star rating, is very good at what it does. I don't need the paper/book/website to make a different scale (i.e. 5 drunk college kids out of a possible 5 drunk college kids instead of 5 stars) for me. But based on, yeah, the opinion of people that like places like this, it's good. It's irrelevant whether the score is higher or lower than that of a different kind of restaurant, as chances are a lot of the people that eat dumplings might never set foot in a sportsbar or, if they do, might judge it by the standards of dumpling houses. I like the idea that both a fine dining restaurant and sportsbar can get five star ratings. If the sportsbar is turning out excellent steaks and such and ticking every single box for what makes an excellent sportsbar, why should its rating be limited by the fact that it's a sportsbar? It's not up to a publisher to make that distinction by actually having a different scale. It's up to the reader to look at it as such.

Bravo, Sir.

Heh, I dare say that many of those reviewers who rated that sportsbar as the tops might (note I'm not saying "will") rate a place like Alinea in Chicago or Le Bernardin in NYC poorly ... they might simply not care for the kind of stuff that Alinea et al puts out, with all the high-falutin' stuff they do ... etc etc ... while foodies and people who wouldn't let anything but the freshest truffles and the best caviar on just-made blinis cross their lips would curl those same lips at the very notion of cheese nachos in, horrors! a sportsbar!!

Then again those cheese nachos may be the very best of its kind - if one were interested in that dish; while those blinis may be just nasty and greasy, even if freshly made in some fancy place.

---------

In a more general sense, certainly for my personal taste the ratings given by the local diners in any one place may not agree with my own - but I need then to remember what the "local tastes" run towards - and if I don't, and am new in town, then I will find out soon!

In my location certain places like PF Changs, for example, is rated very highly for Chinese food - but I know that that is because of the preference for Americanized Chinese food in a certain kind of surrounding. A particular Szechuanese place is rated as only mediocre - and, yes, gfweb, this is where reading the reviews do matter - most of the reviewers are eating the weekday buffet only; whereas the weekend buffet is pretty "traditional" with all the sorts of stuff that most white folks would find disgusting or challenging and which only ONE reviewer noted, while ordering from the Chinese menu (it is translated into English too) one will get pretty traditional Szechuanese dishes prepared fairly well. (Yes, I've eaten there on weekdays and on weekends)

Then, various Italian places get very high ratings locally - but again I know the local clientele loves high salt and copious amounts of cheese on everything in sight and the restaurant obliges them in that. And so it goes.

Edited April 2, 2013 by huiray (log)

[huiray]

Tripadvisor is anal about reviewer identity, makes you swear that you've eaten/stayed at the place you review etc. I think its the most reliable of this unreliable group.

Hmm. Yet it seems from the various reports about the troubles that TA has that at least some owners think some bad reviews were from folks who had never stepped foot in their establishments, at least at the claimed times the reviews related to.

[gfweb]

Tripadvisor is anal about reviewer identity, makes you swear that you've eaten/stayed at the place you review etc. I think its the most reliable of this unreliable group.

Hmm. Yet it seems from the various reports about the troubles that TA has that at least some owners think some bad reviews were from folks who had never stepped foot in their establishments, at least at the claimed times the reviews related to.

Hmmm indeed. What's your point?

[huiray]

Tripadvisor is anal about reviewer identity, makes you swear that you've eaten/stayed at the place you review etc. I think its the most reliable of this unreliable group.

Hmm. Yet it seems from the various reports about the troubles that TA has that at least some owners think some bad reviews were from folks who had never stepped foot in their establishments, at least at the claimed times the reviews related to.

Hmmm indeed. What's your point?

That at least some of those reviewers might have LIED about having eaten/stayed at the place they reviewed.

[gfweb]

Tripadvisor is anal about reviewer identity, makes you swear that you've eaten/stayed at the place you review etc. I think its the most reliable of this unreliable group.

Hmm. Yet it seems from the various reports about the troubles that TA has that at least some owners think some bad reviews were from folks who had never stepped foot in their establishments, at least at the claimed times the reviews related to.

Hmmm indeed. What's your point?

That at least some of those reviewers might have LIED about having eaten/stayed at the place they reviewed.

Of course that's possible. Duh.

One always must consider that an on-line commenter might be FOS. Even on eG. Some eG commenters are unreliable, combative and FOS.

But a reasonable person reads a reasonable review and acts accordingly.

[conifer]

The problem with the reviews on a site like TripAdvisor is that they can have a very large effect on a business. I own a bar that also serves food - small plates of bar food, for the most part, and we used to serve asian tapas/street food style plates as well. Somehow, Tripadvisor had us listed in the restaurant section of our city instead of the bar section. Yes, we serve food, but we most assuredly aren't a restaurant. I live in a city with a very large tourist segment. People started leaving positive reviews of our bar, and we rocketed up to #2 on the TA rankings for restaurants (out of about 3000). This put us above many places with Michelin stars, many fine dining restaurants...well, just about everyone. So people would come expecting a fine dining experience in what is and always has been just a small bar. And then they would leave reviews just slaying us because we weren't what they expected us to be. Things like "Good drinks, friendly staff, great prices, nice atmosphere, but it doesn't deserve to be the #2 place in Barcelona. Zero stars." Well, duh! It took us about 6 months to get TA to move us to the right category, where we are now number one. But again, someone occasionally posts something like "It was too crowded, couldn't get in. Zero stars". I have a friend with a casual restaurant around the corner who has had exactly the same problem.

In a city like this, being highly ranked on TA has a significant effect on the bottom line - very significant. If you get enough people leaving reviews like that, it could undeservedly knock you out of the top rankings, and that can mean an awful lot of money.

What's the solution? I don't think that there is an easy one. Hope that customers read the reviews and take into account that some people are unfair or misguided in their reviews. That seems about it.

[PSmith]

This news item made the papers this week in the UK, when a hostel for the homeless in Glasgow got glowing praise.

http://www.dailymail.co.uk/news/article-2314546/Jokers-Glasgow-hostel-homeless-5-star-reviews-TripAdvisor-sending-countrys-100.html

At least TripAdvisor realised what was happening and took down the reviews.

Must admit that I do use trip advisor for hotels, but rarely for restaurants. I use either local knowledge or personal recommendation.

However, I added a review to a hotel we stayed at recently and gave it a good report. Yet someone else who stayed at the same hotel, the same weekend slated it. Personally I wont give a bad review.

This article makes interesting reading and a few people would do well to take note.

http://mcslimjb.blogspot.co.uk/2013/04/11-reasons-your-yelp-reviews-suck-and.html

[Mjx]

I regularly review hotels on tripadvisor, but almost never review restaurants; within any given demographic, there's just a lot more variety of tastes related to dining experiences than there is when it comes to hotels, and I can't imagine that most people get much from strangers' restaurant reviews (whereas they'd probably be glad of a heads-up regarding a hotel with a bedbug problem first reported three years ago that is still unaddressed). I've broken my rule a couple of times (one, for Osteria Francescana, which I expect no one to take seriously, although I tried to report my great time there as objectively as possible; two for a couple of restaurants where I eat really often, and know well).

I just can't take any of these restaurant review sites seriously, and continue to be somewhat surprised that anyone uses them for something other than checking booking information, opening hours, and addresses.

[huiray]

Here's a recent dust-up on Google reviews regarding a restaurant and the immediate effect on its rating due to one-star reviews from folks who had mostly never even eaten there:

http://evgrieve.com/2014/05/a-google-glass-feast.html

http://gothamist.com/2014/05/24/east_village_restaurant_asks_woman.php

Many more links on the topic out there.

Edited May 28, 2014 by huiray (log)

[Plantes Vertes]

The TripAdvisor site for my home town has given a kebab van the same score as Le Manoir Aux Quat' Saisons, which has two Michelin stars.

 

Online reviews emerge from a self-selecting demographic that disproportionately consists of young educated people such as those in the university city where I come from. Therefore TripAdvisor reports for such cities will be stupidly skewed in favour of the places where those people like to eat. Like kebab vans.

[Tammi Townsley]

I've taken to simply scanning the reviews to see how fairly I think the individual rating ends up. I discount those "I didn't eat there cuz I hated the under/over priced menu...so I'm giving it all zeros.." type reviews. The overall ratings on many of these types of websites tend to be off kilter IMHO. So I find it impossible to take the number at face value.

[gfweb]

The TripAdvisor site for my home town has given a kebab van the same score as Le Manoir Aux Quat' Saisons, which has two Michelin stars.

 

Online reviews emerge from a self-selecting demographic that disproportionately consists of young educated people such as those in the university city where I come from. Therefore TripAdvisor reports for such cities will be stupidly skewed in favour of the places where those people like to eat. Like kebab vans.

True.

But still, the written part of a TA review tells you what you need to know.