Star Ratings - Isn't There A Better Way

[weinoo]

Most posters on eGullet tend to agree that the star ratings given to restaurants, whether it's the NY Times, Time Out magazine, SF Chroncle or numerous other publications are fairly useless. A one-star review one week may not objectively be the same as a one-star review a few weeks down the road. One star out of five is certainly different than one-star out of four or out of six.

But how can it be done differently? I'm wondering if perhaps the use of a 50 or 100 point rating system, ala Iron Chef, would work. That is, a restaurant is given 1 - 10 (or 1 - 20) points on various aspects of the experience; for example, taste, originality, atmosphere, service and let's say wine are each judged and awarded 1 - 10 points. Now, Zagat's suggests that their ratings are similarly based by "averaging" the ratings that reviewers like you and me give to restaurants for food, service and decor; whether or not that figures in the final rating a restaurant gets from Zagat can be left to each individual reader of the guide to decide - I personally haven't bought a copy in years, because I can find all the info I now need online.

The bottom line is that I think that with a review system based on certain criteria, with points given to each of the criteria being tallied for the final rating, we'd end up with a more usable system - that is, one in which we can compare any restaurant with any other restaurant in order to be able to somewhat accurately assess what our dining experience might be.

Agree? Or any better ideas?

[Lisa Shock]

I don't think that increasing the number of available stars is the best answer.

IMO, places should be categorized and then rated within the category. As things stand now, a carryout only place has no chance at being 2-3 stars in the Michelin guide, because there is no real 'front of house' service to rate. Many review standards are geared towards fine dining, and, while it's a fair way to rank fine dining places it doesn't do places like Pizzaria Bianco here in Phoenix justice.

[oakapple]

I may be in the minority, but I don't find them useless. One does need to know how the system works—which most publications do not properly explain—to derive utility from them. In New York City, for example, the New York Times three-star restaurants generally do live up to the literal meaning ascribed to them: "excellent" for the type and style of cuisine they represent. It so happens that I have been to all of the NYT 3* places myself, so I no longer rely on the paper for that information. But if I hadn't been to these places, I feel I would have a pretty good idea of what the rating meant.

Stars are only a filtering mechanism that helps you narrow down what you might be looking for. There is no substitute for reading the review, where any caveats are clearly stated. In NYC, for instance, Momofuku Ssäm Bar doesn't take reservations, and guests sit on backless stools. That is not normally the case for a three-star restaurant, and most people would want to know that.

The one- and two-star ratings are more problematic, for a few reasons. (I am referring again to the New York Times system; others may work differently.) One- and two-star restaurants are far less likely to be re-reviewed, so there is a higher likelihood that if the review was written a while ago, it no longer accurately describes the restaurant. The other problem is that the Times uses two stars not just for "very good casual restaurants," but also for "failed luxury restaurants." It is therefore much more likely that two restaurants with the same rating won't be remotely comparable.

I agree with Lisa Shock that a finer-grained rating system (20, 50, or 100 points) would not cure whatever defects the system has. On my blog, I use half-stars (so do some newspapers), which provides a bit more granularity. However, it still doesn't change the fact that you've got restaurants with the same rating that are not directly comparable. Though the New York Times does not explicitly say this in their explanation, I think it is safe to say that stars are comparable only when the establishments themselves are similar.

I do think it would be helpful to give separate ratings for food, service, and ambiance, as Zagat does, along with an overall rating. This is something the Times could do immediately, without invalidating any of the ratings they've previously given out. The Michelin Guide sort of does this. In addition to the stars, each restaurant is assigned a number of couverts (one to five), which is a measure of the "level of comfort," as they call it. But the couverts receive nowhere near the level of attention that the stars do.

[David A. Goldfarb]

I'd rather get rid of the star and ratings systems culture altogether. Make people read narrative reviews and think.

[oakapple]

I'd rather get rid of the star and ratings systems culture altogether. Make people read narrative reviews and think.

Of the various "solutions" that are offered from time to time, that is the least likely to occur. In NYC, one paper actually dropped the stars a few years ago and recently brought them back because they were demanded by many readers. In the intervening period, several publications added stars that hadn't previously had them.

[Chris Amirault]

I've signed on as a cocktail contest judge, and that experience has made it clear that an add-em-up point system like the one Mitch proposes doesn't work. It's what we're using for this contest, and it's driving me crazy. Pretty drink, nice explanation, tastes like crap: I need to be able to give that libation a low score, but using the scale, I can't fairly do so. Ditto for Mitch's restaurant scale:

[A] restaurant is given 1 - 10 (or 1 - 20) points on various aspects of the experience; for example, taste, originality, atmosphere, service and let's say wine are each judged and awarded 1 - 10 points.

Using that scale, you would have to give a luxe joint serving utterly original junk 10s in all categories except the one that would lead most other raters to give the place no stars at all.

Rating scales can be built to take that sort of thing into account. For example, you could assign an overall rating based on the lowest category score, which solves the luxe junk problem but doesn't solve what we might call the Imperial Palace problem: given the wine list, Sifton would've been required to give that place a bottom-rung rating.

[jsmeeker]

I'd rather get rid of the star and ratings systems culture altogether. Make people read narrative reviews and think.

This is the only real alternative. A local (Dallas) movie critic does just that. He's on TV, it's not a "written" reivew, but he doesn't use stars or movie reels or points or anything like this. Just commentary on the film.

Of course, a system like this has real challenges in the modern "Google" world. People will want to search for restaurants with some minimum "score". A review without that makes that impossible.

[taion]

[A] restaurant is given 1 - 10 (or 1 - 20) points on various aspects of the experience; for example, taste, originality, atmosphere, service and let's say wine are each judged and awarded 1 - 10 points.

Using that scale, you would have to give a luxe joint serving utterly original junk 10s in all categories except the one that would lead most other raters to give the place no stars at all.

Rating scales can be built to take that sort of thing into account. For example, you could assign an overall rating based on the lowest category score, which solves the luxe junk problem but doesn't solve what we might call the Imperial Palace problem: given the wine list, Sifton would've been required to give that place a bottom-rung rating.

But this is basically exactly what Zagat does, and, net of issues with the reliability of Zagat ratings themselves, is not inherently flawed.

In general some sort of numerical rating should probably be in place. Everything else aside, reviews are there to be useful to the reader, and being able to sort and filter to find a specific restaurant adds a lot of value. Ultimately a review is supposed to help me figure out where to eat; being forced to read through every single one helps me a lot less than being able to, say, look for all restaurants with 2 or more stars in a given area.

[Kent Wang]

For Fearless Critic we have ratings from 1.0 to 10.0 for food and feel. It seems like a major problem with a lot of guides like Zagat (and I think NY Times, to a lesser degree) is that great casual and ethnic restaurants never get a high rating. We say nuts to that. Our Austin Top 10 has three barbecue places and two southern cuisine places. Really, if you come to Austin and you want to know where you should eat, barbecue and southern cuisine are going to be where you want to focus on, instead of high-end New American that you can get anywhere in the country.

And we use the full range 1-10. For a lot of other guides 7 is the lowest rating they give, so then you're only having a 3 or 4 point scale.

We also break out ratings for wine, beer, and cocktails for restaurants that have a significant program. So that includes restaurants with great wine lists and wine bars that make a big fuss of their wine program but really suck.

[Holly Moore]

No stars and define the column's mission as seeking out good places to eat. Especially in a large city, too many good restaurants never get ink because the reviewer squanders column inches on bad experiences.

If readers know the review is going to be about a restaurant that the reviewer liked, stars become less necessary.

For those without the patience to read a thousand words about the place, an "executive summary" that condenses the review into a single paragraph.

[Sneakeater]

I think any kind of numerical rating is an insult to both the reader and the subject of the review. It's for people who are too lazy to read a set of words in sentences in paragraphs and figure out what the author is saying.

The Times doesn't give ratings to shows or to concerts or to operas. It's an insult to gastronomy that they give stars to restaurants.

[oakapple]

The Times doesn't give ratings to shows or to concerts or to operas. It's an insult to gastronomy that they give stars to restaurants.

The proof that this is false comes from the chefs and restauranteurs themselves. Did you hear Daniel Boulud's reaction when he got the third Michelin star? He was floating on air. If he felt insulted, it was a marvellous acting job to pretend the opposite. I do realize that a few restauranteurs have given up their Michelin stars, but this is comparatively rare. Nearly all chefs profess themselves delighted when they receive a high star rating.

Obviously the chefs who don't get the rating they expect are unhappy, but any meaningful system needs to disappoint some people. Even then, I seldom hear chefs and restauranteurs saying that they wish the stars would go away; they just wish they had more of them.

[oakapple]

For Fearless Critic we have ratings from 1.0 to 10.0 for food and feel. It seems like a major problem with a lot of guides like Zagat (and I think NY Times, to a lesser degree) is that great casual and ethnic restaurants never get a high rating.

I can't speak for other cities, but in New York that is untrue. For instance, in the NY Zagat Guide, Di Fara Pizza has a 27 food rating (the highest is 28). Momofuku Ssäm Bar, an extremely casual restaurant, has three New York Times stars.

If anything, Zagat tends to give casual and ethnic restaurants higher ratings than they truly warrant, not the reverse.

[Chris Amirault]

But this is basically exactly what Zagat does, and, net of issues with the reliability of Zagat ratings themselves, is not inherently flawed.

Mitch was proposing one score based on adding up the individual scores. IIRC Zagat doesn't give one accumulated score; they list scores by food, decor, and service, right? I haven't used a Zagat guide for years....

[TAPrice]

But this is basically exactly what Zagat does, and, net of issues with the reliability of Zagat ratings themselves, is not inherently flawed.

Mitch was proposing one score based on adding up the individual scores. IIRC Zagat doesn't give one accumulated score; they list scores by food, decor, and service, right? I haven't used a Zagat guide for years....

That's correct. People score restaurants in three categories (Food, Decor and Service) on a 1-3 scale. Those scores are averaged and then multiplied by 10 (i.e. a 2.7 average becomes a 27 score).

[Full disclosure: I've worked as a local editor for Zagat, but I am in no one speaking on behalf of them.]

In my day job, I'm a lecturer in Spanish. It's amazing how these debates over rating restaurants mirror the arguments we have about how to grade student composition in basic level classes.

I agree with Chris that simply aggregating several categories often gives you results that just don't make intuitive sense. I'd rather have the subjective overall judgment a good critic.

[TAPrice]

The Times doesn't give ratings to shows or to concerts or to operas. It's an insult to gastronomy that they give stars to restaurants.

Sure, but shows, movies and concerts are all, relatively speaking, fleeting products. Restaurants stick around a lot longer and there is value, as a consumer, in having a quick way to rank them. When I'm thinking about going to an art exhibit or concert, I don't need to consider three years worth of critical output from a publication. If I'm deciding where to go out to eat, then I very well might want that range of opinion and in that situation the stars serve as a useful shorthand.

[Kent Wang]

For Fearless Critic we have ratings from 1.0 to 10.0 for food and feel. It seems like a major problem with a lot of guides like Zagat (and I think NY Times, to a lesser degree) is that great casual and ethnic restaurants never get a high rating.

I can't speak for other cities, but in New York that is untrue. For instance, in the NY Zagat Guide, Di Fara Pizza has a 27 food rating (the highest is 28). Momofuku Ssäm Bar, an extremely casual restaurant, has three New York Times stars.

If anything, Zagat tends to give casual and ethnic restaurants higher ratings than they truly warrant, not the reverse.

I don't have the current Zagat guide, but by looking at the top 25 on their website, Di Fara is the only one. 25-50 has Sripraphai. Out of 50 top restaurants in the city, only 2 are less than $20 a head. If your friend was visiting from out of town for a month and could visit only 25 restaurants, would you recommend 24 expensive places and Di Fara. No Chinese, no other ethnic?

There's about 10 sushi restaurants like Nobu in the top 50 that I would rather go to Sripraphai before going to them.

Momofuku is casual, but it ain't cheap. The food is definitely high-end. Maybe I should have clarified this.

[Fat Guy]

Probably the best piece ever written on this subject was James Poniewozik's "Media Circus" column in Salon more than a decade ago. He makes a number of arguments against star and numeric ratings (which are the same thing), most importantly that they are self-defeating for critics: that once you reduce real criticism to numbers you create the conditions for not needing real criticism.

if critics are someday supplanted by search utilities and kibitzing online "neighbors," they -- or their editors -- will have paved the way. For these mechanisms are the logical, idealized extension of slapping numbers on subjective writing: criticism as a massive, searchable, compilable, multifunctional database.

While the New York Times and other publications surely believe that giving up star ratings would be a commercial blunder, the reality may be that the star ratings are exactly what is allowing mainstream restaurant reviews to become irrelevant.

But no matter, stars are here to stay. We can safely say "they're a bad idea, but we're stuck with them" and move on to the question of how to have them be as useful as possible. This is where those who award stars have traditionally fallen down on the job. The New York Times and others should at least try to explain their star systems better, rather than simply issue smug and murky verdicts that are less useful than they can be.

Edited October 30, 2009 by Fat Guy
I see Poniewozik cites me in that piece, so of course he's right (log)

[Sneakeater]

The Times doesn't give ratings to shows or to concerts or to operas. It's an insult to gastronomy that they give stars to restaurants.

The proof that this is false comes from the chefs and restauranteurs themselves. Did you hear Daniel Boulud's reaction when he got the third Michelin star? He was floating on air. If he felt insulted, it was a marvellous acting job to pretend the opposite. I do realize that a few restauranteurs have given up their Michelin stars, but this is comparatively rare. Nearly all chefs profess themselves delighted when they receive a high star rating.

Obviously the chefs who don't get the rating they expect are unhappy, but any meaningful system needs to disappoint some people. Even then, I seldom hear chefs and restauranteurs saying that they wish the stars would go away; they just wish they had more of them.

That makes no sense at all.

I don't know what Chef Boulud thinks, but it would be entirely possible to find the idea of numerical ratings insulting as a concept, but still be EXTREMELY happy to get a high one -- especially when it is well known what a positive effect it has on your business.

[Sneakeater]

Poniewozik is right. So I guess FG must have been.

[Fat Guy]

Whether or not chefs are for or against star ratings is irrelevant. They don't get to decide what's an insult to gastronomy. Most of them aren't even well-suited to making determinations like that. Quantitative ratings for art are insulting to art. Whether or not the artists agree, or know enough to agree or disagree, doesn't affect that determination.

[Sneakeater]

Thanks. I wanted to write something like that, but you said it better anyway.

Edited October 30, 2009 by Sneakeater (log)

[oakapple]

Whether or not chefs are for or against star ratings is irrelevant. They don't get to decide what's an insult to gastronomy. Most of them aren't even well-suited to making determinations like that. Quantitative ratings for art are insulting to art. Whether or not the artists agree, or know enough to agree or disagree, doesn't affect that determination.

Sure it does. I should think that if numerical/star ratings are an insult, the feelings of those to whom the offense is purportedly directed ought to count for something. If there aren't very many of them saying, "I am insulted by this," then maybe it just isn't that insulting.

[oakapple]

If anything, Zagat tends to give casual and ethnic restaurants higher ratings than they truly warrant, not the reverse.

I don't have the current Zagat guide, but by looking at the top 25 on their website, Di Fara is the only one. 25-50 has Sripraphai. Out of 50 top restaurants in the city, only 2 are less than $20 a head.

But doesn't that strike you as intuitively correct? I mean, it would be passing strange if it were routinely possible to produce food at $20 a head that is just as good as $100 a head. There are exceptions, but generally you get what you pay for.

I realize that, for a variety of reasons, the $20 meal may be the one we want on some occasions, or maybe all occasions. The Zagat guide allows you to filter on price, and you can select the highest-rated places at whatever level you are looking for. But to pretend they are objectively as good as those that cost more is, in most cases, nonsense.

[David A. Goldfarb]

Whether or not chefs are for or against star ratings is irrelevant. They don't get to decide what's an insult to gastronomy. Most of them aren't even well-suited to making determinations like that. Quantitative ratings for art are insulting to art. Whether or not the artists agree, or know enough to agree or disagree, doesn't affect that determination.

Sure it does. I should think that if numerical/star ratings are an insult, the feelings of those to whom the offense is purportedly directed ought to count for something. If there aren't very many of them saying, "I am insulted by this," then maybe it just isn't that insulting.

An insult to restaurateurs is quite a different thing from an insult to gastronomy as a creative activity. An individual chef or restaurateur may or may not be offended by all kinds of things, but a larger question is whether numeric rating systems reduce the nuance and complexity of aesthetic experience to something meaningless, as I believe they do.

Edited October 30, 2009 by David A. Goldfarb (log)