Zagat 2005

[gmi3804]

Zagat raters must rate on a scale of 1 to 3, three being the highest. It's unclear how raters are supposed to award those points, but they're certainly not adhering to Michelin standards. It's not surprising that Per Se would get mostly threes for food, service, and decor if 1-3 were represented in the broad classifications of, respecitvely, "good," "better," and "best," in comparison with other restaurants around town.

[Todd36]

Fewer than 100 in a city of 7 million strikes me as very statistically insignificant.

←

Without knowing more about the raw data Zagat uses, none of us are in a position to know whether 100 votes is statistically significant or not. Don't forget that current national political polls are based on a sample size of 1200 or fewer people, and are sampling a much more varied population I think than people who go to Per Se or Nobi.

[oakapple]

Zagat raters must rate on a scale of 1 to 3, three being the highest.  It's unclear how raters are supposed to award those points, but they're certainly not adhering to Michelin standards.  It's not surprising that Per Se would get mostly threes for food, service, and decor if 1-3 were represented in the broad classifications of, respecitvely, "good," "better," and "best," in comparison with other restaurants around town.

←

Actually, it's quite clear how the raters assign the points: they do whatever they please. It's definitely not the Michelin system, and one could write a book about the methodological problems. What's amazing is that, despite these problems, the guide is generally dependable. There are some mistakes, but guides written by professional critics have mistakes too.

And remember, all polling—even if it follows the most scrupulously correct statistical methodology—is subject to sampling error. I would estimate that, 90-95% of the time, a Zagat rating is within +/- 2 points of its "correct" rating. That isn't too bad, and there's probably no methodology that would do significantly better.

[Mulcahy]

Without knowing more about the raw data Zagat uses, none of us are in a position to know whether 100 votes is statistically significant or not.  Don't forget that current national political polls are based on a sample size of 1200 or fewer people, and are sampling a much more varied population I think than people who go to Per Se or Nobi.

←

This is a fair point. But I believe (and any statisticians out there can correct me) that the national political polls are conducted by polling a specifically selected group of people. Zagats is the reverse, it's a self-selecting group of people who volunteer for the poll. And, unlike the nationwide political polls, Zagats often doesn't know who their poll-takers are. It's an inherent difference.

Also, because Zagats voters vote "1,2, or 3" and then those numbers are totalled and divided by the number of votes to reach the final rating then multiplied by ten (someone please correct me if I am getting this wrong) a smaller sample size might cause a disproportionately high or low rating.

[R Washburn]

All the numbers really mean is how consistently the meal met the diner's expectations. Looking at my 2003 votes I gave 3's in food to the following places: ADNY, Jeans-Georges, Daniel, Cafe Boulud, DB, Bouley, Blue Hill, Fairway, and Old Homestead.

I felt that my experiences at these places met with the voting standards:

3 = excellent

2 = very good

1 = good

0 = fair-poor

That doesn't mean that i think that Fairway is in the same league as ADNY.

[oakapple]

This is a fair point.  But I believe (and any statisticians out there can correct me) that the national political polls are conducted by polling a specifically selected group of people.

Not exactly. National political polls are random samples. However, the reputable pollsters normalize their samples before reporting a result. For example, if they reach a higher percentage of housewives than usual (e.g., because a high percentage of men were watching the Yankee game last night, and didn't come to the phone), they adjust the result to reduce the influence of that subset of the sample. You are correct that the Zagat respondents are self-selected, and this introduces some biases.

Also, because Zagats voters vote "1,2, or 3" and then those numbers are totalled and divided by the number of votes to reach the final rating then multiplied by ten (someone please correct me if I am getting this wrong) a smaller sample size might cause a disproportionately high or low rating.

←

You are correct about the 1-2-3 system, but not about the conclusion you draw from it. Since Zagat voters vote 1-2-3, it turns out that 20 is an "average" rating in their system. I suspect that many Zagat readers don't understand this. Since the published ratings are on a 1-30 scale, people might assume that 15 is average, which it isn't. Once you understand that 20 is average, the ratings start to make pretty good sense.

The 1-2-3 system doesn't have anything to do with disproportionately high or low ratings in case of small sample sizes. Regardless of the sample size, the average restaurant is 20, rather than 15. With small sample sizes, there might be a danger of ballot-stuffing, although if you look at the actual guide it's hard to find many cases where this appears to have happened. (This does not mean that one necessarily agrees with the numbers in every case—only that ballot-stuffing isn't necessarily the reason for them.)

[gmi3804]

All ratings are relative to the category of food the particular place offers. In the Zagat guide, rewarding a three for food for a place like, say, Grimaldi's (their food rating is 26) means that they do the best possible job with the food they're offering. Yet Per Se will also presumably get a three for the same reason, yet the craft and skill required to earn that three is light years ahead of those of Grimaldi's accomplishments. Of course when decor and service is factored into the equation, Grimaldi's average drops way below that of Per Se. Still, it's always a bit funny to see Grimaldi's food rating the same as those of such diverse places as Craft, Picholine, Blue Hill, La Grenouille, The Four Seasons, Felidia, Aquavit, etc.

[Laksa]

A travesty would be Olive Garden #1, Ruby Tuesday #2, and TGI Friday's #3.

←

Word! There's no way Ruby Tuesday is better than TGIF. No freakin' way.

[jbonne]

the sample size issue cuts both ways: most reviews are composed from tiny representations of the overall dining populace in a given city. but it also is composed from diners who self-selected to dine at a given establishment. Zagat may have gotten just 100 ratings for Per Se, but how many people out of the dining universe in NYC have actually eaten at Per Se? given the proportionate universe of Zagat respondents to the overall dining populace, is that an unrepresentative sample? probably not, i'd argue.

but by that same system, somewhere like Nobu is going to get extra credit because it has a high number of diners to rate it -- especially as Zagat expands its survey universe to include a greater number of less-experienced foodies. to that extent, it *does* increasingly function as a democracy. i'm sure that when you broaden the dining universe from hard-core diners to the whole damn tri-state area, Nobu seems like a pretty good option. in which case, the hell with democracy: i'm not sure i want dining advice from people who'd also dig the Cheesecake Factory.

The 1-2-3 system doesn't have anything to do with disproportionately high or low ratings in case of small sample sizes. Regardless of the sample size, the average restaurant is 20, rather than 15.

in fact, i think the 1-3 system (it's actually a 0-3 system, and i've given zeroes before) is one of Zagat's biggest flaws. not only has it moved toward an average of 20 -- with the 20-23 range offering wildly divergent quality levels -- but it's another move toward the, um, dumbening of the system.

the moment i've eaten in a place, i know exactly whether it's a 24 or a 26: having to choose between a 2 and a 3 diminishes the validity of my input. i suppose Zagat might reply that my comments are really what counts, not the score -- but that's also what the 100-point wine reviewers say.

[Mulcahy]

Not exactly. National political polls are random samples. However, the reputable pollsters normalize their samples before reporting a result. For example, if they reach a higher percentage of housewives than usual (e.g., because a high percentage of men were watching the Yankee game last night, and didn't come to the phone), they adjust the result to reduce the influence of that subset of the sample. You are correct that the Zagat respondents are self-selected, and this introduces some biases.

You are correct about the 1-2-3 system, but not about the conclusion you draw from it. Since Zagat voters vote 1-2-3, it turns out that 20 is an "average" rating in their system. I suspect that many Zagat readers don't understand this. Since the published ratings are on a 1-30 scale, people might assume that 15 is average, which it isn't. Once you understand that 20 is average, the ratings start to make pretty good sense.

←

Thank you for the clarification about the national political polls. I, by the way, happen to agree with you that Zagat is usually right +/- 2 points.

[bloviatrix]

the sample size issue cuts both ways: most reviews are composed from tiny representations of the overall dining populace in a given city. but it also is composed from diners who self-selected to dine at a given establishment. Zagat may have gotten just 100 ratings for Per Se, but how many people out of the dining universe in NYC have actually eaten at Per Se?  given the proportionate universe of Zagat respondents to the overall dining populace, is that an unrepresentative sample?  probably not, i'd argue.

←

It's not only how many people have actually eaten at Per Se, but what percentage of those diners actually complete the Zagat survey?

[jbonne]

It's not only how many people have actually eaten at Per Se, but what percentage of those diners actually complete the Zagat survey?

←

as well as the inverse: how many diners who complete the Zagat survey have managed to eat at Per Se?

[nypork]

Zagat's-a trickle down economy for jerks-I'm a newly appointed chef at a restaurant in Brooklyn that has had a downward spiral in Zagats-23-20-19 over the last three issues-today I get a phone call and the caller starts lauding me on the achievement of gaining a 19 in the book-my first instinct is to say"sucker" but I'm looking for people to enjoy my food and let him ramble on-he reads the entry and all-which was semi-flattering but I knew the history-so I go thanks alot-can I get you a reservation-he stammers and goes well I'm form comapny so and so and we create the plaques with the zagat listing. We'll while I hope to bring a better rating to this place next year-I politely declined, but then it became an aggressive sale to the point of rude, so I hung up-I'll keep the one that says 23 in the window and hope for the best. Moral of the story, I don't know-I t was just my Zagat story and I'm drunk rightnow. Woohoo!!

[Dryden]

Weirdest. Post. Ever.

[rich]

I may be suffering from brain freeze, so forgive me in advance. Years ago (mid 80's to early 90's), I submitted to the Zagat survey regularly. I thought it was a 0-30 system, not a 0-3 system. Has this changed over the years or is the gray matter suffering from dillusional malfunction?

[Pan]

Rich, I think you're probably misremembering. I filled out surveys for a few years, and it was always a 0-3 rating system. I think it's pretty inconceivable that they've ever allowed surveyors to use a 0-30 rating system. Think about it: What instructions would they give for each whole number in the rating?

[menton1]

Saw the 2005 Zagat on someone's desk last week... The absolutely funniest listing was Starbucks. (Not even all the locations). Now there's a review we needed!!

[phaelon56]

Fewer than 100 in a city of 7 million strikes me as very statistically insignificant.

I think we can safely assume that a tiny percentage of this city of 7 million participates in the Zagat survey and an even smaller number eat at higher end places such as Per Se. Factor in their low number of daily covers and even after being open another year there will likely be a fairly small number of respondents for Per Se. it will still be interesitng to see where it falls in the rankings.

I'm truly indifferent to the whole thing but do find some of the comments (in the guide vs this thread) to make for amusing reading.

[oakapple]

This site has an article with some interesting statistics from the 2005 Zagat guide. Among the highlights:

— 2004 was a great year for the NY restaurant industry. Among restaurants Zagat tracks, there were 226 openings (to 174 in 2003), and only 93 closings.

— 45% of those surveyed say they're spending more money on dining out than the year before.

— The average check in New York is $37.45 per person, which is only 1% higher than the year before. It's well above the national average of $31.51, but well below Paris ($62), London ($64) or Tokyo ($71).

— Steakhouses are most definitely in. Peter Luger and Sparks, both established favorites on the NY scene, saw significant jumps in popularity (9th to 6th and 45th to 26th respectively), while the Top Ten Newcomers' list included both Wolfgang's and BLT Steak. (Anecdotal evidence is that both restaurants remain extremely popular. Compass also re-invented itself as a steakhouse, although too late for the guide.)

— Only 7% of those surveyed said they make reservations on-line.

— Per Se was the first restaurant in the history of the New York guide to earn a 29 in all three categories (food, décor, service).

— Décor is in. Three new restaurants vaulted into the top 10 for décor: Asiate, Matsuri, and Spice Market. Megu might well have made that list if it had opened earlier in the year.

— Zagat says there are only four restaurants left in New York that require a jacket and tie: Alain Ducasse, La Grenouille, Rainbow Room and 21 Club.

You can crow all you want about methodological faults in Zagat, but I think there is some validity to these trends.

Edited January 14, 2005 by oakapple (log)