Bruni and Beyond: NYC Reviewing (2007)

[oakapple]

Restaurant rating systems, be they insipid or inspired, are inevitable.  The least one can ask is that there not be inequity in their inaneness.

←

But had Frank Bruni awarded the 2, 3, or 4 stars you believe Katz's deserved, it would have solved none of the problems you complain of. If anything, it would have exacerbated the inequity, since there are hundreds of extant NYT ratings awarded according to the "old rules." Ironically, you are making your complaint at the very point that Katz's received the highest compliment it could have received under the existing system. What's more, a restaurant like Katz's isn't very sensitive to the rating anyway.

[plattetude]

To add to the Zagat tangent of this discussion, I don't think Zagat's suggested "very good to excellent" range of 20-25 holds much water. Sure, it's a 30-point scale, but that's derived from individuals ranking on a 3 (okay, 4, counting zero) point scale. Someone can't say "gee, that was a little better than a 2, so I'll give it a 2.2." By default, then, you're stuck with a system where you can rate it 0 if you hate it, 1 if you tolerate it, 2 if you like it, and 3 if you love it. Realistically, a 2 then becomes the default rating if something wasn't glaringly bad. To me, that says there's gonna be a boatload of 2's, equating to 20 "points" for places that are ostensibly just passably good.

Christopher

[Nathan]

To add to the Zagat tangent of this discussion, I don't think Zagat's suggested "very good to excellent" range of 20-25 holds much water.  Sure, it's a 30-point scale, but that's derived from individuals ranking on a 3 (okay, 4, counting zero) point scale.  Someone can't say "gee, that was a little better than a 2, so I'll give it a 2.2."  By default, then, you're stuck with a system where you can rate it 0 if you hate it, 1 if you tolerate it, 2 if you like it, and 3 if you love it.  Realistically, a 2 then becomes the default rating if something wasn't glaringly bad.  To me, that says there's gonna be a boatload of 2's, equating to 20 "points" for places that are ostensibly just passably good.

Christopher

←

it gets worse.

zagat's rules are:

poor -- 0

good -- 1

very good -- 2

excellent -- 3

essentially, the default rating is "good" or "very good"

[oakapple]

On top of that, most diners spent the bulk of their dining dollars at restaurants they like. Unlike Frank Bruni, most diners feel no obligation to spend time at bad restaurants as a journalistic responsibility.

So, Zagat voters are likely to feel pretty positive about the restaurants they're voting on. It's therefore no surprise that a Zagat food rating below 15 is extremely rare.

Edited May 31, 2007 by oakapple (log)

[JohnL]

Any rating system can be nit picked endlessly (never to death though).

They either work or they don't and believe me if they don't work then they are useless.

There is a key reason that Zagat Guides (for all their faults) are so successful.

Their ratings are pretty much on. By and large, one gets a reasonably accurate snap shot of what a restaurant offers in terms of food, decor and service.

Any consumer needs to calibrate any ratings or reviews based upon their understanding of the who and how the ratings are arrived at and their own criteria.

Actually, the Zagat (and others) system offers some leeway. People know what 20-25 "means" . I would argue that most people would have no problem placing most of the restaurants that receive say 20-25 points in their own "good" to "excellent" range. Thus Zagat's succeeds.

If, on the other hand, Zagat's awards 20-25 points and a majority of consumers real experiences would be 15-20, then there is a problem--Zagat loses credibility and people stop buying the guides.

interestingly in the Katz's example that 23 (again this from the 2006 guide) seems to be pretty much in synch with Bruni's prose and pretty damn accurate (lying in the good to excellent transitional range for food).

where Zagat falls a bit short, IMOP, there may be one or two outstanding dishes at a given restaurant amid a sea of just ok or good ones and Zagat's sometimes does not convey this with specifics).

I would agree that a very good critic and a comprehensive review can be more valuable to a serious diner. However, these are fewer and farther between these days. There is a lot of information about restaurants on the net for those who are serious enough to seek it out.

I believe that Zagat's rating system is pretty accurate and easy to calibrate one's tastes with. I also believe that the Times system is difficult to comprehend and calibrate to one's individual tastes.

In the end, a rating system is shorthand. It is up to the consumer as to how to use it and what importance it has. At a very basic level it merely states that the methodology used to apply it indicates one place is better than another.

[JohnL]

Just to add:

Ratings systems aside.

If someone like Johnny Apple wrote that X was the best corned beef he had ever tasted" my response would be--"damn, I gotta try that!" That's de facto a Times Four star rating!

I just don't have this response much anymore!

There isn't a single place touted on the pages of the Times recently where I got excited and thought--"I gotta try that" or "I need to get a reservation."

[Nathan]

On top of that, most diners spent the bulk of their dining dollars at restaurants they like. Unlike Frank Bruni, most diners feel no obligation to spend time at bad restaurants as a journalistic responsibility.

So, Zagat voters are likely to feel pretty positive about the restaurants they're voting on. It's therefore no surprise that a Zagat food rating below 15 is extremely rare.

←

well...yeah! leaving aside the mechanical issues...the Zagat system is inherently and completely compromised by the very simple fact that its voters are a self-selected group.

[JohnL]

There is no perfect system for reviewing or rating anything.

I would posit that every "system" is compromised to one degree or another.

As for Zagat's, only repeat diners would form this self selecting group. Many people who visit a place once and rating that place with a low score would certainly "balance" things out a bit.

Also Zagat probably takes steps to eliminate problems and ensure some integrity in the system.

In the end, if Zagat did not "deliver" reasonable perceived accuracy, they would quickly be out of business.

[Nathan]

Any rating system can be nit picked endlessly (never to death though).

They either work or they don't and believe me if they don't work then they are useless.

There is a key reason that Zagat Guides (for all their faults) are so successful.

Their ratings are pretty much on. By and large, one gets a reasonably accurate snap shot of what a restaurant offers in terms of food, decor and service.

I 100% disagree in the strongest possible terms.

If, on the other hand, Zagat's awards 20-25 points and a majority of consumers real experiences would be 15-20, then there is a problem--Zagat loses credibility and people stop buying the guides.

←

eh. your underlying assumption for this entire argument is hopelessly flawed. regardless of whether you mean "the majority of consumers" or "the majority of Zagat voters"....the problem with either formulation is that both of these groups are almost complete idiots (the simple fact is that the set of Zagat voters (at least in NY) is synonymous with the set of well-to-do people who dine out a lot at the same restaurants over and over again).

furthermore, a Zagat rating means nothing to me. it tells me nothing. I'll take the Times' star ratings, the Michelin Guide...etc. over Zagat in a heartbeat. as flawed as either of those rating systems may be...they're still much more accurate than Zagat. as for Bruni, I know his tastes and biases...as do we all. so, that makes his reviews of places I haven't been to quite useful...as opposed to Zagat where it means nothing.

example: I don't know what Sri's Zagat rating is but I'm sure it's high. why? because the set of people willing to trek out to Woodside to eat really authentic Thai food is small and composed of people who will love it.

now...place Sri on the UWS and change its name so no one has heard of it. it'll probably get a much more mediocre rating (if it makes the guide at all)....much too spicy and "weird" for the average Zagat voter.

edit: in other words, the accurate Zagat ratings are anomalies....they have extenuating circumstances.

Mimi Sheraton did a nice takedown on Zagat awhile back here:

http://www.foodandwine.com/invoke.cfm?obje...2620002B3309983

Edited May 31, 2007 by Nathan (log)

[Nathan]

There is no perfect system for reviewing or rating anything.

I would posit that every "system" is compromised to one degree or another.

As for Zagat's, only repeat diners would form this self selecting group. Many people who visit a place once and rating that place with a low score would certainly "balance" things out a bit.

Also Zagat probably takes steps to eliminate problems and ensure some integrity in the system.

In the end, if Zagat did not "deliver" reasonable perceived  accuracy, they would quickly be out of business.

←

There's a difference between being entirely compromises and partially compromised. The second still has utility...the first is only fit for compost.

by self-selecting, I mean the entire set of Zagat voters are self-selected. the Zagat concept only works on any level at all if they simply poll a representative cross-section of the population.

furthermore, Zagat is inherently biased because most Zagat voters don't keep a running total of every restaurant they go to with a score. they fill out all of their ratings at once...and the restaurants they mainly remember are the ones they've been to more than once (or that anniversary meal at Union Square Cafe).

as for Zagat delivering perceived accuracy....yes, its perceived as accurate by the majority of diners...who also happen to be idiots and without taste.

I'll take Bruni or even Randall Lane over the average Zagat voter...

[JohnL]

Any rating system can be nit picked endlessly (never to death though).

They either work or they don't and believe me if they don't work then they are useless.

There is a key reason that Zagat Guides (for all their faults) are so successful.

Their ratings are pretty much on. By and large, one gets a reasonably accurate snap shot of what a restaurant offers in terms of food, decor and service.

I 100% disagree in the strongest possible terms.

If, on the other hand, Zagat's awards 20-25 points and a majority of consumers real experiences would be 15-20, then there is a problem--Zagat loses credibility and people stop buying the guides.

←

eh. your underlying assumption for this entire argument is hopelessly flawed. regardless of whether you mean "the majority of consumers" or "the majority of Zagat voters"....the problem with either formulation is that both of these groups are almost complete idiots (the simple fact is that the set of Zagat voters (at least in NY) is synonymous with the set of well-to-do people who dine out a lot at the same restaurants over and over again).

furthermore, a Zagat rating means nothing to me. it tells me nothing. I'll take the Times' star ratings, the Michelin Guide...etc. over Zagat in a heartbeat. as flawed as either of those rating systems may be...they're still much more accurate than Zagat. as for Bruni, I know his tastes and biases...as do we all. so, that makes his reviews of places I haven't been to quite useful...as opposed to Zagat where it means nothing.

example: I don't know what Sri's Zagat rating is but I'm sure it's high. why? because the set of people willing to trek out to Woodside to eat really authentic Thai food is small and composed of people who will love it.

now...place Sri on the UWS and change its name so no one has heard of it. it'll probably get a much more mediocre rating (if it makes the guide at all)....much too spicy and "weird" for the average Zagat voter.

edit: in other words, the accurate Zagat ratings are anomalies....they have extenuating circumstances.

Mimi Sheraton did a nice takedown on Zagat awhile back here:

http://www.foodandwine.com/invoke.cfm?obje...2620002B3309983

←

It is amusing that so many people basically spend a lot of time attacking Zagat's based upon audience or participant analysis which is based upon things they believe rather than any real facts. Either the people who participate in the surveys are suspect or worse ('idiots?"); or, the people who rely on the guides for information (ok there's overlap here however common sense says many more people buy these things that don't participate).

Or

It's the rating point system or it's a hypothetical case:

Yes "Sri" is rated highly and the comments noted are right on. So rather than conclude that the "idiots" got it right and the system worked, the critics say: "oh yeah, well if Sri was in a different location...." It's always something. (to quote Roseanne Rosannadanna!)

I am not trying to carry any water for the Zagat's--I have never participated in their surveys and I don't rush out to get the latest guides (I am still in possession of the 2006 New York edition).

I do not believe they are the be all and end all of restaurant reviewing. I agree they have their drawbacks.

But!

I would argue that for the most part, they get it right! Somehow, some way despite all their flaws......they get it right! maybe it's luck..who knows.

I would also argue that there is far from universal agreement on what Frank Bruni's standards/criteria are. Seems to me there is quite a bit of debate over just what four or two or whatever number of Times stars means exactly. (and not just here either).

All I am saying is give Zagat their due. Use em or don't use em. If there was something horribly wrong with them they would be out of business. (unless of course, everyone who buys these things is an idiot!--in which case let's let all the idiots have their fun and move on).

[oakapple]

The truth about Zagat is somewhere between what John and Nathan say.

The list of restaurants with Zagat ratings around 23-25 or higher is reasonably well correlated with various other lists of NYC's best restaurants. Whatever faults Zagat may have, at the top end their ratings are more-or-less correct. Yes, there are some anomalies, but there are anomalies in the Michelin guide too, and don't get me started with Frank Bruni's anomalies.

But once you get into the low twenties, the Zagat ratings are just an undifferentiated scrum — bearing in mind that 20-21 is average, and anything within two points of that is within the standard deviation. This is not because Zagat voters are idiots. It's because Zagat themselves have created a flawed system, and even with geniuses voting, you would get these results.

[JohnL]

You may be correct. I am not sure you are. The problem is, I do not believe that Zagat reveals specific numbers for a given restaurant or for a rating.

There is no "average" for them in terms of score. 10-15 they say is "fair to good" and "16-19 good to very good." The assumptions about what is average are just assumptions.

One needs to not only look at just their ratings or numbers but rather, the verbiage.

As well as all the rating numbers and cost.

Zagat's, obviously, does not simply regurgitate votes--there is some "editing" at play.

I keep going back to my belief that within certain parameters, they seem to get most restaurants correctly regardless of the specific rating.

I have never come across any egregious discrepancies. I also have not spent an inordinate amount of time looking for them. I do know that if they were not at least providing reasonably accurate information to a lot of people (of whatever intelligence) who dine out; they would not be successful.

It seems to me that most of their critics seem to take shots at their methodology, about which not that much is known--again, we have little idea how they process the numbers and select the quotes for the copy. Or, folks will disparage the participants or users. Anecdotal information is dangerous but, I know a lot of pretty savvy diners who utilize these guides.

Also assumed is that the people who use these things rely solely upon them. Many probably do but many also get dining information from other sources and process it with Zagats.

So, rather then delve into methodology with very little real information, or look at what I perceive to be the audience or participants and try to determine if some standard of deviation is at play--I simply look at the reviews and I just do not see any egregious problems with the information Zagats provides based on my own (admittedly limited) experience.

Will other reviews provide more information about Sri and better perspective--more in depth and even more accurate information? yes. But if one wants a fairly accurate snapshot of Sri (or most other places of all pretensions) then Zagat does a pretty good job!

[Nathan]

Yes "Sri" is rated highly and the comments noted are right on. So rather than conclude that the "idiots" got it right and the system worked, the critics say: "oh yeah, well if Sri was in a different location...." It's always something. (to quote Roseanne Rosannadanna!)

I'd say that the fact I was able to exactly predict the Sri entry without having seen it is telling. one could do that for every restaurant...and its comments (which are undifferentiated and useless).

All I am saying is give Zagat their due. Use em or don't use em. If there was something horribly wrong with them they would be out of business. (unless of course, everyone who buys these things is an idiot!--in which case let's let all the idiots have their fun and move on).

←

if there was something horribly wrong with Britney Spears as a musician she wouldn't sell any records.....

the problem is...taste is not a democratic attribute.

[Nathan]

The list of restaurants with Zagat ratings around 23-25 or higher is reasonably well correlated with various other lists of NYC's best restaurants. Whatever faults Zagat may have, at the top end their ratings are more-or-less correct. Yes, there are some anomalies, but there are anomalies in the Michelin guide too, and don't get me started with Frank Bruni's anomalies.

I almost agree. But the Grocery ranking (which was likely simply ballot stuffing by people in its neighborhood) and USC's top rating year after year are greater anomalies than anything at the top end by Bruni or Michelin...they'd have to give the highest ranking to A di la or something to reach that point.

But once you get into the low twenties, the Zagat ratings are just an undifferentiated scrum — bearing in mind that 20-21 is average, and anything within two points of that is within the standard deviation.

←

exactly. and that only accounts for most of the restaurants in the guide!

edit: and this is why Zagat is really useless...everyone knows what the top restaurants are!

Edited May 31, 2007 by Nathan (log)

[Nathan]

You may be correct. I am not sure you are. The problem is, I do not believe that Zagat reveals specific numbers for a given restaurant or for a rating.

huh?

Zagat's, obviously, does not simply regurgitate votes--there is some "editing" at play.

huh? they insist they simply average the votes! I believe them. the editing is of the comments.

It seems to me that most of their critics seem to take shots at their methodology, about which not that much is known--again, we have little idea how they process the numbers and select the quotes for the copy.

←

huh? we know everything about their methodology. its transparent. it's not Michelin. and it's not like people who used to be editors for them haven't talked since....a little googling will reveal a lot here...

[oakapple]

Yes, there are some anomalies, but there are anomalies in the Michelin guide too, and don't get me started with Frank Bruni's anomalies.

I almost agree. But the Grocery ranking (which was likely simply ballot stuffing by people in its neighborhood) and USC's top rating year after year are greater anomalies than anything at the top end by Bruni or Michelin...they'd have to give the highest ranking to A di la or something to reach that point.

Remember, USC is still carrying three stars from the New York Times, meaning there are only five restaurants rated higher on their scale (i.e., the five 4* restaurants). Dicta in Frank Bruni's Bar Room/11MP review suggest that USC is no longer worthy of its three stars, but for now it still has them.

But once you get into the low twenties, the Zagat ratings are just an undifferentiated scrum — bearing in mind that 20-21 is average, and anything within two points of that is within the standard deviation.

←

exactly. and that only accounts for most of the restaurants in the guide!

edit: and this is why Zagat is really useless...everyone knows what the top restaurants are!

←

You greatly exaggerate the number of people who track the NYC restaurant industry as closely as you and I do. "Everyone" doesn't know what the top restaurants are, unless Zagat tells them so. Edited May 31, 2007 by oakapple (log)

[JohnL]

The list of restaurants with Zagat ratings around 23-25 or higher is reasonably well correlated with various other lists of NYC's best restaurants. Whatever faults Zagat may have, at the top end their ratings are more-or-less correct. Yes, there are some anomalies, but there are anomalies in the Michelin guide too, and don't get me started with Frank Bruni's anomalies.

I almost agree. But the Grocery ranking (which was likely simply ballot stuffing by people in its neighborhood) and USC's top rating year after year are greater anomalies than anything at the top end by Bruni or Michelin...they'd have to give the highest ranking to A di la or something to reach that point.

But once you get into the low twenties, the Zagat ratings are just an undifferentiated scrum — bearing in mind that 20-21 is average, and anything within two points of that is within the standard deviation.

←

exactly. and that only accounts for most of the restaurants in the guide!

edit: and this is why Zagat is really useless...everyone knows what the top restaurants are!

←

If the restaurant ratings by Zagats reflect your own impressions then I'd say they are doing a pretty good job--by your own admission(and your own standards)!

Yes most of us know what the top restaurant are. I would gather that most people do not use these guides to simply find out what top restaurants are.

Zagats are clearly not attempting to be arbitors of taste. (Bruni and the Times are wrestling with that one, though I am not all that sure).

Rather, Zagat is more like a directory with many uses and purposes. I think a lot of their critics can't grasp that Zagat is really not about setting standards or establishing taste beyond reflecting where a chunk of the dining public is currently.

That's why I keep pointing up the fact that taken in context their scores and copy etc are fairly accurate--they should be--they have to be.

If they are not useful to some people --so be it! They are not useful not because they are inaccurate. that is all I am saying.

[Nathan]

You greatly exaggerate the number of people who track the NYC restaurant industry as closely as you and I do. "Everyone" doesn't know what the top restaurants are, unless Zagat tells them so.

←

the even more self-selected group of Zagat voters that dine at four-star restaurants which is a small portion of the self-selected group of Zagat voters in aggregate know who the top restaurants are.

[Nathan]

If the restaurant ratings by Zagats reflect your own impressions then I'd say they are doing a pretty good job--by your own admission(and your own standards)!

what I said was name a restaurant and I can tell you the Zagat rating and the comments without having seen the Zagat entry. that has nothing to do with my personal view of the restaurant.

I think a lot of their critics can't grasp that Zagat is really not about setting standards or establishing taste beyond reflecting where a chunk of the dining public is currently.

←

no. Zagat is a somewhat accurate survey of the views of a self-selected group of the dining public....no one disputes that. so?

the claim was that Zagat is "more accurate" than the Times or Michelin in some larger sense....not "Zagat is more accurate than the Times or Michelin in reflecting the views of Zagat voters".

what I don't understand is the foundational assumption that the majority of Zagat voters have taste.

[oakapple]

You greatly exaggerate the number of people who track the NYC restaurant industry as closely as you and I do. "Everyone" doesn't know what the top restaurants are, unless Zagat tells them so.

←

the even more self-selected group of Zagat voters that dine at four-star restaurants which is a small portion of the self-selected group of Zagat voters in aggregate know who the top restaurants are.

←

What I meant was that Zagat readers—not voters—rely on Zagat to inform them what the top restaurants are. By "top," I didn't merely mean four-star restaurants. From around 23-24 and up, the Zagat ratings are reasonably reliable, and that range that encompasses more than just the four-stars.

[Nathan]

What I meant was that Zagat readers—not voters—rely on Zagat to inform them what the top restaurants are. By "top," I didn't merely mean four-star restaurants. From around 23-24 and up, the Zagat ratings are reasonably reliable, and that range that encompasses more than just the four-stars.

←

what I meant was that the reason Zagat generally gets the top sort of right (explain Grocery!!!!) is because that's being determined by a pretty small group of voters who are generally gathering dining info from other sources...(I know some of this group...attorneys or bankers or execs who have a favorite restaurant (such as Aureole) that they eat at regularly and then they hit places like Jean Georges or Per Se once or twice a year. They read the Times reviews...they hear word of mouth when GR or the like opens. but are they foodies? would they trek out to Sri in a million years? heck no.)

Zagat doesn't get anything besides the top right because after the top it just becomes a blob of (primarily tasteless and uninformed) people voting for their favorite restaurants.

edit: to illustrate my Sri example, if someone has a guide handy...could they post the respective rankings for Joya, Sea (either branch), and Pam Real?

what are the odds that they all have similar rankings..or even that Joya and Sea are ranked higher than Pam Real?

(the point being...Joya and Sea suck, Pam Real is probably the best Thai in Manhattan)

Edited May 31, 2007 by Nathan (log)

[oakapple]

what I meant was that the reason Zagat generally gets the top sort of right (explain Grocery!!!!) is because that's being determined by a pretty small group of voters who are generally gathering dining info from other sources...

Okay, I see your point. I don't know whether that's true or not, but my only claim was that Zagat is fairly close to accurate at the 23+ level. I am not trying to figure out how it got that way. "Grocery" explains why Zagat is "fairly close to accurate," instead of "totally accurate."

edit:  to illustrate my Sri example, if someone has a guide handy...could they post the respective rankings for Joya, Sea (either branch), and Pam Real?

what are the odds that they all have similar rankings..or even that Joya and Sea are ranked higher than Pam Real?

(the point being...Joya and Sea suck, Pam Real is probably the best Thai in Manhattan)

←

The suck/best dichotomy on the Internet is so last-decade. You need to get over it.

But here are your answers...you were partially right: Joya: 25; Sea: 21; Pam: 24. I am too lazy to check other sources that rate Thai restaurants, to see if there is a rational basis for these results. If others rate Joya more-or-less comparably to Pam, then Zagat is rational—and that's all you can expect a guide to be.

[oakapple]

Gourmet's blog has this post about a recent event about the food criticism profession, with Ruth Reichl and Adam Platt as guests.

They suggested that being a restaurant critic's close friend is a better deal than being a restaurant critic, because the friend gets invited to many of the meals, with no oblgation to write about them afterwards.

But both Reichl and Platt believe that the friends should resist the temptation to become "quote feeding" machines—the critic is there to form his or her own opinion. Reichl has said that before, but it's a little surprising coming from Platt, because he does indeed quote his dining companions occasionally. Bruni, of course, does it all the time.

[Nathan]

what I meant was that the reason Zagat generally gets the top sort of right (explain Grocery!!!!) is because that's being determined by a pretty small group of voters who are generally gathering dining info from other sources...

Okay, I see your point. I don't know whether that's true or not, but my only claim was that Zagat is fairly close to accurate at the 23+ level. I am not trying to figure out how it got that way. "Grocery" explains why Zagat is "fairly close to accurate," instead of "totally accurate."

edit:  to illustrate my Sri example, if someone has a guide handy...could they post the respective rankings for Joya, Sea (either branch), and Pam Real?

what are the odds that they all have similar rankings..or even that Joya and Sea are ranked higher than Pam Real?

(the point being...Joya and Sea suck, Pam Real is probably the best Thai in Manhattan)

←

The suck/best dichotomy on the Internet is so last-decade. You need to get over it.

But here are your answers...you were partially right: Joya: 25; Sea: 21; Pam: 24. I am too lazy to check other sources that rate Thai restaurants, to see if there is a rational basis for these results. If others rate Joya more-or-less comparably to Pam, then Zagat is rational—and that's all you can expect a guide to be.

←

Joya is a Sea clone. it sucks. as to why it has that high a ranking...it's very popular with the Brooklyn Heights set.