Bruni and Beyond: Reviewing (2008)

[Nathan]

nah. music and theater critics note performance deficiencies all the time in their reviews. performance deficiencies that can well be just "off nights". and they make these judgments off of one performance (visit)!

it looks to me like the real issue is the very nature of reviewing.

[Fat Guy]

They note performance deficiencies that occur under optimal circumstances: opening night, with the whole cast there (or the first-string lead singer or whatever), operating with the knowledge that all the critics are in the house (because of course the critics are comped), viewed from excellent seats, etc. In other words the critics make a conscious decision to review performances at their best. In addition, performance deficiencies are mostly noted in reviews of performances of repertoire classics where performance is the relevant point of comparison. New material is more often evaluated for its underlying content.

But yes, it's all about the nature of reviewing and also the nature of what's being reviewed. If we think restaurant reviews are a species of consumer reporting and restaurants are consumer goods and services then that argues for the undercover, consistency-oriented approach (though the approach is still mathematically nonsensical and is better served by Zagat-type mechanisms anyway). If we think restaurant reviews are arts criticism and chefs are creative artists (or members of some related species) then we should be focusing on content.

[oakapple]

Fat Guy, I entirely understand your math. I think Bruni understands it too.

I am just asking why you think service is any different than food. All restaurants, even four-star restuarants, have off-nights in the kitchen, and they botch some plates even on their better nights. If, by dumb luck, some of those plates are headed for Bruni's table, it's liable to to be mentioned in the review.

Bruni often signals that he realizes the sample is unscientific. He'll say, "The pork chop was dry both times I had it," or "Fish consistently came out over-seasoned." He seems to mention service glitches only when there's a pattern that he feels is something beyond random chance.

[Sneakeater]

Some of this is beside the point, though, because whatever any of us might think, the Times pretty clearly views restaurant reviewing as a species of consumer reporting rather than arts criticism.

[Fat Guy]

But what Bruni (or you, or I, or any normal human) just knows is more than mere random chance can possibly be does not in two or three visits rise to a mathematical level that would actually distinguish it from randomness.

Sure, all this reasoning applies to food too. Which is why I go farther than most (or perhaps anyone) in opposing anonymity. I think restaurants should be given the chance, as in the performing arts, to bring their A games to a critic. In other words, they should have a shot at preparing every menu item up to spec. Yes, yes, I know there could be some shenanigans. But they happen now too, and there are plenty of ways to guard against them. But in the end the question is should we be reviewing restaurants at their best or trying to catch them at their worst?

[Nathan]

neither

[Sneakeater]

Ditto.

[Fat Guy]

Some of this is beside the point, though, because whatever any of us might think, the Times pretty clearly views restaurant reviewing as a species of consumer reporting rather than arts criticism.

←

I agree with that statement, but I think it has to be viewed in a larger context. We are in the middle of a creative revolution in gastronomy that is transforming the field from a craft focused on repetition to a creative art-form. The Times should try to get ahead of the curve, not be dragged behind it.

Were I in charge over there -- which I will never be -- I'd try to hire someone like Colman Andrews as the next critic. Someone who has a sweeping knowledge of nearly all aspects of cuisine, a serious historical perspective, and a strong grasp of the craft-to-art progression that's occurring not only at the high end but also at the Momofuku level. And I'd tell him only to review restaurants that are on the art end of the spectrum. Let some other writer -- one with more of a consumer orientation (perhaps a reporter like Frank Bruni or William Grimes) -- review the steakhouses and such.

[Fat Guy]

neither

←

So you must think option three is that reviewers should try to evaluate some sort of average consumer experience. But I think it's already been established that they can't.

[oakapple]

I agree with that statement, but I think it has to be viewed in a larger context. We are in the middle of a creative revolution in gastronomy that is transforming the field from a craft focused on repetition to a creative art-form. The Times should try to get ahead of the curve, not be dragged behind it.

I agree that The Times has largely failed at this. Bruni simply doesn't have the background—and it appears he never will—to appreciate current trends in any kind of historical context. (By the way, I don't think I have the background either; then again, I don't sell my reviews as a professional product.)

Were I in charge over there -- which I will never be -- I'd try to hire someone like Colman Andrews as the next critic... And I'd tell him only to review restaurants that are on the art end of the spectrum.

←

That might be going a bit too far. I'm not sure there are enough restaurants on the "art" end of the spectrum to fill 52 reviews a year. But you've been saying for a long time that the Times needs a critic for the "middle" of the spectrum (something between fine dining and $25&U), and I agree.

So you must think option three is that reviewers should try to evaluate some sort of average consumer experience. But I think it's already been established that they can't.

←

I'm not so sure we've "established" that. For a critic that's (supposedly) recognized up to 80% of the time, he's pretty good at sussing out service issues and inconsistency in the kitchen. It's actually not surprising that he often gets this right, since it's the main part of the job, as he sees it.

[Nathan]

neither

←

So you must think option three is that reviewers should try to evaluate some sort of average consumer experience. But I think it's already been established that they can't.

←

I want their best effort.

[Leonard Kim]

If we take FG's service failure rates at face value for the moment, then I think even isolated incidents of bad service are more than meaningless.

I've written before that one characteristic of Bruni's three star reviews is that he has no (or trivial) complaints about the food. I haven't checked to see whether the same is true for service, but it makes sense that's approximately true.

Given a failure rate of 1% for a 3/4 star restaurant (FG's 50 out of 5000 tables), if Bruni or you or me take four of those tables (assuming that's typically what goes into a NY Times review), we would not expect *any* service failures. The probability of having even one is about 4%, and there is basically no chance of having two bad tables. (I'm just using a binomial distribution here.)

Given a failure rate of 10% for a 1 star restaurant (500 out of 5000) on the other hand, sure, in four visits you still don't expect any service failures, but nor is it unexpected if you do have a bad night -- there's a 34% chance, better than one in three, if you go four times, at least one will be bad.

So if I'm reviewing a restaurant, and I go four times, and one of them is bad, sure there's a chance I'm at a three/four star place and am one of the unlucky 4%, which means, if I've reviewed thirty or so of these places, I've probably underrated one unfairly because of the bad service. But it's far more likely that the bad incident is meaningful in helping peg the service level.

Obviously, the numbers depend on FG's estimates being in the ballpark. Actually, I think the best way to do this is for you guys to make up a hypothetical restaurant with an established service level and "consistency." For example, take a three-star restaurant -- on a given night what's the probability of getting four-star service? two-star service? one star service? With those numbers, it should be fairly easy to calculate the probability of guessing the wrong rating on the basis of four visits.

[Nathan]

now assume that Bruni is recognized three out of four times at any given three star restaurant. the odds of a rare service failure on the one visit that he's not recognized are even lower...unless that restaurant really does have a high rate of service failures. in which case it's not a restaurant with good service.

[Fat Guy]

Nathan you have it exactly backwards: the fact that Frank Bruni is recognized 80% of the time give or take -- and that multiple anonymous visits to a given restaurant are, for him, a rarity -- is why his occasional one anonymous visit does so little to distinguish itself mathematically from randomness.

[Nathan]

your point is that it's effectively a random sample. right.

the point is that if a restaurant really does have outstanding service....then the odds of them screwing up on Bruni's table on the rare night that they don't recognize him are very low.

therefore, if a restaurant does screw up, the odds are quite high that that restaurant frequently screws up.

[Fat Guy]

Obviously, the numbers depend on FG's estimates being in the ballpark.  Actually, I think the best way to do this is for you guys to make up a hypothetical restaurant with an established service level and "consistency."  For example, take a three-star restaurant -- on a given night what's the probability of getting four-star service?  two-star service?  one star service?  With those numbers, it should be fairly easy to calculate the probability of guessing the wrong rating on the basis of four visits.

←

Actually I don't think it would make sense to look at the "probability of getting four-star service" etc. You never get four-star service at a middle-market restaurant, because four-star service, at least traditionally, is a whole style of service that depends on certain human resources, structures, procedures, etc., that can't just happen idiopathically. You'll notice that in my hypothetical I referred to 3/4-star service. That's because the failure rate (or whatever model you want to use) at the high-3 and 4-star level is going to be very, very close -- meaning you'd need a very large number of visits to expose the difference. You couldn't do it in 4 visits and you certainly can't do it in 1 (which would be more normal for Bruni). That's why instances like the Ducasse fiasco are random "gotchas" rather than statistically meaningful occurrences. But this all backs into an interesting point: you can go into a restaurant -- anonymously or not -- and very quickly identify that it's trying to offer four-star service or one-star service. Whether or not it succeeds is the big open question. And you can't answer that question unless you get a significant number of repeat, anonymous visits under your belt -- which so rarely happens that it's news when it does.

[Fat Guy]

the point is that if a restaurant really does have outstanding service....then the odds of them screwing up on Bruni's table on the rare night that they don't recognize him are very low.

therefore, if a restaurant does screw up, the odds are quite high that that restaurant frequently screws up.

←

I'm no statistician but my limited knowledge of the subject says you're wrong about that. Happy to hear from someone who knows more.

[Nathan]

the point is that if a restaurant really does have outstanding service....then the odds of them screwing up on Bruni's table on the rare night that they don't recognize him are very low.

therefore, if a restaurant does screw up, the odds are quite high that that restaurant frequently screws up.

←

I'm no statistician but my limited knowledge of the subject says you're wrong about that. Happy to hear from someone who knows more.

←

assume that a 3/4 star restaurant screws up 2% of the time.

assume that a 1/2 star restaurant screws up 25% of the time.

if I visit a restaurant once and they don't screw up...it doesn't tell me much.

if I visit a restaurant five times and they don't screw up...the odds of it being a 3/4 star restaurant are relatively high.

if I visit a restaurant once and they screw up....the odds of that restaurant being 1/2 are quite high. a 3/4 would have to quite unlucky to screw up on my sole visit.

if I visit a restaurant seven times and they screw up once, the odds of that restaurant being either a 3/4 or a 1/2 are relatively even. (there are ways to crunch this exactly...maybe a stats person here will do it)

[slkinsey]

the point is that if a restaurant really does have outstanding service....then the odds of them screwing up on Bruni's table on the rare night that they don't recognize him are very low.

therefore, if a restaurant does screw up, the odds are quite high that that restaurant frequently screws up.

I'm no statistician but my limited knowledge of the subject says you're wrong about that. Happy to hear from someone who knows more.

Let's assume that a Screw Up Rate ("SUR") of 1% represents outstanding service. I'm basing this on Steven's postulated 50 out of 5000 tables screwed up per month. We have no idea whether this is accurate, but it's a useful jumping-off point.

This means that every time Bruni sits down in a given restaurant, he would have a 1% chance of experiencing an evening of "unpreventable" poor service. This, as Nathan points out, is a low rate. But it is not zero. It also suggests that, for every 100 times Bruni sits down in a restaurant with a 1% SUR, he can as a general rule of thumb expect one evening of "unpreventable" poor service. However, just as it is possible for a simple coin flip to come up heads 100 times in a row, it's also possible for Bruni to hit either the jackpot or snake-eyes any number of times in a row. With a sample size of 4, it's entirely possible for Bruni to experience a SUR of 50% at a restaurant where the overall SUR is 1%. Indeed, given the number of restaurant meals consumed by Bruni, this is almost certain to happen.

Edited June 17, 2008 by slkinsey (log)

[Nathan]

With a sample size of 4, it's entirely possible for Bruni to experience a SUR of 50% at a restaurant where the overall SUR is 1%.  Indeed, given the number of restaurant meals consumed by Bruni, this is almost certain to happen.

←

except that the contention was that he's usually recognized at high end restaurants. so, you can't normally expect him to get away with more than one anonymous visit. in which case, his SUR at a given restaurant is 1%.

edit:

now, yes, it's reasonably likely that about once every two years he experiences an SU at a restaurant with an SUR of 1%. so?

the benefit of the doubt should go to the consumer not the restaurant. so even if one restaurant every two years is wrongly downgraded based upon service, he'll have pegged multiple others considerably more accurately than if he dined completely non-anonymously.

Edited June 17, 2008 by Nathan (log)

[slkinsey]

Nathan, that makes no difference. All it says is that there is a low percent chance that he'll experience a screwup event when he sits down. That defines his experience, but doesn't allow us to extend his experience to the restaurant at large, because the sample size is way too small (never mind the fact that we all understand that even a restaurant with great service will still occasionally give sub-par service to a valued customer or recognized critic).

[Nathan]

Nathan, that makes no difference.  All it says is that there is a low percent chance that he'll experience a screwup event when he sits down.  That defines his experience, but doesn't allow us to extend his experience to the restaurant at large, because the sample size is way too small (never mind the fact that we all understand that even a restaurant with great service will still occasionally give sub-par service to a valued customer or recognized critic).

←

see my edit.

as for sample size, that's purely a function of confidence level and allowable sampling error.

in this matter, neither needs to be strong for us to have meaningful results.

[Nathan]

and as I noted above,

a visit with poor service is vastly more significant than a visit with good service.

[oakapple]

Nathan, that makes no difference.  All it says is that there is a low percent chance that he'll experience a screwup event when he sits down.  That defines his experience, but doesn't allow us to extend his experience to the restaurant at large, because the sample size is way too small (never mind the fact that we all understand that even a restaurant with great service will still occasionally give sub-par service to a valued customer or recognized critic).

←

The trouble is, I understand what you're against, but I don't see what you're for.

Do you envision the Times saying one day, "From now on, we no longer report on service, because we've decided we can't."

Or do you envision the Times deliberately sending Bruni out to be feted as if he were the King of England, then writing reviews that disingenuously suggest that every customer can expect a similar experience?

Or do you envision a series of reviews like Craig Claiborne's famous $4,000 dinner, which no one claimed could be reproduced by the ordinary consumer, but which he reviewed mainly for the readers' entertainment?

[Nathan]

and what I'm saying is that one anonymous visit a restaurant, statistically, will very rarely screw over a deserving restaurant.

the odds will still be in the restaurant's favor. just not quite as skewed as they are when all of the visits are non-anonymous.