242 replies to this topic

[oakapple]

Like FG, I think the Times would be able to hire more authoritative reviewers if it didn't care so much about anonymity.

If that is the reason why we have Frank Bruni and Adam Platt, then I would become a buyer of at least part of FG's argument: that is too steep a price to pay.

But there are plenty of extremely knowledgeable people following some version of the "Bruni rule," i.e., they don't adopt weird disguises, but they don't go out of their way to call attention to themselves. That still seems like the best system to me, for the reasons Nathan gave, a post or two upthread.

[Fat Guy]

But what you call "the Bruni rule" is not an accurate statement of New York Times procedure. The Times would never hire David Rosengarten, even though Rosengarten is a far superior critic to Bruni, because Rosengarten had a show on the Food Network and is widely recognized in New York restaurants. They wouldn't say "Well, it's okay because he'll just follow the Bruni rule."

[slkinsey]

4.  look at it this way, if he visits Per Se four times and manages to dine anonymously for just one of them...he now has quite a bit more data to work with.

"Quite a bit more data"? He has an arguably useless random sampling that, empirically, hasn't made his reviews any better than when he's recognized every time. "I wonder about Adour" hardly establishes anything to the contrary.

To be clear... What makes this a "useless random sampling" is that it's possible, for example, let's say that the critic is reviewing a restaurant with excellent service. But let's suppose that the restaurant's FOH staff has one of those rare "every so often" bad nights on the critic's first visit. Let's further say that they are back to their usual level of excellence for the critic's subsequent three visits, which also happen to be post-recognition visits. What is the quasi-anonymous critic to deduce from this? If he writes that the restaurant gives crappy service to non-VIPs, he is making an assumption based on his "useless random sampling" and writing something in his review that is not correct. There are any number of extremely plausible scenarios one could envision in which the quasi-anonymous critic's "quite a bit more data" based upon a (presumed) anonymous visit could lead to an erroneous review.

[oakapple]

4.  look at it this way, if he visits Per Se four times and manages to dine anonymously for just one of them...he now has quite a bit more data to work with.

"Quite a bit more data"? He has an arguably useless random sampling that, empirically, hasn't made his reviews any better than when he's recognized every time. "I wonder about Adour" hardly establishes anything to the contrary.

To be clear... What makes this a "useless random sampling" is that it's possible, for example, let's say that the critic is reviewing a restaurant with excellent service. But let's suppose that the restaurant's FOH staff has one of those rare "every so often" bad nights on the critic's first visit.

How is that sample any more useless than his sampling of the food? Suppose he pays one of his visits on the night that there's a substitute line cook manning the fish station, and all of the fish comes out tasting gummy? One bad night like that could be the difference between two stars and three.

[Fat Guy]

Everyone -- critics and readers included -- who has or learns of a bad service experience at a restaurant gives in to the human instinct to generalize from it. But an accurate statistical model doesn't.

If we assume that every restaurant -- even a four-star restaurant -- has incidents of bad service (for the purposes of a mathematical model let's divide service instances into "good" and "bad," though reality is probably closer to a point system, say 1 to 10) then every restaurant has what we could call a "service failure rate." Let's say a busy restaurant with 150 seats has 50 tables (average table size in a lot of restaurants works out to 3 and change, I believe) and does 3 sittings at dinner and 1 at lunch. That's 200 tables served per day. Assuming a 6-day week and some holiday closings we're looking at maybe 300 days of service. So that's 60,000 tables served per year, or 5,000 per month.

Just to make this simple, let's define our "period" as a month. Out of those 5,000 tables served in a month, how many are likely to experience service failure? At a restaurant with a superb service team, the number could be pretty low, maybe just a couple of tables a day. Having spent many a full service in restaurants that have won every good-service award, I can say with confidence that it's almost unheard of for a service to go by without some unfortunate disaster or another unfolding. But let's be exceedingly conservative and say the service failure rate for the period (one month) at our hypothetical three- or four-star-level restaurant with excellent service is 50.

Now let's say that at a two-star-level restaurant the service failure rate is 150 for the period. Triple the other restaurant's service failure rate, assuming same size and number of sittings. In other words we're defining -- just for the sake of this model -- three/four-star service as a service failure rate of 50/5,000 and two-star service as a service failure rate of 150/5,000. And the one-star rate, let's call that 500. Of course this oversimplifies, but it's useful for the illustration.

Okay, so as a visitor taking a random sampling, how many visits do I need to make in order to tell whether a restaurant has a three/four-star service failure rate, a two-star service failure rate, or a one-star service failure rate? I think we can all agree without resort to any high-level math that 1 instance of service failure has no bearing whatsoever on predicting whether we've just dined in a restaurant with a service failure rate of 50, 150 or 500. How about 2 or 3 instances? The one mathematician I asked said: "Unless the good restaurant failure rate predicts less failures than the number of critics visits, not sure how you could assess differences in the rates from that small number. If the good rate predicts less than 2 bad nights per period and you observe 2 or 3, maybe you can sort of say something. But if good rate predicts 35 bad nights and bad rate predicts 70 bad nights, and you observe 3 bad nights, I don't think you can statistically classify into good or bad." Feel free to dispute that, but it makes sense to me.

All the same modeling can be applied to certain aspects of food preparation. For example when critics write that such-and-such is "consistently" overcooked, they simply are not rendering a mathematically meaningful judgment as to consistency. Again, if there's an overcooking rate of X per Y at the best restaurant and Z per Y at a mediocre one, your visits at least need to approach X before you can distinguish between the best and the mediocre on the consistency point. No critic makes anything near that number of visits.

As I've said before, if you want to judge consistency, use a mechanism like Zagat but with a more serious approach. When you get 1,000 people responding, you get some meaningful numbers on the consistency point. Critics shouldn't be in the business of judging consistency -- it's mathematically not possible for them to do so -- so they should focus on higher-level issues that don't depend on a false concept of consistency. Which is exactly what other arts critics do.

[Nathan]

nah. music and theater critics note performance deficiencies all the time in their reviews. performance deficiencies that can well be just "off nights". and they make these judgments off of one performance (visit)!

it looks to me like the real issue is the very nature of reviewing.

[Fat Guy]

They note performance deficiencies that occur under optimal circumstances: opening night, with the whole cast there (or the first-string lead singer or whatever), operating with the knowledge that all the critics are in the house (because of course the critics are comped), viewed from excellent seats, etc. In other words the critics make a conscious decision to review performances at their best. In addition, performance deficiencies are mostly noted in reviews of performances of repertoire classics where performance is the relevant point of comparison. New material is more often evaluated for its underlying content.

But yes, it's all about the nature of reviewing and also the nature of what's being reviewed. If we think restaurant reviews are a species of consumer reporting and restaurants are consumer goods and services then that argues for the undercover, consistency-oriented approach (though the approach is still mathematically nonsensical and is better served by Zagat-type mechanisms anyway). If we think restaurant reviews are arts criticism and chefs are creative artists (or members of some related species) then we should be focusing on content.

[oakapple]

Fat Guy, I entirely understand your math. I think Bruni understands it too.

I am just asking why you think service is any different than food. All restaurants, even four-star restuarants, have off-nights in the kitchen, and they botch some plates even on their better nights. If, by dumb luck, some of those plates are headed for Bruni's table, it's liable to to be mentioned in the review.

Bruni often signals that he realizes the sample is unscientific. He'll say, "The pork chop was dry both times I had it," or "Fish consistently came out over-seasoned." He seems to mention service glitches only when there's a pattern that he feels is something beyond random chance.

[Sneakeater]

Some of this is beside the point, though, because whatever any of us might think, the Times pretty clearly views restaurant reviewing as a species of consumer reporting rather than arts criticism.

[Fat Guy]

But what Bruni (or you, or I, or any normal human) just knows is more than mere random chance can possibly be does not in two or three visits rise to a mathematical level that would actually distinguish it from randomness.

Sure, all this reasoning applies to food too. Which is why I go farther than most (or perhaps anyone) in opposing anonymity. I think restaurants should be given the chance, as in the performing arts, to bring their A games to a critic. In other words, they should have a shot at preparing every menu item up to spec. Yes, yes, I know there could be some shenanigans. But they happen now too, and there are plenty of ways to guard against them. But in the end the question is should we be reviewing restaurants at their best or trying to catch them at their worst?

[Nathan]

neither

[Sneakeater]

Ditto.

[Fat Guy]

Some of this is beside the point, though, because whatever any of us might think, the Times pretty clearly views restaurant reviewing as a species of consumer reporting rather than arts criticism.

I agree with that statement, but I think it has to be viewed in a larger context. We are in the middle of a creative revolution in gastronomy that is transforming the field from a craft focused on repetition to a creative art-form. The Times should try to get ahead of the curve, not be dragged behind it.

Were I in charge over there -- which I will never be -- I'd try to hire someone like Colman Andrews as the next critic. Someone who has a sweeping knowledge of nearly all aspects of cuisine, a serious historical perspective, and a strong grasp of the craft-to-art progression that's occurring not only at the high end but also at the Momofuku level. And I'd tell him only to review restaurants that are on the art end of the spectrum. Let some other writer -- one with more of a consumer orientation (perhaps a reporter like Frank Bruni or William Grimes) -- review the steakhouses and such.

[Fat Guy]

neither

So you must think option three is that reviewers should try to evaluate some sort of average consumer experience. But I think it's already been established that they can't.

[oakapple]

I agree with that statement, but I think it has to be viewed in a larger context. We are in the middle of a creative revolution in gastronomy that is transforming the field from a craft focused on repetition to a creative art-form. The Times should try to get ahead of the curve, not be dragged behind it.

I agree that The Times has largely failed at this. Bruni simply doesn't have the background—and it appears he never will—to appreciate current trends in any kind of historical context. (By the way, I don't think I have the background either; then again, I don't sell my reviews as a professional product.)

Were I in charge over there -- which I will never be -- I'd try to hire someone like Colman Andrews as the next critic... And I'd tell him only to review restaurants that are on the art end of the spectrum.

That might be going a bit too far. I'm not sure there are enough restaurants on the "art" end of the spectrum to fill 52 reviews a year. But you've been saying for a long time that the Times needs a critic for the "middle" of the spectrum (something between fine dining and $25&U), and I agree.

So you must think option three is that reviewers should try to evaluate some sort of average consumer experience. But I think it's already been established that they can't.

I'm not so sure we've "established" that. For a critic that's (supposedly) recognized up to 80% of the time, he's pretty good at sussing out service issues and inconsistency in the kitchen. It's actually not surprising that he often gets this right, since it's the main part of the job, as he sees it.

[Nathan]

neither

So you must think option three is that reviewers should try to evaluate some sort of average consumer experience. But I think it's already been established that they can't.

I want their best effort.

[Leonard Kim]

If we take FG's service failure rates at face value for the moment, then I think even isolated incidents of bad service are more than meaningless.

I've written before that one characteristic of Bruni's three star reviews is that he has no (or trivial) complaints about the food. I haven't checked to see whether the same is true for service, but it makes sense that's approximately true.

Given a failure rate of 1% for a 3/4 star restaurant (FG's 50 out of 5000 tables), if Bruni or you or me take four of those tables (assuming that's typically what goes into a NY Times review), we would not expect *any* service failures. The probability of having even one is about 4%, and there is basically no chance of having two bad tables. (I'm just using a binomial distribution here.)

Given a failure rate of 10% for a 1 star restaurant (500 out of 5000) on the other hand, sure, in four visits you still don't expect any service failures, but nor is it unexpected if you do have a bad night -- there's a 34% chance, better than one in three, if you go four times, at least one will be bad.

So if I'm reviewing a restaurant, and I go four times, and one of them is bad, sure there's a chance I'm at a three/four star place and am one of the unlucky 4%, which means, if I've reviewed thirty or so of these places, I've probably underrated one unfairly because of the bad service. But it's far more likely that the bad incident is meaningful in helping peg the service level.

Obviously, the numbers depend on FG's estimates being in the ballpark. Actually, I think the best way to do this is for you guys to make up a hypothetical restaurant with an established service level and "consistency." For example, take a three-star restaurant -- on a given night what's the probability of getting four-star service? two-star service? one star service? With those numbers, it should be fairly easy to calculate the probability of guessing the wrong rating on the basis of four visits.

[Nathan]

now assume that Bruni is recognized three out of four times at any given three star restaurant. the odds of a rare service failure on the one visit that he's not recognized are even lower...unless that restaurant really does have a high rate of service failures. in which case it's not a restaurant with good service.

[Fat Guy]

Nathan you have it exactly backwards: the fact that Frank Bruni is recognized 80% of the time give or take -- and that multiple anonymous visits to a given restaurant are, for him, a rarity -- is why his occasional one anonymous visit does so little to distinguish itself mathematically from randomness.

[Nathan]

your point is that it's effectively a random sample. right.

the point is that if a restaurant really does have outstanding service....then the odds of them screwing up on Bruni's table on the rare night that they don't recognize him are very low.

therefore, if a restaurant does screw up, the odds are quite high that that restaurant frequently screws up.

[Fat Guy]

Obviously, the numbers depend on FG's estimates being in the ballpark.  Actually, I think the best way to do this is for you guys to make up a hypothetical restaurant with an established service level and "consistency."  For example, take a three-star restaurant -- on a given night what's the probability of getting four-star service?  two-star service?  one star service?  With those numbers, it should be fairly easy to calculate the probability of guessing the wrong rating on the basis of four visits.

Actually I don't think it would make sense to look at the "probability of getting four-star service" etc. You never get four-star service at a middle-market restaurant, because four-star service, at least traditionally, is a whole style of service that depends on certain human resources, structures, procedures, etc., that can't just happen idiopathically. You'll notice that in my hypothetical I referred to 3/4-star service. That's because the failure rate (or whatever model you want to use) at the high-3 and 4-star level is going to be very, very close -- meaning you'd need a very large number of visits to expose the difference. You couldn't do it in 4 visits and you certainly can't do it in 1 (which would be more normal for Bruni). That's why instances like the Ducasse fiasco are random "gotchas" rather than statistically meaningful occurrences. But this all backs into an interesting point: you can go into a restaurant -- anonymously or not -- and very quickly identify that it's trying to offer four-star service or one-star service. Whether or not it succeeds is the big open question. And you can't answer that question unless you get a significant number of repeat, anonymous visits under your belt -- which so rarely happens that it's news when it does.

[Fat Guy]

the point is that if a restaurant really does have outstanding service....then the odds of them screwing up on Bruni's table on the rare night that they don't recognize him are very low.

therefore, if a restaurant does screw up, the odds are quite high that that restaurant frequently screws up.

I'm no statistician but my limited knowledge of the subject says you're wrong about that. Happy to hear from someone who knows more.

[Nathan]

the point is that if a restaurant really does have outstanding service....then the odds of them screwing up on Bruni's table on the rare night that they don't recognize him are very low.

therefore, if a restaurant does screw up, the odds are quite high that that restaurant frequently screws up.

I'm no statistician but my limited knowledge of the subject says you're wrong about that. Happy to hear from someone who knows more.

assume that a 3/4 star restaurant screws up 2% of the time.
assume that a 1/2 star restaurant screws up 25% of the time.

if I visit a restaurant once and they don't screw up...it doesn't tell me much.
if I visit a restaurant five times and they don't screw up...the odds of it being a 3/4 star restaurant are relatively high.
if I visit a restaurant once and they screw up....the odds of that restaurant being 1/2 are quite high. a 3/4 would have to quite unlucky to screw up on my sole visit.

if I visit a restaurant seven times and they screw up once, the odds of that restaurant being either a 3/4 or a 1/2 are relatively even. (there are ways to crunch this exactly...maybe a stats person here will do it)

[slkinsey]

the point is that if a restaurant really does have outstanding service....then the odds of them screwing up on Bruni's table on the rare night that they don't recognize him are very low.

therefore, if a restaurant does screw up, the odds are quite high that that restaurant frequently screws up.

I'm no statistician but my limited knowledge of the subject says you're wrong about that. Happy to hear from someone who knows more.

Let's assume that a Screw Up Rate ("SUR") of 1% represents outstanding service. I'm basing this on Steven's postulated 50 out of 5000 tables screwed up per month. We have no idea whether this is accurate, but it's a useful jumping-off point.

This means that every time Bruni sits down in a given restaurant, he would have a 1% chance of experiencing an evening of "unpreventable" poor service. This, as Nathan points out, is a low rate. But it is not zero. It also suggests that, for every 100 times Bruni sits down in a restaurant with a 1% SUR, he can as a general rule of thumb expect one evening of "unpreventable" poor service. However, just as it is possible for a simple coin flip to come up heads 100 times in a row, it's also possible for Bruni to hit either the jackpot or snake-eyes any number of times in a row. With a sample size of 4, it's entirely possible for Bruni to experience a SUR of 50% at a restaurant where the overall SUR is 1%. Indeed, given the number of restaurant meals consumed by Bruni, this is almost certain to happen.

Edited by slkinsey, 17 June 2008 - 11:05 AM.

[Nathan]

With a sample size of 4, it's entirely possible for Bruni to experience a SUR of 50% at a restaurant where the overall SUR is 1%.  Indeed, given the number of restaurant meals consumed by Bruni, this is almost certain to happen.

except that the contention was that he's usually recognized at high end restaurants. so, you can't normally expect him to get away with more than one anonymous visit. in which case, his SUR at a given restaurant is 1%.

edit:

now, yes, it's reasonably likely that about once every two years he experiences an SU at a restaurant with an SUR of 1%. so?

the benefit of the doubt should go to the consumer not the restaurant. so even if one restaurant every two years is wrongly downgraded based upon service, he'll have pegged multiple others considerably more accurately than if he dined completely non-anonymously.

Edited by Nathan, 17 June 2008 - 11:19 AM.

[slkinsey]

Nathan, that makes no difference. All it says is that there is a low percent chance that he'll experience a screwup event when he sits down. That defines his experience, but doesn't allow us to extend his experience to the restaurant at large, because the sample size is way too small (never mind the fact that we all understand that even a restaurant with great service will still occasionally give sub-par service to a valued customer or recognized critic).

[Nathan]

Nathan, that makes no difference.  All it says is that there is a low percent chance that he'll experience a screwup event when he sits down.  That defines his experience, but doesn't allow us to extend his experience to the restaurant at large, because the sample size is way too small (never mind the fact that we all understand that even a restaurant with great service will still occasionally give sub-par service to a valued customer or recognized critic).

see my edit.

as for sample size, that's purely a function of confidence level and allowable sampling error.

in this matter, neither needs to be strong for us to have meaningful results.

[Nathan]

and as I noted above,

a visit with poor service is vastly more significant than a visit with good service.

[oakapple]

Nathan, that makes no difference.  All it says is that there is a low percent chance that he'll experience a screwup event when he sits down.  That defines his experience, but doesn't allow us to extend his experience to the restaurant at large, because the sample size is way too small (never mind the fact that we all understand that even a restaurant with great service will still occasionally give sub-par service to a valued customer or recognized critic).

The trouble is, I understand what you're against, but I don't see what you're for.

Do you envision the Times saying one day, "From now on, we no longer report on service, because we've decided we can't."

Or do you envision the Times deliberately sending Bruni out to be feted as if he were the King of England, then writing reviews that disingenuously suggest that every customer can expect a similar experience?

Or do you envision a series of reviews like Craig Claiborne's famous $4,000 dinner, which no one claimed could be reproduced by the ordinary consumer, but which he reviewed mainly for the readers' entertainment?

[Nathan]

and what I'm saying is that one anonymous visit a restaurant, statistically, will very rarely screw over a deserving restaurant.

the odds will still be in the restaurant's favor. just not quite as skewed as they are when all of the visits are non-anonymous.