Let's create a better Restaurant Rating System

[jhlurie]

Obviously this system isn't perfect, but its got some real advantages.

Especially if you have like ten hours trying to figure out where to have dinner.

The beauty of ANY rating system is that you can choose to use it or not. If you want to invest the time to look at two numbers instead of a single number, well... than it might be handy. If you are in a real rush than I suppose you can glance at a copy of the NY Times and try to decipher what "2 Stars" really means.

[Steve Plotnicki]

If you are in a real rush than I suppose you can glance at a copy of the NY Times and try to decipher what "2 Stars" really means.

Well the problem is that they aren't consistant. It makes sense that Daniel is four stars, Union Pacific is three, but no sense that Otto is two. So we aren't having this conversation because they star system is inherently a bad system, we are having it because the NY Times doesn't follow their own guidelines when it comes to rating restaurants.

[Fat Guy]

A good system would be to take the various styles of cuisine and break each into high-end, middle-market, and cheap-eats groupings.

You mean like my 2001 Guide to Dining.

I haven't finished reading it yet.

Rankings are good for a certain kind of user. Ratings work better for a broader base of users. Most people who have limited time to do research, want to know what the top restaurants are and that's it. I have no way of knowing but, I bet the list of the 25 most popular restaurants at the beginning of each Zagat's is extremely useful for people who want to eat well, but aren't maniacal about every last detail.

You don't have to be "maniacal about every last detail" to want Ducasse and Soup Kitchen International on two separate lists. Aggregating them makes no sense and in fact requires work to sort out -- you have to follow through the listings in order to learn the price range, and in order to learn that Soup Kitchen International isn't the type of place you'd take that important client for lunch. The top-level decision isn't likely to be "I want to eat at one of the 25 most popular restaurants in New York." The relevant top-level decisions are likely to be 1) level of luxury, and 2) style of cuisine (I'm ignoring location because it's a function better served by an index than as a structural element of a ranking or rating system). Within that context, people want to know what is the best, hottest, etc. I say this with some authority, as the recipient of thousands of e-mails each year from people who trying to choose restaurants in New York City.

There aren't so many categories. Creating them well is a challenge, of course, but it can be done. Just for example, the list would look something like (the restaurants listed are just examples, not necessarily my actual picks):

French

-Luxury

1. ADNY

2. Jean-Georges

3. Le Bernardin

etc.

-Mid-range

1. Balthazar

2. Pastis

etc.

-Cheap eats

1. La Bonne Soup

2. La Crepe

etc.

Japanese

-Luxury

1. Sushi Yasuda

2. Kuruma Zushi

3. Nobu

etc.

-Mid-range

1. Tomoe

2. Blue Ribbon Sushi

etc.

-Cheap eats

1. Daikichi Sushi

2. Teriyaki Boy

etc.

For those who want aggregation, it's easy enough to start the whole thing with some overall top-10 lists: Top 10 Luxury Restaurants, Top 10 Dining Values, etc. These are going to suffer from the apples-to-oranges problem, but they'll be desirable from a usability standpoint. I'm assuming each city/region gets treated separately.

The way I'd implement it is to pre-qualify all "inspectors." There would be a baseline dining-experience requirement and a certain level of knowledge and discernment would have to be displayed. Were we to do it on eGullet, I'd probably just say 100 posts plus having eaten at 20 restaurants on the current ranking list qualifies you to participate. Those who are approved as "inspectors" would get access to the rankings and would add their evaluations each time they eat at a given restaurant. These evaluations would be converted to numbers that would serve to push a restaurant higher or lower in its category, or leave its ranking unchanged. Categories would vary in size based on the overall size and quality of the pool of potentially relevant restaurants to each category. The initial list would have to be created by an executive committee of experienced diners. Inspectors' evaluations would be publicized to all the other inspectors in a discussion forum so that some degree of consensus could be reached -- even where there is disagreement, this would at least work to minimize the impact of subjectivity. The aggregate Top 10 lists would have to be created by the executive committee.

[Explorer]

And the best thing about this kind of system would be its real-time dynamic nature.

But would we allow new input to immediately influence new rankings, or would we "hold" new input and release it simultaneoulsy after a given period like a month or 2 months, etc.?

[Steve Plotnicki]

Fat Guy - I'm not opposed to as many lists/rankings as possible. But a book where dining is dissected into all of those small parts is difficult to use by anyone who doesn't want to put in a lot of time and effort. Most people who use guidebooks are visitors to a city or region, and they are there for 2-4 days and they want a quick way to glance at all the information. In reality, information really falls into two categories. Where to eat dinner, and where to eat lunch. A book that is well organized this way is Patricia Wells Food Lover's Guide to Paris. She has all restaurants, regardless of price and type of restaurant (Chinese is listed with Taillevent) in the restaurant category, and then she has split out cafes and wine bars into their own categories. Now to me that makes sense. Like if there was a category worth having of better Greek coffee shops and other lighter meals, that would make sense. But I see no reason to split the restaurants out from each other. To me, Daniel, Peter Luger's and New York Noodletown are properly listed together. Dining in NYC is really about eating at the upper middle/high end. And restaurants where the cuisine derives from people who originated in a different country (do you like my replacement phrase for "ethnic food," catchy eh ? ) is less important and you only need two, three, four of the best examples. As a practical matter, anyone who needs to know about more then three or four Indian restaurants is in a different market from the average vivistor and needs a different kind of book that specializes in CDFPWOIADC (that's even catchier.)

But of course if you want all of the information laid out in one place, you need it organized differently. In that instance the Time Out Guide's listing by category is sufficient. But they are void of any rankings and that makes it cumbersome. They used to rank the top five places in each category but they stopped doing that two years ago. But those rankings were useful when they had them.

[Pan]

Steve Plotnicki:

Most people who use guidebooks are visitors to a city or region, and they are there for 2-4 days and they want a quick way to glance at all the information.

But it sounds like Steven is talking about a guide on eGullet. If that's the case, it'll be for people living in the New York area and those already somewhat familiar with New York as well as gastronomic travellers from hither and yon. I find the prospect exciting.

Dining in NYC is really about eating at the upper middle/high end. And restaurants where the cuisine derives from people who originated in a different country (do you like my replacement phrase for "ethnic food," catchy eh ? ) is less important and you only need two, three, four of the best examples. As a practical matter, anyone who needs to know about more then three or four Indian restaurants is in a different market from the average vivistor and needs a different kind of book that specializes in CDFPWOIADC (that's even catchier.)

Yeah, good end-around.

But are you really claiming that the average visitor to New York is spending all or most of his/her time eating in upper-end restaurants? You're not claiming that, right? So do I take it that the statement that "Dining in NYC is really about eating at the upper middle/high end" and the statement that "anyone who needs to know about more then three or four Indian restaurants is in a different market from the average vivistor and needs a different kind of book that specializes in CDFPWOIADC" are unrelated to each other?

[Steve Plotnicki]

But are you really claiming that the average visitor to New York is spending all or most of his/her time eating in upper-end restaurants? You're not claiming that, right?

No I am claiming that the market that wants the information about high end restaurants and the market that wants it about cheap eats are completely different. And it isn't that the high end market doesn't want to know about NY Noodletown, but their interest in that level of dining is limited.

It is like when I go to San Francisco. Their Chinatown is better then ours. But I don't need endless information. Knowing that Yank Sing has the best dim sum, and that Yuet Lee makes a salt and pepper crab from another dimension, and maybe one more recommendation, that is enough info for the average tourist who is interested in the SF dining scene. And if I needed more information about cuisines that derive from people who originate in another country, I need a book that specializes in that type of dining. Having all that information together and ranked, won't help anyone. The markets have different needs and have different values.

[stefanyb]

And if I needed more information about cuisines that derive from people who originate in another country , I need a book that specializes in that type of dining.

Is this the phraseology you have arrived at to replace the term "ethnic"? A little cumbersome, no?

[Fat Guy]

The markets are far from monolithic, and the people who will actually pick up a restaurant guide or look online (as opposed to those who just ask the concierge or go to Hard Rock Cafe) are already a self-selected minority. Of that group, if Fat-Guy.com's demographics are indicative, approximately 1/3 are from the city (in my case New York), 1/3 are from the greater metro area (i.e., suburban commuter types), and 1/3 are tourists. Of the tourists, a healthy percentage have interest beyond just hearing the top two or three places in a category -- especially those who are repeat visitors. The way to address all the markets is to provide depth and detail, and to offer abbreviated/aggregated lists to address those who want only top-level information. Top 10 lists are ideal for this if the categories are well chosen. They don't necessarily have to derive from the ratings, either. There can be independent lists such as Top 10 Chinese-restaurant dishes and where to get them. That sort of thing. The trick is to make the top-level lists easy to access and read, but to have the deeper lists available as backup for the more interested readers.

[Steve Plotnicki]

Is this the phraseology you have arrived at to replace the term "ethnic"?  A little cumbersome, no?

Stef - Ha, ha, ha. I was waiting for someone clever to figure that out .

Fat Guy - But the people who go to your website are foodies. They want unlimited information. They aren't the typical visitor who is coming to NYC for 3-4 days and wants a varied dining experience. It's the same for me. I'm as hungry and as interested in eating as anyone else but I don't need that many recommendations when I go somewhere. I really want to know what the amazingly good places are. The rest really has no meaning to me.

[stefanyb]

Is this the phraseology you have arrived at to replace the term "ethnic"?  A little cumbersome, no?

Stef - Ha, ha, ha. I was waiting for someone clever to figure that out .

Steve- You know I was trying to think back to the dark ages of my childhood and I seem to recall the phrase "foreign food". Maybe we could resurrect that one?

[Elissa]

The way to address all the markets is to provide depth and detail, and to offer abbreviated/aggregated lists to address those who want only top-level information. Top 10 lists are ideal for this if the categories are well chosen. They don't necessarily have to derive from the ratings, either. There can be independent lists such as Top 10 Chinese-restaurant dishes and where to get them. That sort of thing. The trick is to make the top-level lists easy to access and read, but to have the deeper lists available as backup for the more interested readers.

Excellent idea. The board members could perhaps present their own criteria, preferences and prejudices up front. A panel of X members from a given city all submit twenty five top tens, from which the 'official' answers might be calculated. These have both a 'populist' glean of objectivity, being the average of several people's opinions, and the insurance of well educated palates.

Readers/users might also be able to compare top tens from board members whose tastes they have reason to sypathize with (or reject), if they could set the search to lists made by panel members who are primarily seeking top quaity, creativity, authenticity or value.

[Steve Plotnicki]

What's the difference between what you suggested and Zagat?

[Elissa]

Zagat lets anyone chime in about their favorite. The panel FG proposes would be made of people who have presumably agreed on some notion of criteria. Therefore the ratings would be less indiscriminately, popularly-based than Zagat's.

Also, the ratings could be interactive, tabulated by the more particular preferences a user shares with select contributors within the panel.

Edited March 3, 2003 by lissome (log)

[Steve Plotnicki]

Also, the ratings could be interactive, tabulated by the more particular preferences a user shares with select contributors within the panel.

Yes interactive is good. Maybe we can wire the panel so when they go out to eat, the results are changing by the moment. You know like the read out by the Midtown Tunnel that supposedly keeps track of the national debt.

[Fat Guy]

Fat Guy - But the people who go to your website are foodies. They want unlimited information. They aren't the typical visitor who is coming to NYC for 3-4 days and wants a varied dining experience.

You're making an assumption that happens to be incorrect. Most people who come to my Web site come because they searched for "New York restaurants" on Google. They're the same people who go to Zagat and CitySearch, just not as many. I also have die-hard foodies in my audience, but they're a minority. Anyway, you're not paying attention to my idea. I've said a few times now that the low-information-requirement crowd can be served by a few top-10 lists. Do you disagree with that? I don't see how you could. But if that's all you offer -- which seems to be what you're advocating -- you lose the rest of the market, which is substantial. You lose all the people who live in the market, and near it. And even for the tourists, not everybody has a dismissive attitude about dining in other cities. Some people actually care to go beyond the top two or three places in a category, especially when they're repeat visitors to a city.

Edited March 3, 2003 by Fat Guy (log)

[Steve Plotnicki]

Not everybody has such a dismissive attitude about dining in other cities. Some people actually care to go beyond the top two or three places in a category, especially when they're repeat visitors to a city.

I find that few cities have more then 2-3 places worth eating in to begin with. If they do, it's usually in the upper middle and above category. All other categories are usually expressed in one or two places that tower above the rest. In the "foreign food" category, aside from NYC, L.A.. possibly S.F. because the Chinatown is so large, what city has multiple choices in non-upper middle level places? I go to Miami each year for a week and the choices outside of the big boys is piss poor. Yes there is Rascal House, yes there is Versailles and yes there is that chain of Nicaraguan steak houses. But there isn't really much more when it comes right down to it. And that city is an ethnic hub. What city aside from the big boys, doesn't fit this profile? And that isn't to say that there isn't good Vietnamese or Indian in Seattle. But do I really need to know about 15 places?

But if that's all you offer -- which seems to be what you're advocating -- you lose the rest of the market, which is substantial. You lose all the people who live in the market, and near it.

I'm not dismissing this market. It is handled quite well by the Time Out Guide to Dining. The problem is they don't do any ratings or rankings or recommendations. If they did it would be an improvement. But if we are talking about a guidebook, the amount of diversity you need to offer is much less.

[Pan]

But Steve, when I go to San Francisco, I want to eat the food that is strongest there and, not coincidentally, weaker in New York. So I'm much less interested in haute cuisine or even moderately-priced French restaurants than Mexican, Central American, Thai, Chinese, and other East and Southeast-Asian cuisines. Of course, you might disagree with me on what SF is strongest in.

[Steve Plotnicki]

But Steve, when I go to San Francisco, I want to eat the food that is strongest there and, not coincidentally, weaker in New York. So I'm much less interested in haute cuisine or even moderately-priced French restaurants than Mexican, Central American, Thai, Chinese, and other East and Southeast-Asian cuisines. Of course, you might disagree with me on what SF is strongest in.

The world would be a better place to eat in if you were actually a reflection of the market. But for example, in the 2003 Zagat Guide, there is not a single ethnic restaurant in the Top 50 most popular places. You would figure one place would make the list. Katz's, NY Noodletown, John's Pizza, something. but they don't. And plenty of stupid places make it like Ruby Foo's or Angelo & Maxie's.

[Fat Guy]

How does Zagat help to define the market you're talking about? I thought you were talking about tourists. I doubt a significant percentage of the Zagat surveyors are tourists.

[Steve Plotnicki]

Well the tourists want to know what restaurants are hot, and where the hidden away gems are. Zagat happens to do a fairly good job in both categories. You can find Granercy Tavern and Sahara in Brooklyn.

To me the entire issue breaks down into two categories. Dinners and casual dinners/lunches. We know what the dinner places are. And we know what the casual dinner places/lunch places are. Most people's dining needs fall into these two categories (at least mine do.) But if you are not interested in fancy dinners, I think you need a different type of guide book with extensive casual eating recommendations. I mean you can spend an entire week in NYC eating Chinese food or Thai food. You can even make a week out of Vietnamese or Indian/Pakistani. But if you are that type of diner you need something detailed about Asian cuisine that doesn't pay any attention to the Union Square Cafe. But once you include USC and Gramercy etc., few people are having dinner at those places on Monday night and at Bukhara on Coney Island Avenue the next night. I know it happens, but that's for the hardcore foodies and in reality those people only want/need to know the very top places.

Edited March 3, 2003 by Steve Plotnicki (log)

[Explorer]

Aside from the discussion on categorization (which is a how-to vs. what-to), I think a key issue with existing guides is the level of accuracy and recency they provide. Being in print, many of them aren't updated for another year or more, so part of the info becomes obsolete.

The innovation would be in having a continously up-todate type of guide, no matter how it's sliced and diced.

I think we should find ways to translate the dynamism, recency of information and attention to detail that exist on this website into a list project. That would be the "better restaurant rating system"; i.e. one that is always accurate and always relevant.

If we can agree on the "what" first, it might make the discussion of the "how" a bit easier.

[Fat Guy]

The time/currentness issue mostly pertains to new restaurants. It's not likely that an established place is going to improve or decline massively in a year -- it happens, but the overwhelming majority of such places are pretty stable. The big issue is covering new restaurants, especially the 50 or so most important new ones that open in a given year.

Obviously, the Web is an ideal medium. All my posts have assumed that. Were the effort to be assembled into a print guidebook, it would have to be subject to the same limitations as any other. The only workaround might be a looseleaf-type format like the DayTimer, where you'd offer people print-on-demand for new restaurants and major revised ratings. But it's so cumbersome I doubt it would make sense. Better just to keep the info online and deliver it to PDAs.

[Explorer]

The big issue is covering new restaurants, especially the 50 or so most important new ones that open in a given year

That idea on its own is worth a lot if we can keep feeding the new info into the list. Nobody does it well; there's Zagat wire that's good at announcing (for major cities), but not necessarily at critiquing til after the popular votes are in.

It's not likely that an established place is going to improve or decline massively in a year -- it happens, but the overwhelming majority of such places are pretty stable

For the discriminating, fuddy-duddy like me, even that variation is important- not knowing that a chef has left or come in, or that the last 5 reviewers I respect didn't like something anymore is valuable. North American restaurants tend to be less stable than ones in France for e.g.; we're such a fast and furious society that anything can happen quickly. It's exciting to trust the innovation and energy of a new Chef that's 28 years old, but there is no track record that proves he/she will keep at the top of the game unless it is regularly proven that they do. I think we're talking quarterly earnings vs. annual report.

Will Cote D'Or be the same without Bernard Loiseau? I don't know, but I know somebody is strongly considering cancelling their 4-day stay over there, for e.g.

Also, new things can happen to old places which could catapult them into the top 50 again; and I want to capture that as well when it happens, not one year later.

[jhlurie]

The time/currentness issue mostly pertains to new restaurants. It's not likely that an established place is going to improve or decline massively in a year -- it happens, but the overwhelming majority of such places are pretty stable. The big issue is covering new restaurants, especially the 50 or so most important new ones that open in a given year.

I'm still a bit dubious about how you would determine "the 50 or so most important new ones". This breaks down on the lower end, and its my contention that this is where most current systems already dissapoint. While its true that I absolutely wouldn't care to see ranking or ratings of every little hamburger joint on every corner, I WOULD like to see a system that realistically included the best of them somehow. One of the major premises behind this thread, I thought, was that existing systems had no way to indicate/recognize excellence in these areas in a way which both showed their own primacy, yet simultaneously indicated their proper distance from "finer" food.

And unlike the Plotnickis of the world I can't magically categorize in my head. I eat at the top, in the middle and at the bottom indiscriminately, as long as the food is good. And in fact, Steve made a very good point in that "the 2003 Zagat Guide, there is not a single ethnic restaurant in the Top 50 most popular places". That's exactly the problem with a ranking system that mixes disparate restaurant types.