Eater recently pointed out that the "Vote For Me in the Zagat" season is upon us again. Not coincidentally we received an email from zagat.com this week encouraging us to vote and promising a free 2009 New York City Zagat guide for our troubles. Each year Zagat is faced with a number of criticisms of its guides and in keeping with this grand tradition I thought I'd share two issues that put into question Zagat's claim of being a "trusted source to help consumers make informed decisions about restaurants."
The first is the Zagat custom of rewarding voters with a free printed guide for submitting votes. This incentive seems unnecessary in NYC given that Zagat claims to have a local voter base of nearly 35,000 diners. While the incentive might be needed to help jump start smaller markets like Seattle, where only 3,165 local surveyors contributed to the 2008 guide, it seems logical that this incentive would diminish the overall quality of vote submissions. Why not take five minutes to register for zagat.com and quickly vote for a handful of restaurants that you may or may not have been to recently in exchange for a free $16 guide book, after all?
The fact is that Zagat has no way of knowing if a person has actually been to a particular restaurant. In
a post about how OpenTable is now collecting reviews from diners after their meals, Dan Entin, a former Zagat product manager, recently stated in his personal blog, "OpenTable is in an incredibly unique and desirable position when it comes to restaurant reviews: they actually know where and when you ate. When I was at Zagat this was information that we wanted badly so we could do exactly this kind of outreach to spur review-writing." This issue isn't limited to Zagat, of course, but given the use of an incentive that likely taints the voter pool with low quality and potentially specious votes, it's hard to put a tremendous amount of faith in the results.
The second issue relates to the commercial impact of a positive Zagat rating, its affect on industry behavior and the difficulty of filtering out bad votes. It's apparently a common practice for restaurants to reward chefs based on their Zagat food rating. In a 2007 interview Tim Zagat stated, "A lot of restaurants actually give bonuses to their chefs based on the ratings. They judge themselves because this is the only way they can really have a systematic rating of themselves." This is a direct incentive for a "get out the vote" campaign if I've ever heard one.
Not surprisingly the Zagats claim to employ costly filters to help eliminate biases such as these. In the same interview Zagat said, "We don't want just anybody going in and throwing their comments on the wall, and therefore we have about 30 filters between coming in and voting and actually being counted as a vote. We don't want restaurants to be able to get in and influence the vote on the restaurants, obviously. So we have spent millions of dollars protecting the process by developing a whole series of filters, including having our local editor and our editors in New York checking the results to make sure that nothing extraordinary and unusual is happening."
Anecdotal evidence, including my own experience of quickly submitting a few lame restaurant reviews last year and subsequently getting my free book, leads me to believe that getting around these multi-million dollar filters isn't so tough after all.
Share on Facebook
Save to del.icio.us
Stumble It
Twit This!
Get Free RSS Updates
6 Comments
A takedown of Zagats from a competitor. That is credible. As a Red Sox fan I'm on my way to post how awful the Yankees are as an organization. Just got finished reading that article by the CEO of Pepsi on why Coke's ingredients are not good.
Rob - We're tickled that you would have us play Red Sox to Zagat's Yankees or Pepsi to their Coke, but seriously, we'd love to hear you comments on the merit of the post itself.
Zagats, should post the number of voters for each establishment next to the ratings. In the past some restaurants have received stellar or dismal ratings with minimal voting (less than 100 votes). The book advertises that is has more than 300,000 voters but in reality, even the top 50 restaurants sometimes have less than 3000 votes! Hardly plurality!
By your request Chris here is my response to your post. You are repeating the same claims that have been thrown around for years so you are not adding anything new to the argument.
You present barely circumstancial evidence poiting to you getting around the voting safeguards. And saying Zagats has no way of knowing if you have ever been to a restaurant... neither does Citysearch, Yelp, etc or any startup that collects user reviews. In fact you have no idea if someone owns a product on Amazon or Best Buy or any user review system. Your issue is not with Zagat but with user generated content as a whole. Don't crucify Zagats for the faults of the "You" generation.
I think you also forgot that YOU COLLECT USER REVIEWS ON YOUR SITE. How does anyone miss that hypocracy???? I personally have posted reviews anonymously. How do you verify anything? There is pot calling the kettle black and then there is calling your identical twin ugly.
I also consider that since Zagat pulls out quotes then someone is reading the reviews. There is an assumed level of quality control given that... more so than sites that only show the user generated content.
And J Smith how do you have inside information into how many votes make up a rating?
Hi Rob. Your response that I'm throwing out claims that have been made many times before may be true, but I've yet to hear or read anything from the Zagat folks that leads me to believe that their "filters" are in fact effective and I should therefore trust their reviews.
The argument in my post that I do believe is new is that Zagat's NYC ratings would benefit from an increase in quality and trustworthiness if they removed the perverse incentive of offering a free book for submitting as few as three rather lame reviews.
The fact that most all UGC, including ours, is unverifiable is indeed an issue and it's a problem that we're actively working on solving (more on that later). I was merely pointing out that this lack of verification in addition to the free book incentive has led to a system that is less trustworthy.
Good or bad, or rather, good AND bad, UGC is here to stay. This is the reality and we need to understand that it's the whole new world is being created right before us.
In this game, the guy that has the most reliable information is ultimately going to win (look at where it got Google). Which puts Opentable in a very strategic spot.
Zagat has a strong brand but they could be shooting themselves in the foot by providing incentives. The verification process is a clumsy attempt at fixing the problem they themselves have created in the first place.
I'd be interested to know what you guys plan on doing to maintain the high quality of your survey data.