The passage of California’s Proposition 8 (the gay marriage ban) has created a nasty problem for Yelp and other online reviews sites.
The situation is a good example of the complexities of running a social site, and the ability of web-organized groups to distort a social ratings system. I’m not sure what the lesson is from the whole situation, but review sites are becoming more influential in business, so it’s important for us all to understand how they work and how they can break down.
If you don’t live in California, it may be hard to understand how polarizing the aftermath of Proposition 8 has been. The campaign was ugly, with heavy TV advertising playing on a lot of fears, and people were passionate on both sides of the debate. In the wake of the proposition’s passage, there has been a fair amount of protest, and efforts by some Proposition 8 opponents to boycott businesses that supported it.
Case in point: A Los Angeles restaurant called El Coyote, whose manager contributed $100 to Proposition 8. Since the election, the restaurant has been the site of protests, and business is off substantially (link). The restaurant’s Yelp site has been flooded with one-star reviews trashing the restaurant’s food and service, some specifically mentioning Proposition 8 (link). A few excerpts from recent one-star reviews:
Never “loved” this place, but enjoyed it every so often, especially with a few margaritas helping the canned Mexican food go down. Then I find out the owner gave money in support of Prop 8, and that sealed the deal for me. Until they apologize and right their wrong, don’t expect me to stop by.”
“I’ve been here a few times, and this is without a doubt the worst place I know of in California. Do not go here unless you don’t mind bad food, high prices, a horrible vibe, and can turn your back on risks of food poisoning and human rights and health code violations!”
“Maybe I have a weak stomach, but I couldn’t stop vomiting after eating here.
Although it’s impossible to tell for sure which of the reviews are fake, the averages tell an interesting story. Prior to the election, El Coyote had a rating of over three stars (fairly good for a casual restaurant). Since then, its average rating is less than two (quite bad). On Citysearch, the El Coyote reviews page (link) shows a similar pattern of recent one-star reviews, many referencing Proposition 8. But those reviews don’t appear to have changed the restaurant’s overall rating. The restaurant’s rating hovers about 4.5 stars, although the average user score is about three stars. I don’t know how that works. Citysearch claims the ratings it displays are a composite of the user ratings (link), but clearly they’re doing something to adjust the numbers. This seems to be true of some other restaurants as well, so it’s not just a Proposition 8 thing. I couldn’t find an explanation on Citysearch’s site; please comment if you know the answer.
Yelp has been pushing back on the protesters by deleting some reviews that reference Proposition 8. As the company explained (link):
I’m writing to let you know about our decision to remove a few of your reviews, those only pertaining to Proposition 8….Our review guidelines… state: “Reviews aren’t the place for rants about a business’s employment practices, political ideologies, or other matters that don’t address the core of the normal customer experience.”
Those deletions have angered some Yelp users, and apparently led the Mayor of San Francisco to cancel the city’s planned Yelp Day (link).
This is a complete no-win situation for Yelp. Parts of the Yelp user base are now polarized on both sides of the issue (link), and the company has lost a PR opportunity. More importantly, despite the deletions, the perceived integrity of Yelp’s rating system has been compromised. Many reviews still mention Proposition 8, so the deletions look arbitrary. And the rating for El Coyote has clearly been distorted. But Yelp can’t do much about it because there’s no way to tell for sure which of the reviews are fake.
That raises the question — which other online reviews have been gamed? If a company like Yelp can’t filter out a publicly-declared protest, how many other reviews there and on other sites have been distorted by more subtle campaigns, either in favor of or against a particular business?
I don’t honestly know. And neither do you.
This has important implications for any company involved in online ratings. If they’re influential in your market, you should watch carefully to make sure a competitor isn’t subtly distorting your ratings. And you should be very wary of angering organized groups that can use social sites against you. In particular, you should be aware that political contributions are part of the public record, and require that the contributor’s employer be identified. Those records are easily searched on the web.
If you run a ratings site, you have a couple of choices. You can decouple your overall ratings from the user ratings, as Citysearch apparently does. But that reduces the credibility of your site — in our survey of web users, they trusted user reviews a lot more than reviews written by editors. If you want to rely on user reviews, being transparent is probably your best defense. Yelp’s ability to show ratings over time is useful, and probably needs to be featured more prominently, so users can spot strange trends. There should also probably be a way for users to flag a review page that shows unusual activity, just like Wikipedia pages can be flagged when something unusual is happening with them. And it makes sense to let users sort reviews according to the track record and credibility of reviewers.
Unfortunately, all of these measures implicitly say to a casual user that the site’s reviews might be suspect, which in itself hurts the credibility of the site.
In the end, I don’t think there are any easy answers. Like the situation in computer security, there’s going to be a long-term arms race between rating sites and people who want to game them. The best we can do is be aware of the problem and try to make the gaming process as hard as possible.
(Note: this post was co-authored during my time leading Rubicon.)