The passage of California’s Proposition 8 (the gay marriage ban) has created a nasty problem for Yelp and other online reviews sites.
The situation is a good example of the complexities of running a social site, and the ability of web-organized groups to distort a social ratings system. I’m not sure what the lesson is from the whole situation, but review sites are becoming more influential in business, so it’s important for us all to understand how they work and how they can break down.
If you don’t live in California, it may be hard to understand how polarizing the aftermath of Proposition 8 has been. The campaign was ugly, with heavy TV advertising playing on a lot of fears, and people were passionate on both sides of the debate. In the wake of the proposition’s passage, there has been a fair amount of protest, and efforts by some Proposition 8 opponents to boycott businesses that supported it.
Case in point: A Los Angeles restaurant called El Coyote, whose manager contributed $100 to Proposition 8. Since the election, the restaurant has been the site of protests, and business is off substantially (link). The restaurant’s Yelp site has been flooded with one-star reviews trashing the restaurant’s food and service, some specifically mentioning Proposition 8 (link). A few excerpts from recent one-star reviews:
Never “loved” this place, but enjoyed it every so often, especially with a few margaritas helping the canned Mexican food go down. Then I find out the owner gave money in support of Prop 8, and that sealed the deal for me. Until they apologize and right their wrong, don’t expect me to stop by.”
“I’ve been here a few times, and this is without a doubt the worst place I know of in California. Do not go here unless you don’t mind bad food, high prices, a horrible vibe, and can turn your back on risks of food poisoning and human rights and health code violations!”
“Maybe I have a weak stomach, but I couldn’t stop vomiting after eating here.
Although it’s impossible to tell for sure which of the reviews are fake, the averages tell an interesting story. Prior to the election, El Coyote had a rating of over three stars (fairly good for a casual restaurant). Since then, its average rating is less than two (quite bad). On Citysearch, the El Coyote reviews page (link) shows a similar pattern of recent one-star reviews, many referencing Proposition 8. But those reviews don’t appear to have changed the restaurant’s overall rating. The restaurant’s rating hovers about 4.5 stars, although the average user score is about three stars. I don’t know how that works. Citysearch claims the ratings it displays are a composite of the user ratings (link), but clearly they’re doing something to adjust the numbers. This seems to be true of some other restaurants as well, so it’s not just a Proposition 8 thing. I couldn’t find an explanation on Citysearch’s site; please comment if you know the answer.
Yelp has been pushing back on the protesters by deleting some reviews that reference Proposition 8. As the company explained (link):
I’m writing to let you know about our decision to remove a few of your reviews, those only pertaining to Proposition 8….Our review guidelines… state: “Reviews aren’t the place for rants about a business’s employment practices, political ideologies, or other matters that don’t address the core of the normal customer experience.”
Those deletions have angered some Yelp users, and apparently led the Mayor of San Francisco to cancel the city’s planned Yelp Day (link).
This is a complete no-win situation for Yelp. Parts of the Yelp user base are now polarized on both sides of the issue (link), and the company has lost a PR opportunity. More importantly, despite the deletions, the perceived integrity of Yelp’s rating system has been compromised. Many reviews still mention Proposition 8, so the deletions look arbitrary. And the rating for El Coyote has clearly been distorted. But Yelp can’t do much about it because there’s no way to tell for sure which of the reviews are fake.
That raises the question — which other online reviews have been gamed? If a company like Yelp can’t filter out a publicly-declared protest, how many other reviews there and on other sites have been distorted by more subtle campaigns, either in favor of or against a particular business?
I don’t honestly know. And neither do you.
This has important implications for any company involved in online ratings. If they’re influential in your market, you should watch carefully to make sure a competitor isn’t subtly distorting your ratings. And you should be very wary of angering organized groups that can use social sites against you. In particular, you should be aware that political contributions are part of the public record, and require that the contributor’s employer be identified. Those records are easily searched on the web.
If you run a ratings site, you have a couple of choices. You can decouple your overall ratings from the user ratings, as Citysearch apparently does. But that reduces the credibility of your site — in our survey of web users, they trusted user reviews a lot more than reviews written by editors. If you want to rely on user reviews, being transparent is probably your best defense. Yelp’s ability to show ratings over time is useful, and probably needs to be featured more prominently, so users can spot strange trends. There should also probably be a way for users to flag a review page that shows unusual activity, just like Wikipedia pages can be flagged when something unusual is happening with them. And it makes sense to let users sort reviews according to the track record and credibility of reviewers.
Unfortunately, all of these measures implicitly say to a casual user that the site’s reviews might be suspect, which in itself hurts the credibility of the site.
In the end, I don’t think there are any easy answers. Like the situation in computer security, there’s going to be a long-term arms race between rating sites and people who want to game them. The best we can do is be aware of the problem and try to make the gaming process as hard as possible.
(Note: this post was co-authored during my time leading Rubicon.)
You left out one major factor. The food was never the main attraction of El Coyote. It’s notoriously bad actually. The main reason people went there was the sense of community and inclusiveness. All that went out the window when word got out that the manager voted away the rights of some of her customers. Her $100 donation helped finance negative attack ads that vilified some of El Coyote’s most loyal patrons. It’s hard to feel welcome at the place knowing that.
Also the boycott has triggered a counter-boycott. El Coyote is now getting business from a considerably more straitlaced crowd — people who aren’t necessarily as tolerant of gay people as the regular customers. This has completely changed the mood of the place. So it’s not entirely wrongheaded to address Prop 8 when posting a restaurant review.
Thanks very much for the info, Jon. Good points. I tried to stay out of the question of whether the restaurant did right by its clientele. I’m too far away to judge that.
And I agree with you that a change in the atmosphere of a restaurant is a relevant topic for a review.
But that’s not what most of the reviews talk about. Check out Citysearch’s listing for El Coyote — the user reviews are dominated by people posting one star because they opposed Prop 8 and posting five stars because they favor it (or anyway oppose the boycott of the restaurant). The usefulness of that review page has been destroyed. Since our focus is on advising the tech industry, not restaurants, that’s the problem I was trying to talk about. It points out a weakness in community review sites, and as the review sites get more popular I think the problems may grow.
From what I’ve seen, there are several patterns by which online ratings sites/tools can get gamed:
1) the owner fakes some reviews, and/or asks some friends/family to do reviews (good reviews for their company, and/or bad reviews for the competitors). This is the most common game.
2) person writes a bad review to vent his/her bad feelings, or to “punish” someone with whom they’ve had a bad encounter (didn’t get job offer, fired employee, ex-boyfriend/girlfriend). This happens occasionally.
3) the business gets caught in a hot topic issue and there’s a viral pile-on. This is the El Coyote problem. This is pretty rare compared to the first two but it’s very visible.
While no one can be certain if a specific review is faked, there are indicators, and some sites take steps to address the issue. For example, TripAdvisor says they contact some reviewers to double-check that they are who they say they are, and that their review is accurate.
As a reader of reviews, there is a process I follow to evaluate their likely credibility(just as I do with any information). I take what I know about customer behaviors (I used to work for a company that specialized in customer loyalty) and I read reviews with that eye. Suspicious: short one-liner comments; a top score/low score with few specifics; gushing platitudes; no customer “story”; etc.
On some sites (incl. Yelp) you can look up the individual reviewer, see their pattern of scores and read their other reviews — this also helps to determine who’s a shill/”review whore” and who’s a regular person. IMO profiles with “too few” (100) reviews are potentially suspect. Profiles without icons or personal information are more suspect that profiles with that info. Etc.
All this said: I still find online review sites to be extremely helpful. I refer to TripAdvisor, Yelp and Amazon reviews frequently.
You’re right that online ratings now drive additional responsibilities for businesses. Every business needs to regularly check on its online ratings/reviews, ask for suspicious reviews to be removed, respond to bad reviews (on the site or to the individual customer), etc.
Businesses need to understand — with the internet, all the actions of that business and business leader/owners are more visible and transparent than *ever before*. “Authenticity” is a big thing for many customers. A policy that looks dodgy, political donation to the “wrong” group, will get publicly reviewed.
And to those customers who complain — they see it as a valid criticism of the business, not an unrelated topic — because it points to the business’ inauthenticity — that “you are not as you appear to be,” ie the business is untrustworthy, putting up a false front. And from that customer’s perspective – if the business is “faking it” in one area — what else are they misrepresenting?
The appearance of dissembling pushes a deep human social hot button — as social animals, we’re genetically evolved to evaluate the trustworthiness of others (who “deserves” our ongoing association, sharing of resources etc). Humans in all societies have ways to determine trustworthiness and to punish those deemed unreliable; that deep human impulse is now directed at organizations, not just individuals, and has new tools like the internet.
*All* a business’ actions need to be consistent with its brand. It’s harder to talk out of two sides of one’s mouth, now that one’s customers (and employees) can find each other together online and compare notes.
Clay Shirky’s recent book “Here Comes Everybody” is a great discussion of this topic (the internet’s power shift between formal organizations like businesses and individuals/grassroots efforts).
Whew — long comment, but it’s an interesting topic.
A major problem with Yelp’s deletion policy is that for many customers, a business’s donation to Prop 8 cannot be decoupled from the overall experience of that customer.
For example, Yelp includes reviews of doctors and other help professionals. Gay and lesbian Yelpers may feel unable to receive quality health care from a doctor who donated to Prop 8, and a negative review based on Prop 8 is absolutely justified. These businesses aren’t just producing widgets and cogs here.
By the way, the notion that you are “too far away” to comment on whether El Coyote did right by its clientele is not credible. Prop 8 was a national issue, with national funding and national media attention.
I got to your post visiting Nilofer pages…
Very interesting indeed. I observed similar side effect on Obama’s web site and to a certain extend in Dell Ideastorm portal.
Here is my post with some data : http://blog.ecairn.com/2009/01/22/dell-ideastorm-and-obama/
What is clear to me is that businesses need marketing strategy before jumping in social web, building social sites, relying on social reviews and so on.
Best