The failure of the wisdom of crowds

One of the key assumptions of Web 2.0 and emerging systems build upon assumptions of immersive wisdom of crowds is that a large body of amateurs will produce knowledge as good or better than a small group of experts. Rotten Tomatoes gives us an interesting case study of how flawed this idea can be.

Rotten Tomatoes gives both critic ranking and ranking by the audience. Very often the two go up and down together but sometimes you see some strange divergence which raises doubts about the value of the crowd to judge culture and art. For example above is the review of the brilliant Werner Herzog's "Cave of Forgotten Dreams", a brilliant 3D documentary we saw several times this year. Caves is not a movie for children, or bored stoned teen agers, it is not a movie for people who want moveis with big explosions and sex scenes.

Professional film critics universally saw the genius of this film, giving it a critics rating of 100%. But the audience, though positive, is dragged down by the ratings of people who probably think moveis are better by having bigger explosions or more sex scenes. The audience rating is only 75%.

Now look at a childish silly movie like "V for Vendetta"

If you follow the audience ranking V is a far better movie than Cave, with a score of 88% over 75%. But if you look at the critics ranking it becomes a very different story, all critics give V a 73%, but looking at the top critics V gets a 46% with 7 rotten to 6 Fresh.

Its also interesting how much energy the critics give to the two movies. V has over 800,000 user reviews, Cave has about 5,300. Both films have 13 top critic reviews.

So not only do we see a huge difference between the professional view and audience view on these movies, but also a massive motivation between the two movies.

It is easy to dismiss the opinions of critics as elitist and trust in the opinions of masses. But the case of these two movies would make anyone with a deeper respect for film to wonder. Its just as easy to dismiss the crowd sourcing results as the ignorant speaking up on things they are not fit to judge. One might grant fans can vote for V because they like it, but the opinions of ordinary users is not so great for a great film live V. But Wikipedia raises doubts about this anti-crowd sourced view.

We think a few things should be kept in mind:

• The audience rating tells you popularity among internet users, a critic review is something else looking more to cultural value. In this way the audience reviews do tell you something. V for Vendetta was a fairly popular film, Cave of Forgotten Dreams was more of an art house film. On the other hand if you had to make a time capsule and you had a government department to determine 10 films to put in to it the fan rankings of V would be neglected for the high respect the film has among critics. Crowds are good at telling you what they like and know. Always keep in mind the crowd that is crowd sourcing your data creation.
• A voting a comment system is not a knowledge creation community. Unlike Wikipedia there is not community of discourse of self government evolving in Rotten Tomatoes. Its pretty passive actually, you log in with your Facebook account, you go to a film, you leave some stars and some comments and press submit. Wikipedia demands much much more for its contributors. Your input is likely to be challenged, altered, struggles emerge, compromises are reached, values are developed, people vote and hold roles, groups have to try and see each others point of view. Wikipedia has evolved a real community of knowledge creation, Rotten Tomatoes is an online poll.

For current sites thinking that adding Facebook to your site will make it a community, think again. Rotten Tomatoes is one of the most Facebook integrated major sites out there, but the users are still limited to nothing much more than creating comments. You don't have the level of participation and community formation you see in the Blogosphere or Wikipedia.

Facebook can't just add community to your site. One of the things to note about Facebook is that it asks that you know people before making connections in Facebook. This is a tacit brilliant admittance by Facebook that it can't provide community formation.