The YouTube recommendation engine: a lesson in transparent analytics

Recommendation engines are all the rage. Whether it is in the realm of social commerce (see IBM Coremetrics Intelligent Offer) or in location-based social applications like Foursquare.

As the attention span of the browsing population shrinks below that of your average goldfish, so the need to create razor-sharp, perfectly honed navigation systems increases. There’s a demand on publishers to use whatever information they have to provide a more contextualized browsing experience.

That’s all well and good, but have you ever looked at a recommendation and wondered what on earth was the system thinking when it picked it? You’ve spent months on the site exploring hardcore thrash metal so why on earth are you being offered a book on floral knitting patterns?

I just went onto Youtube and noticed that they have actually got pretty transparent with their recommendation engine:

Notice the ‘because you watched’.

As we use more analytics systems to build interfaces, being explicit about how decisions are made becomes increasingly important: 

Show what determined the recommendation: This answers the question of why on earth am I seeing this recommendation? In this instance YouTube bases this on what you have watched previously, but this could just as well relate to what others with similar interests have liked (the Amazon approach).

Allow you to interact with the recommendation: YouTube allows you to remove recommendations from the list that you don’t think are appropriate. One thing it doesn’t do is spell out whether that feedback is factored into future recommendations. Some systems (such as Pandora and Netflix) use a thumbs-up/down or rating system with the implicit understanding that this information will be fed into the calculations of future recommendations. As James Taylor, the Decision Management expert pointed out to me some years ago, recommendation engines have their limit. If I booked a once-in-the-lifetime trip to Bermuda last year, there’s no point in showing me vacations to Antigua six months later. Allowing me to vote this kind of recommendation down can help systems disentangle one-shot whims from longer term patterns of behavior.      

The question of privacy: Being transparent about analytics systems and and how exactly visitors are being tracked can go a long way to allay the growing public fears around the growing mountain of data produced by the internet in general and social networking sites in particular. Indeed, here in California there has been considerable press around a bill to increase the privacy of social networks. Justin Brookman, director of the Project on Consumer Privacy at the Center for Democracy and Technology has said, “I think the idea of telling people what is going on and giving them control over their information from the beginning is a good idea for social networks and others places as well”. Privacy advocates are asking publishers to be more open about how data is being used.

As user interfaces become more reliant on analytics tools to offer a more personalized experience, there are significant advantages to displaying upfront exactly why we are being shown the recommendations we see.