Category Archives: recommendation-engine

The YouTube recommendation engine: a lesson in transparent analytics

Recommendation engines are all the rage. Whether it is in the realm of social commerce (see IBM Coremetrics Intelligent Offer) or in location-based social applications like Foursquare.

As the attention span of the browsing population shrinks below that of your average goldfish, so the need to create razor-sharp, perfectly honed navigation systems increases. There’s a demand on publishers to use whatever information they have to provide a more contextualized browsing experience.

That’s all well and good, but have you ever looked at a recommendation and wondered what on earth was the system thinking when it picked it? You’ve spent months on the site exploring hardcore thrash metal so why on earth are you being offered a book on floral knitting patterns?

I just went onto Youtube and noticed that they have actually got pretty transparent with their recommendation engine:

Notice the ‘because you watched’.

As we use more analytics systems to build interfaces, being explicit about how decisions are made becomes increasingly important: 

Show what determined the recommendation: This answers the question of why on earth am I seeing this recommendation? In this instance YouTube bases this on what you have watched previously, but this could just as well relate to what others with similar interests have liked (the Amazon approach).

Allow you to interact with the recommendation: YouTube allows you to remove recommendations from the list that you don’t think are appropriate. One thing it doesn’t do is spell out whether that feedback is factored into future recommendations. Some systems (such as Pandora and Netflix) use a thumbs-up/down or rating system with the implicit understanding that this information will be fed into the calculations of future recommendations. As James Taylor, the Decision Management expert pointed out to me some years ago, recommendation engines have their limit. If I booked a once-in-the-lifetime trip to Bermuda last year, there’s no point in showing me vacations to Antigua six months later. Allowing me to vote this kind of recommendation down can help systems disentangle one-shot whims from longer term patterns of behavior.      

The question of privacy: Being transparent about analytics systems and and how exactly visitors are being tracked can go a long way to allay the growing public fears around the growing mountain of data produced by the internet in general and social networking sites in particular. Indeed, here in California there has been considerable press around a bill to increase the privacy of social networks. Justin Brookman, director of the Project on Consumer Privacy at the Center for Democracy and Technology has said, “I think the idea of telling people what is going on and giving them control over their information from the beginning is a good idea for social networks and others places as well”. Privacy advocates are asking publishers to be more open about how data is being used.

As user interfaces become more reliant on analytics tools to offer a more personalized experience, there are significant advantages to displaying upfront exactly why we are being shown the recommendations we see.

Foursquare to use predictive analytics to beat Facebook?

There’s a growing battle in the location-based services business between Foursquare and Facebook. Foursquare, with its past emphasis on gaming and status building (who wants to be the mayor of the local laundromat?) is now focusing on a more functional aspect: helping people decide where they should go next. According to a report in Brandweek (backed up by this article on a recent job ad), Foursquare sees offering recommendations as its chance to avoid being squeezed out of existence by Facebook, who, with over 500 million users, is the ostensible gorilla in the room.

How does it plan to do this? Brandweek suggests it will adopt predictive services which are common on sites like Amazon and Netflix:

"Those services crunch behavior data—what movies you watch and books you read—to suggest new products. Foursquare wants to do the same, only with recommendations of real-world activities."

For instance, let’s say you are a sushi freak living in Chicago who’s been active on Foursquare for the last year. You’ve been using Foursquare to capture badges for most of the top local Japanese eateries. Foursquare can see your penchant for fine sushi in the windy city and look across its network for others in your area who share the same passion. It realizes that there is a new joint downtown and can suggest you check this out.

How does this crunching work? The data is mined along a process which runs something like this for each individual visitor:

  • What are the past actions you have recorded
  • What patterns can be determined from your actions
  • Who else in the network is like you
  • Where are the gaps between your actions and their actions?
  • Offer as predictions these actions that people like you have performed

Note, this obviates the need for a user to fill in a vast registration form listing all their likes and interests. The system can figure this out by looking at past behavior.

In terms of making predictions, systems need to be smart enough to factor in elements that can cause shifts in our patterns of behavior:

  • Seasonality (no taste for raw fish when snowing)
  • Change in tastes (eg. pregnancy pushes sushi off the menu)
  • Removing system bias (eg. not only favoring well-established popular places, but allowing new entrants a chance to prove themselves)

Whether Foursquare makes a concerted move in this direction remains to be seen, but as web and mobile applications creep further into every aspect of our existence (with their inherent ability to track behavior), expect to see an increasing use of business intelligence and predictive analytics to create smarter systems offering us more relevant information.