Measuring the Unmeasurable

Key Scientific Challenges, Entry #3: Microeconomics and Social Systems

On January 27 we announced the kick-off of our 2010 Key Scientific Challenges Program.  To highlight the scientific challenge areas included in the program, we launched a series of guest blog posts earlier this month on Yodel Anecdotal. Read our previous post on privacy and security, “Data, Data Everywhere, but How to Keep it Safe.”

Another big challenge are Yahoo!’s research scientists are continually examining is microeconomics and social systems. In this entry, Sharad Goel from Yahoo! Labs shares some thoughts on how Yahoo! is tackling the new opportunities for research into the social sciences that the Web is making possible and why it’s a fascinating field.

What do your friends really know about you? How much do they influence your decisions? How often do we stray from the cultural herd? How do groups organize to solve complex problems?

Answers to such fundamental questions about social behavior have often eluded us. With microscopes we peered into the intangibly small building blocks of life, and with telescopes we found our place in an unimaginably expansive universe. But without the tools to faithfully document human activity—a challenge that by comparison seems so palpable—we had no way to investigate the inner workings of our own communities. Now with an explosion of information on every aspect of our everyday existence—from what we buy, to where we travel, to whom we know—we can measure what until quite recently was thought unmeasurable. In the Microeconomics and Social Systems Group at Yahoo! Labs, we are using this proliferation of data to explore how societies function. It’s a fascinating area of study that is just beginning to shed light on new layers of human behavior, making it a perfect fit for the Key Scientific Challenges Program.

In a recent study that’s garnered some attention, for example, we asked, “How eccentric are people?” Looking at consumer preferences across movies, music, and web browsing, we came to the surprising conclusion that ordinary people have pretty extraordinary tastes. In particular, we found that typical Netflix and Yahoo! Music users regularly watch movies and listen to songs that are not even available in the largest brick-and-mortar retailers. This result not only challenges stereotypes about people blindly following the herd, but also highlights the importance of offering consumers broad selection. That is, specialty products may dramatically boost user satisfaction by providing buyers the convenience of “one-stop shopping” for both their mainstream and niche interests.

In other work, we used web search queries to forecast the commercial success of movies, songs, and video games. Weeks, sometime even months, before a movie opens or a video game is released, one can find traces of pent up consumer demand in the search query logs. We found that these telltale signs of early interest are remarkably good predictors of future success. The catch? Although the search logs do reflect user intent, more mundane indicators, such as production budgets and reviewer ratings, perform equally well at forecasting sales. Thus, the benefit of web search as a prediction tool may have less to do with its superiority over other methods than with its generality, low cost, and real-time nature.

At a time when we are drowning in data, at Yahoo! Labs we’re asking a simple question: what can you do with it? The answer is limited only by our imaginations.

Back in July, I posted here about the $1 million Netflix Prize and the amazing accomplishment of one of our senior research scientists, Yehuda Koren. At the time, Yehuda and a team of academic and industry researchers from around the world had cracked the mythical 10-percent threshold in the contest – making their team, BellKor’s Pragmatic Chaos, contenders for the grand prize. In a dramatic turn of events at the finish line, the team ended up tied with another group of researchers that submitted a similar improvement later that same day.

But today, at a press conference in New York, after verifying the results with their top technical brass, Netflix officially declared Yehuda and his team the grand prize champions. To achieve this honor, they bested 41,305 teams from 186 countries.

To reiterate, this was no easy feat. Every person in the world has a different taste in movies. Yehuda, for example, loves the Godfather, but is not much of a James Bond fan (whether it’s Connery, Moore, Daulton, Brosnan, Craig, or the other guy). And it’s not just Yehuda — everyone has personal tastes that don’t perfectly correspond to box office results, DVD sales, or Academy Awards. As a result, figuring out what movies should be recommended to you is an intense scientific problem.

In fact, it is such an impressive piece of research that Yehuda’s paper detailing one of his main contributions to the prize-winning team won the best paper award at KDD-09 (otherwise know as 15th ACM SIGKDD Conference on Knowledge Discovery and Data Mining).

It’s too complicated a paper to summarize here with any justice, but let me give you the gist: Yehuda figured out that the way people rate movies on Netflix changes over time and, in some cases, from day to day. In other words, not only do my tastes “mature” and shift, but so do my moods. For example, on a Monday, perhaps out of frustration with having to be back in the office, maybe I’m a little harsh on the movie I watched Sunday night, which I give a 1-star rating in spite of it not being that terrible. On the other hand, that film I saw with my kids when we were on vacation – the one that had them laughing all night — gets a warm-hearted 5 stars when it wasn’t exactly an aesthetic masterpiece. Yehuda laid down the research to mathematically model this phenomenon and it’s a major reason why his team cracked the code to get over 10-percent.

Ultimately, this exactly why we founded Yahoo! Labs — to encourage Yahoos to think outside of our own sandbox and contribute to industry-wide technical and scientific challenges that will some day make your online life easier. Sometimes that means just making sure there’s less spam in your email inbox and — in this case — making sure that your Netflix queue is filled with movies you’re really going to like.

• New York Times – Netflix Awards $1 Million Prize and Starts a New Contest
• Forbes – The Netflix R&D Game
• Fortune – Box office boffo for brainiacs: The Netflix Prize

Prabhakar Raghavan
Head of Yahoo! Labs