Back in July, I posted here about the $1 million Netflix Prize and the amazing accomplishment of one of our senior research scientists, Yehuda Koren. At the time, Yehuda and a team of academic and industry researchers from around the world had cracked the mythical 10-percent threshold in the contest – making their team, BellKor’s Pragmatic Chaos, contenders for the grand prize. In a dramatic turn of events at the finish line, the team ended up tied with another group of researchers that submitted a similar improvement later that same day.

But today, at a press conference in New York, after verifying the results with their top technical brass, Netflix officially declared Yehuda and his team the grand prize champions. To achieve this honor, they bested 41,305 teams from 186 countries.

To reiterate, this was no easy feat. Every person in the world has a different taste in movies. Yehuda, for example, loves the Godfather, but is not much of a James Bond fan (whether it’s Connery, Moore, Daulton, Brosnan, Craig, or the other guy). And it’s not just Yehuda — everyone has personal tastes that don’t perfectly correspond to box office results, DVD sales, or Academy Awards. As a result, figuring out what movies should be recommended to you is an intense scientific problem.

In fact, it is such an impressive piece of research that Yehuda’s paper detailing one of his main contributions to the prize-winning team won the best paper award at KDD-09 (otherwise know as 15th ACM SIGKDD Conference on Knowledge Discovery and Data Mining).

It’s too complicated a paper to summarize here with any justice, but let me give you the gist: Yehuda figured out that the way people rate movies on Netflix changes over time and, in some cases, from day to day. In other words, not only do my tastes “mature” and shift, but so do my moods. For example, on a Monday, perhaps out of frustration with having to be back in the office, maybe I’m a little harsh on the movie I watched Sunday night, which I give a 1-star rating in spite of it not being that terrible. On the other hand, that film I saw with my kids when we were on vacation – the one that had them laughing all night — gets a warm-hearted 5 stars when it wasn’t exactly an aesthetic masterpiece. Yehuda laid down the research to mathematically model this phenomenon and it’s a major reason why his team cracked the code to get over 10-percent.

Ultimately, this exactly why we founded Yahoo! Labs — to encourage Yahoos to think outside of our own sandbox and contribute to industry-wide technical and scientific challenges that will some day make your online life easier. Sometimes that means just making sure there’s less spam in your email inbox and — in this case — making sure that your Netflix queue is filled with movies you’re really going to like.

• New York Times – Netflix Awards $1 Million Prize and Starts a New Contest
• Forbes – The Netflix R&D Game
• Fortune – Box office boffo for brainiacs: The Netflix Prize

Prabhakar Raghavan
Head of Yahoo! Labs

If you’ve seen “The Godfather,” chances are you might like other Marlon Brando movies. Or films about gangsters. Or those directed by Francis Ford Coppola. But will you like “Napoleon Dynamite”?

This is the central problem posed by the Netflix Prize. Netflix is offering $1 million in prize money to anyone who can substantially improve (by more than 10 percent) the accuracy of its movie recommendation engine. While Netflix suggests movies based on your ratings history, the company isn’t satisfied with how well it can predict what you’ll like.

At Yahoo! Labs, this is just the kind of crazy difficult problem we love to take on. For scientists, it’s a pure challenge, requiring deep study and experimentation across a variety of fields, such as machine learning and data mining.

And for Yahoo! as a whole, these types of scientific problems also happen to be a critical element of what we most want to succeed at: connecting you with the content and information you most want in your life – even if you don’t know it yet.

That’s why we couldn’t be happier to pass along the news that Yehuda Koren, one of our scientists at Yahoo!’s Israel Lab, is part of the first qualifying team for the Netflix Prize.

Yehuda’s team, BellKor’s Pragmatic Chaos, reached first place on the Netflix Prize leaderboard on June 26, with an improvement of 10.05 percent. Achieving a more than ten percent improvement in the quality of movie recommendations is no drop in the bucket. It took Yehuda and his teammates three years to achieve and no other team has matched it yet.

Congratulations to Yehuda and his team. In the past few weeks alone, in addition to the Netflix Prize, Yehuda and his colleagues also received best paper prizes at two of the most important scientific conferences (ACM SIGMOD and ACM SIGKDD) for computer science and the Internet. Yahoo! researchers Christopher Olston, Shubham Chopra, Utkarsh Srivastava, Ashwin Machanavajjhala and Bee-Chung Chen, were also recognized for contributions to the science of how to better query and mine data, which will ultimately make it easier for you to get things done on the Web and beyond.

We may not yet have solved every problem the Internet has thrown our way, but at the very least, you should start feeling a lot more confident about those movies in your Netflix queue.

Prabhakar Raghavan
Head of Yahoo! Labs

Did you know that humans have only used verbal language for the past 50,000 years – a virtual blink of the eye in evolutionary time? This got me wondering how people communicated before language. Since we’ve been thriving on this planet for 160,000 years (or millions more, depending on when you start the “human” clock), how exactly how did we express ourselves? And do we hang on to old non-verbal habits today?

MIT Professor Alex “Sandy” Pentland stopped by to discuss this very topic with us yesterday as the latest in a series of Big Thinkers lectures at Yahoo! Research. He also shared insights into the expansive research he’s done in his career on what he calls “honest signals,” the non-verbal clues and patterns that reveal everything from how people interact on the job to who they date and whether or not they’re going to buy a given product or service when the telemarketer calls.

Professor Pentland is leading the exploration of this new realm of social science – designing new ways to collect data about our non-verbal communication patterns and analyzing the ever-growing mountains of data we’re creating when we use new technologies (like the Internet and especially mobile phones). Pentland’s work is aimed at making the ways we communicate without language a first class part of how we see and understand the world, and, together with his colleagues and students, he’s applying these new ideas to everything from predicting which speed-daters are going to get together, to tackling public health issues, to what makes companies and creative teams productive (here’s a hint: face time at the water cooler actually pays off!).

In the video below, I interviewed Prof. Pentland about his work. In a week or so, his full lecture will be available at the Yahoo! Research Big Thinkers site, but in the mean time, we hope you enjoy this preview. And for those iPhone and Blackberry users out there, you may want to download the CitySense app the next chance you get for a hands-on experience with the types of data and research Professor Pentland is working on.

Since 2006, we’ve had 20 “Big Thinkers” from UC Berkeley, Stanford, Harvard, Cornell, Carnegie Mellon, and others talk to us about everything from economic theory and marketplace design to online amateur media production and other forms of user generated content. They’re a lot of fun and we hope to share more of these interesting discussions and ideas with you as often as we can.

Elizabeth Churchill
Principal Research Scientist, Yahoo! Research

On this day 76 years ago, the U.S. Congress made a proposal that had brewers, vintners, bartenders, wine snobs, and fraternity brothers clinking glasses across the country — the 21st amendment. It would ultimately end Prohibition, shutter the speakeasy, kill profits for organized crime, and make hangovers perfectly legal in all 48 states. Here’s what we toasted to this week:

• Mobile makeover: The gang at Yahoo! Mobile rolled out a beta program this week for a comprehensive new mobile service known as, simply enough, Yahoo! Mobile. It will come to a cell phone near you next month on the mobile Web, as an iPhone app, and as an app developed for smartphones from Nokia, RIM, Samsung, Sony Ericsson, and Motorola, as well as those running Windows Mobile. It will replace Yahoo! Go and packages a variety of great mobile offerings into one location — search, email, messenger, address book, calendar, news, social networks, and even managing all your favorite non-Yahoo! content. To sign up for the beta, just enter http://mobile.yahoo.com in your phone’s browser (and be patient — it make take up to a few weeks to be admitted to the program). Here’s the press release.
• Ogling at the Oscars: Let’s face it — next to finding out the Academy Award winners, we care most about how everyone looks on the red carpet. That’s why the Yahoo! Search team has built a new Oscar-related carousel for Yahoo! Image Search. After Sunday’s show and parties, enter keywords related to Oscar-nominated films or actors and you’ll find a rolling gallery of the newest photos above the regular search results. Toggle your way through Slumdog Millionaire, Mickey Rourke, and Kate Winslet eye candy. More here.
• Watch with me: Raise your hand if you’ve ever IMed a video link to share a laugh with a friend, only to be disappointed when it takes them a few hours to finally get around to watching it. Wouldn’t it be cooler to watch that video together, at the same time? The scientists and engineers at Yahoo! Research think so and have just released a piping hot new version of Zync, a synchronized video player for Yahoo! Messenger 9.0 for Windows. You can now simultaneously view videos on Yahoo! Video, Flickr and, of course, YouTube — with Yahoo! Music and Yahoo! News to come next week. Just paste the video link into your conversation and click the “Watch With Me” button — share, watch, chat. No download necessary. Now go watch the woman who missed her flight in Hong Kong with someone.

Subscribe to the RSS feed (or add it to My Yahoo!) to get this Product Pulse every week.

What do climate change, Kevin Bacon, the snowy tree cricket, Al Qaeda, HIV, the World Wide Web, and your address book have in common? They’ve all played a role in a major science discovery –- the hidden language of networks.

“CONNECTED: The Power of Six Degrees” is a new BBC documentary that unfolds the science behind the popular trivia game “Six Degrees of Kevin Bacon,” whose notion that anyone on the planet can be connected in just six steps of association was supposed to be an urban myth.

The film follows two young scientists, Harvard’s Laszlo Barabasi and Yahoo! Research’s very own Duncan Watts, as they work to uncover the pervasive law that nature uses to organize itself. By studying vast natural and man-made networks — from the connections between Hollywood actors to the nervous system of a worm, the U.S. electric power grid and the WWW — they discover that diverse systems share a common blueprint. It takes us from the hunt for Saddam Hussein to the front-lines of cancer research and shows that the Six Degrees principle doesn’t just relate to people but also to viruses, web pages, neurons, species, molecules, and even fashion.

Watts, a former Australian Navy officer with a passion for extreme rock climbing, was a professor at Columbia University at just 29. He launched the explosion in the new science of networks while studying crickets and the mechanism that allows them to chirp in unison. He’s the author of “Six Degrees: The Science of a Connected Age.” Yodel Anecdotal will post a full interview with Watts soon.

“CONNECTED: The Power of Six Degrees” premieres tomorrow night on the Science Channel. Check your local cable listings for times.

Here’s the trailer:

Nicki Dugan
Blog Editor

We’ve just heard from the National Academy of Engineering (NAE) and it’s my great pleasure to share that our very own Head of Yahoo! Research, Dr. Prabhakar Raghavan, has been elected to join their esteemed body of members for his significant contributions to algorithms and the structure of the Web. This outstanding distinction is a first at Yahoo! (and I imagine, not the last) and is among the highest professional honors in the engineering community. This recognition places him amongst the engineers who have made the most impact, across the United States, on technology that has profound effects on society. One of the highest forms of acknowledgement an engineer could get at a national level.

Over the past couple of years, Prabhakar has led and helped build our Yahoo! Research organization to examine some of the most complex problems facing the Internet. His extensive expertise in search algorithms (he literally wrote the book on it), understanding the fundamental structure of the Web, and the social phenomena emerging from it, continues to play an important role as we work to make Yahoo! the starting point on the Web. Yahoo! Research is charged with a simple but fundamentally powerful vision: “Invent the new sciences underlying the Internet and the new ineteractive media.”

Nothing is more inspiring to me than a collection of brilliant minds. Over the past four years, our research team has grown from one small lab into a world-class research organization with seven locations in four countries (and we’re not stopping there). Under Prabhakar’s leadership, Yahoo! Research is focused on creating and exploring groundbreaking technologies that will improve our users’ online experience, from understanding communities and the new economics of the Web to the new emerging science of computational advertising, and economics and social systems to advanced search technologies.

Thanks for making us proud, Prabhakar.

Usama Fayyad
Chief Data Officer & EVP of Research & Strategic Data Solutions

Photo from sigir2007.