But today, at a press conference in New York, after verifying the results with their top technical brass, Netflix officially declared Yehuda and his team the grand prize champions. To achieve this honor, they bested 41,305 teams from 186 countries.

To reiterate, this was no easy feat. Every person in the world has a different taste in movies. Yehuda, for example, loves the Godfather, but is not much of a James Bond fan (whether it’s Connery, Moore, Daulton, Brosnan, Craig, or the other guy). And it’s not just Yehuda — everyone has personal tastes that don’t perfectly correspond to box office results, DVD sales, or Academy Awards. As a result, figuring out what movies should be recommended to you is an intense scientific problem.

In fact, it is such an impressive piece of research that Yehuda’s paper detailing one of his main contributions to the prize-winning team won the best paper award at KDD-09 (otherwise know as 15th ACM SIGKDD Conference on Knowledge Discovery and Data Mining).

It’s too complicated a paper to summarize here with any justice, but let me give you the gist: Yehuda figured out that the way people rate movies on Netflix changes over time and, in some cases, from day to day. In other words, not only do my tastes “mature” and shift, but so do my moods. For example, on a Monday, perhaps out of frustration with having to be back in the office, maybe I’m a little harsh on the movie I watched Sunday night, which I give a 1-star rating in spite of it not being that terrible. On the other hand, that film I saw with my kids when we were on vacation – the one that had them laughing all night — gets a warm-hearted 5 stars when it wasn’t exactly an aesthetic masterpiece. Yehuda laid down the research to mathematically model this phenomenon and it’s a major reason why his team cracked the code to get over 10-percent.

Ultimately, this exactly why we founded Yahoo! Labs — to encourage Yahoos to think outside of our own sandbox and contribute to industry-wide technical and scientific challenges that will some day make your online life easier. Sometimes that means just making sure there’s less spam in your email inbox and — in this case — making sure that your Netflix queue is filled with movies you’re really going to like.

• New York Times – Netflix Awards $1 Million Prize and Starts a New Contest
• Forbes – The Netflix R&D Game
• Fortune – Box office boffo for brainiacs: The Netflix Prize

Prabhakar Raghavan
Head of Yahoo! Labs

This is the central problem posed by the Netflix Prize. Netflix is offering $1 million in prize money to anyone who can substantially improve (by more than 10 percent) the accuracy of its movie recommendation engine. While Netflix suggests movies based on your ratings history, the company isn’t satisfied with how well it can predict what you’ll like.

At Yahoo! Labs, this is just the kind of crazy difficult problem we love to take on. For scientists, it’s a pure challenge, requiring deep study and experimentation across a variety of fields, such as machine learning and data mining.

And for Yahoo! as a whole, these types of scientific problems also happen to be a critical element of what we most want to succeed at: connecting you with the content and information you most want in your life – even if you don’t know it yet.

That’s why we couldn’t be happier to pass along the news that Yehuda Koren, one of our scientists at Yahoo!’s Israel Lab, is part of the first qualifying team for the Netflix Prize.

Yehuda’s team, BellKor’s Pragmatic Chaos, reached first place on the Netflix Prize leaderboard on June 26, with an improvement of 10.05 percent. Achieving a more than ten percent improvement in the quality of movie recommendations is no drop in the bucket. It took Yehuda and his teammates three years to achieve and no other team has matched it yet.

Congratulations to Yehuda and his team. In the past few weeks alone, in addition to the Netflix Prize, Yehuda and his colleagues also received best paper prizes at two of the most important scientific conferences (ACM SIGMOD and ACM SIGKDD) for computer science and the Internet. Yahoo! researchers Christopher Olston, Shubham Chopra, Utkarsh Srivastava, Ashwin Machanavajjhala and Bee-Chung Chen, were also recognized for contributions to the science of how to better query and mine data, which will ultimately make it easier for you to get things done on the Web and beyond.

We may not yet have solved every problem the Internet has thrown our way, but at the very least, you should start feeling a lot more confident about those movies in your Netflix queue.

Prabhakar Raghavan
Head of Yahoo! Labs

Our research found that web searchers are regularly affronted by articles, blogs, facts, and pages that contain perspectives directly contradicting their own personal beliefs and values –- whether political, religious, economic, scientific, philosophical, etc. If consumers have the freedom in whether they navigate to the HuffingtonPost.com or FOXNews.com, why not extend that same choice to search? Until today, no other search engine could provide this level of personalization –- ensuring that consumers can search with the utmost confidence, knowing that they won’t be antagonized by their results.

Ideological Search, built on Yahoo! BOSS, is the result of extensive research conducted by virtually every scientist at Yahoo!. The team applied the latest research from the fields of sentiment analysis, intent detection, eye tracking, clustering and empathic reasoning to create this revolutionary service. We also found that adaptations were required within existing technologies to ensure ideologically-biased results. For example, Pig, the large-scale data processing environment, was not compatible with all beliefs, particularly among vegetarians. As a result we developed a sister codebase called Tofu, which proved to be more flexible and gelatinous, albeit less optimized.

To give Ideological Search a test drive, type in “global warming” or “stimulus package” and see for yourself. For more background on how we developed Ideological Search, visit the Yahoo! Labs site.

We hope this is the start of a more peaceful, conflict-free Yahoo! Search experience for you.

Try Ideological Search today!

Prabhakar Raghavan
Head of Yahoo! Labs

Academic research is facing new challenges in today’s Internet age. Universities often don’t have the equipment –- hardware and software -– to maintain in-depth research at Internet scale. Academic researchers are limited in the research they can conduct, and this, ultimately constrains the amount of large-scale Web innovations coming to the marketplace.

Here at Yahoo!, we believe in open and collaborative research as the best way towards building the next generation of the Web. As part of our dynamic Academic Relations program, we’re teaming up with academia, as well as other companies and governments across the globe, to invest in and pool together the large-scale computers that will let researchers conduct truly breakthrough work on cloud computing and data storage systems.

The HP/Intel/Yahoo! Cloud Research Testbed is a significant step in that it will not only allow researchers to run applications and data on large-scale supercomputers, they will be able to experiment and conduct research on a massive scale. It’s like letting them simulate a true Web environment and that’s exactly what you need to ensure which ideas will work in the wild.

Joining us in this first-ever large-scale international consortium are Infocomm Development Authority of Singapore (IDA), the University of Illinois at Urbana-Champaign (UIUC), and the Karlsruhe Institute of Technology in Germany, with contributions from the National Science Foundation (NSF) as well.

This is all part of Yahoo!’s overall focus and investment in cloud computing and data infrastructure. We recently announced the formation of the Cloud Computing and Data Infrastructure Group (CCDI), a new group dedicated to building out our next-generation cloud infrastructure.

In addition, in November 2007, Yahoo! deployed a supercomputing-class data center, called the M45, for cloud computing research; Carnegie Mellon University was the first institution to take advantage of this supercomputer. In March 2008, Yahoo! announced an agreement with Computational Research Laboratories (CRL) to jointly support cloud computing research in India. The CRL supercomputer is one of the world’s top five supercomputers and is the first supercomputer available to academic institutions in India.

I speak for our many research scientists when I say we’re excited about this open testbed and being able to collaborate with leaders who share our same vision. The HP/Intel/Yahoo! Cloud Research Testbed is a truly global research effort, and more partners and researchers will be invited to join and participate in the program when all of the systems are up and running later this year. The sky is the limit from here on out.

Prabhakar Raghavan
Head of Yahoo! Research

Photo from Nicholas_T

What if your favorite news site had a search engine you could turn to when news breaks that not only surfaces the best articles on that topic, but also the most relevant content from across the Web? Or what if your search engine understood your social relationships or your site preferences and presented results with that in mind? When you’re searching for new music, wouldn’t it be nice if there was a search engine that knew your and your friends’ taste in music, drew in reviews from across the Web, and recommended songs you might enjoy?

Today we’re announcing a service that brings ideas like these a bit closer to reality.

Yahoo! Search BOSS, which stands for Build Your Own Search Service, is a new open Web services platform from Yahoo! Search that will allow developers and companies of all sizes to leverage one of the most valuable assets on the Web, the Yahoo! Search infrastructure. It lets them realize their own vision of what a search experience should be, enabling unlimited mash-ups and disrupting the search landscape.

BOSS will enable developers and companies to easily enter the search industry –- without large capital or resource expenditures — unleashing a wave of search innovation beyond any one of today’s search principals. Now anyone –- your favorite shopping site, an entrepreneur with a great idea or a start-up developer –- can tap into Yahoo!’s technology to create their own Web search experience without the infrastructure or talent resources needed to create one from scratch.

The result is that you’ll have more choice and flexibility in finding whatever you’re looking for on the Web –- just as you now have far more programming options with cable TV than when there were just three networks. (For examples of how BOSS technology can be used or to get started building your own search engine, visit the Yahoo! Developer Network.)

Stay tuned for more to come on how we are fundamentally changing the way you search.

Prabhakar Raghavan
Chief Strategist for Yahoo! Search