Prabhakar Raghavan talks to Wired UK about the growth and innovation of Yahoo! Labs and how our scientists support Yahoo!’s Product Strategy.

Yahoo! Labs kicked off 2012 with a feature story on Wired UK this week: an interview with Chief Strategy Officer, Executive Vice President and Head of Yahoo! Labs Prabhakar Raghavan.

The article focuses on key aspects of the evolution of Yahoo! Labs and the work our scientists and researchers do to support our Product Strategy, including personalization/ C.O.R.E. and some recent product launches (IntoNow, Livestand, Yahoo! Activity). The article also includes the announcement of the recent EU roll-out of Yahoo! Activity.

Key quotes from the article include:

• ‘Raghavan says that the key to [Yahoo! Labs’] success has been bringing together machine and human intelligence, and specifically, gathering minds from a wide range of disciplines.’
• ‘Personalisation is key and Yahoo! Labs is utilising technology that it has long used for its homepage to power the new tablet tools. Three years ago, it created the Content Optimisation and Relevance Engine (C.O.R.E.), which uses computer algorithms and human expertise to pick, for example, the news stories that you will see when you visit the Yahoo! homepage.’
• ‘But what of the future for Yahoo! Labs? Well, more personalisation and more social interaction. Just launched in the UK is Yahoo! Activity, which allows people to share the news that they are reading using Facebook.’
• ‘But the biggest selling point is the variety that his staff enjoy. He says: ‘I feel fortunate to work at Yahoo! Labs because what we do is such a young and nascent area. Its early growth is certainly going to outlast my career. A lot of the work comes back to how users are going to behave; and there’s also new forms of interaction appearing all of the time. The tablet, for example, has taken us by storm in the last three years–bringing out new human dynamics in ways that we hadn’t anticipated.’ And with new devices constantly being developed, and more users coming online, this isn’t going to change.’

Hear more from Prabhakar and read the full article from Wired here.

The Signal launches today with a blog and a few real-time election predictions and will grow quickly over the next few weeks and months. Eventually, The Signal will consist of three main elements: Yahoo! Predictions and Sentiment, Prediction Games, and The Signal Blog.

The Signal Blog is where Yahoo! Research experts, along with other notable authorities in the field, make sense of the data in a clear, enlightening and entertaining way. Some articles will show how the world changes before and after major events; an example would be our recent article showing the movement in Herman Cain’s likelihood of gaining the Republican nomination before and after one of the women accusing him of sexual harassment held a press conference. Other articles will be wonky, explaining how or why people should or should not care about the information in the news; an example our recent article discussing the latest polls on the fate of Obama’s landmark healthcare law.

The following chart, published on The Signal on Tuesdsay, shows the progress of the Republican primary from Labor Day 2011 to Election Day 2011 including the impact of several major events. Our data–at this time an average of the two largest prediction markets Betfair and Intrade–signaled the rise and fall of Trump, Perry, Christie, and Cain well before the polls.

The Signal will quickly grow to include real-time consensus forecasts for hundreds of outcomes surrounding Election 2012 (and, eventually, thousands of outcomes spanning Sports, Entertainment, and Finance), including who will win the Presidential primaries and the election overall; who will win in each state; who will win in every Senate, Gubernatorial and Congressional race; and the future levels of unemployment rates, real estate prices, oil prices, Obama’s approval ratings, etc., each easily bookmarked, and available as tables and charts. Our sentiment polling will focus on the mood of the country in political, economic and general wellbeing. Our predictions and sentiments will be driven by a myriad of data: prediction markets, polls, fundamental indicators (past results, ideology indictors, and economic indicators), Yahoo! search, Yahoo! polling, and Yahoo! prediction games.

The Signal’s interactive prediction games will be fueled by Yahoo! users’ own predictions. For the election, the game is based around predicting who will win in every state, and combinations of states. There will be a simple “pick’em” version and an expert-level version that’s much like Predictalot, the fantasy-sports-meets-stock-market prediction game that Yahoo! previously ran for March Madness, World Cup, and India Cricket. These games serve three purposes for The Signal. First, the data will enter into our models and help inform both predictions and sentiment. Second, we will be testing new procedures for both polling and prediction games that we hope will generate unique information from Yahoo! users. The Signal is not just reporting data, but we will experiment with the most advanced procedures for generating data. Third, we hope that these interactive elements will help you, our users, feel more invested in and informed about the election than ever before.

So if you’re keeping track of the elections or want to feel like you’re part of the action, make sure you check out The Signal, it’s the only place to get up-to-the-minute data and predictions on all your election needs.

In light of this, with the roll out of the new Yahoo! Mail last May, we recently launched more advanced anti-phishing defenses and enhanced spam protection to help protect your inbox.  And we’re happy to report its success.  The Yahoo! team put together new technology that’s now blocking nearly 550 billion spam messages from hitting your inboxes each month – that’s approximately 1,800 emails for every Yahoo! Mail user.

We think the technology we’re using to keep your mailboxes safe is quite awesome and we wanted to show you how we do it, so today we’re going live with the Yahoo! Mail Visualization Project – a view of what no one has seen before using live data…how we use cloud computing and Apache Hadoop technology to filter spam and re-route email for the 300 million mail users we have across the globe.

We won’t bore you with the techy details in writing because it’s much more compelling to see…

So why is it important for us to reveal how Yahoo! Mail works? Because we’re proud to show the science and technology powering the products you use every day.  Through the visualization project, we can demonstrate the technology behind the Yahoo! Network and show the impact that the tech has on you, the consumers of Yahoo!’s great technology.  As a quick example, the anti-spam protection we mentioned earlier uses Hadoop to aggregate anonymous data from the billions of emails sent and received each day and as a result has helped us reduce spam reports by 65%.  Our technology analyzes all this anonymous data (with the help of Hadoop) and identifies spam patterns so we can then use really intelligent algorithms to predict future email patterns that will differentiate “good” and “bad” senders.

To give some perspective on how powerful our computing technology is, think about these fun facts:

• For every loving message from mom, Yahoo! Mail blocks 4 spam messages from the people she warned you about.
• In the time it takes you to read this, our Hadoop software (the brain behind Yahoo! Mail) has learned from more than 4 million user actions.
• In the time it takes sunlight to reach earth, Hadoop processes 332 million user actions, helping reduce spam in Yahoo! Mail.
• Hadoop processes 1 terabyte of data (equivalent to 4,000 years of continuous typing) in the time it takes the fastest human to run a mile.
• We add over 70 million Yahoo! Mail accounts per year – that’s the combined population of London, NYC, Paris and Tokyo.

With that we’ll leave you with the link to the Mail Visualization project.  It’s super interactive and ready for you to check it out for yourself: http://visualize.yahoo.com/

-Markus Weimer, Yahoo! Labs
-Andreas Neumann, Grid Architect

Editor’s Note: This article is part of a series entitled, “In My Own Words,” that gives employees the opportunity to share their own stories about Yahoo!. Feel like yodeling your own? Post your Yahoo! story on your social networks using the hashtag #myYahoostory.

Earlier this year in late August twenty-four exceptional PhD students from universities around the world visited Yahoo! for the 2011 Key Scientific Challenges Graduate Student Summit, a two day gathering where the annual winners of the popular Yahoo! Labs program discuss the cutting edge challenges facing the future of Web sciences. Yahoo! Labs’ Key Scientific Challenges Program (KSC) aims to nurture the minds of tomorrow through funding (winners earned $5000) and mentoring from some of Yahoo! Labs leading researchers and scientists. This year’s winners, which were announced in April, hailed from universities as close by as Stanford and Berkeley and as far as Bangalore and Israel.

Day one of this year’s KSC Summit began with a line-up of highly regarded headline speakers, including Andrei Broder, Yahoo! Fellow and Vice President for Computational Advertising, who gave an overview of research at Yahoo!.  Andrei spoke about computer science in the context of human social systems, outlining the sweet spot at Yahoo! where scientists combine tech, social interactions and economics to deeply analyze and measure data across the vast Yahoo! network, which Andrei defined as the biggest ‘Webscope’ on the Internet where scientists can test theories and develop models and research.

David Chaiken, Yahoo!’s Chief Architect, spoke about Yahoo!’s unique culture and how the company designs its technology specifically to enable close collaboration between product, engineering and science teams so that research has a direct impact on Yahoo!’s business day in and day out. He explained how Yahoo! Labs creates personalization algorithms for Yahoo!’s homepage, for example, which is a project we blogged about here on Yodel during Yahoo! Labs’ Science Week.

This is just a taste of the topics discussed in a room full of leading thinkers, with an exciting mix of ambition and experience. Justin Rao, a Research Scientist at Yahoo! who most recently appeared in a Forbes panel with our Head of Yahoo! Labs, Prabhakar Ragavan, spoke about his latest work on the intricacies of measuring advertising effectiveness. And Daniel Goldstein, also a Research Scientist at Yahoo!, presented on how you can predict information diffusion based on interactions across social networks, questioning the traditional notion of “viral” content.

Overall, the 2011 Key Scientific Challenges Graduate Student Summit was an engaging and productive event. You can learn more about all of this year’s winners here. All of them have great research in the pipeline, so watch out for these names as you read about future breakthroughs in Web science – you heard about them here first.

Yahoo!’s impact at the event is huge.  We use knowledge discovery and data mining every day to make the Yahoo! experience deeply personal and relevant for our users and advertisers alike. For example, as we described earlier this month, we use these principles to personalize the Yahoo! homepage with 45,000 different variations of our “Today Module” every five minutes. If you’re interested in learning more about why data is so important to Yahoo!, check out this video interview with some of our leading scientists that recently appeared on Forbes.com too.

We have some of our best researchers and top talent joining scientists from Stanford, Rutgers and Princeton, as well as the research labs of other leaders in the field, to share insight and case studies around such things as social media analytics, Internet ad systems, behavioral analytics and Hadoop.

We have Yahoo! researchers flying in from across the globe, presenting a total of seventeen papers, two tutorials, and of course, the KDD Cup!

This year’s KDD cup competition centered on Yahoo! Music, which has amassed billions of user ratings for a variety of musical pieces.  The competition challenged scientists to work with the raw ratings that encode information on how songs are grouped, which hidden patterns link various albums, which artists complement each other, and which songs users would like to listen to. Check out the details here: external KDD Cup Website.

Here’s a photo of Here’s a photo of one of the winning teams (Team Commendo) claiming their prize on Sunday.

We’re proud to have such a great team of professionals representing Yahoo! and driving scientific leadership across the industry.

Judd’s research in the area of incentives and motivations for online collaboration has landed him on this distinguished list, which includes some A-list influencers, including Yahoo!’s Jerry Yang and David Pennock, Facebook’s Mark Zuckerberg and Google’s Sergey Brin.

Core to Judd’s research is his focus on using creative techniques to measure and analyze motivation and participation in depth. For example, Judd has critiqued popular notions about what motivates online participation – specifically around “gamification,” or virtual goods (think points, levels, and badges) that many social media systems use to incentivize user participation. Despite common belief in their effectiveness, Judd argues that game dynamics can “crowd out” intrinsic motivations, discourage collaborative behavior, stifle creativity, and ultimately do more harm than good in some online contexts.

Judd also challenged the notion that gamification is an equal opportunity motivator. By looking for statistical connections between game-based motivation and dispositional beliefs about trust and privacy, he found evidence that the social risks and rewards inherent in game dynamics make them less motivating for less trusting people.

You should also check out Technology Review’s September issue to read Judd’s column on the phenomena of gamification. Judd dives deep into the psychology of game mechanics to identify what truly motivates people to engage in online behavior, clarifying much of what is commonly misunderstood about the growth of online and mobile games.

The annual EmTech@MIT Conference, to be held in October at MIT, will honor the TR35 winners with a dedicated awards ceremony. Congratulations again to Judd for a very well deserved honor!

In support of this initiative, Yahoo! Labs has developed LinkUp — an online application for organizing event participants. LinkUp provides three key features. First, it provides a set of waypoints along the route where people can go to participate in the event. These waypoints are displayed through an interactive version of Yahoo! maps. Second, participants can join the route at a specific location and post messages related to event activities. Finally, participants can link their Facebook accounts and share this information with friends, engaging them to participate.

Hands Across California is attracting the support and active involvement of a number of high-profile celebrities, community leaders, and activists. Among the celebrities are Ryan Seacrest, George Lopez Guy Fleri, Mark Harmon, Quincy Jones, MC Hammer, Yvette Nicole Brown, Ed Begley Jr., Arsenio Hall, and many more. Among the political figures, President Bill Clinton, Congresswoman Karen Bass, Lt. Gov. Gavin Newsom, and Los Angeles Mayor Antonio Villaraigosa have all pledge to participant in this event.

Yahoo! Labs has been working with the organizers of this event on developing a route that will literally join nearly all of the 112 community colleges in California in a vast statewide human line. A similar event took place 25 years ago with “Hands Across America”, which saw 6.5 million Americans join hands from California to New York. Times have changed and Yahoo! technology is now the primary tool for organizing such a massive event. To learn more about HAC, be a part of this historic event or donate to the cause, visit www.handsacrosscalifornia.org.

Many thanks to the folks at Yahoo! Labs and across Yahoo! who volunteered their time and hacking skills to make this event happen.

Yahoo!’s scientific leadership was clearly evident at WWW2011 – Yahoo! Labs presented 26 papers and posters and participated in several signature discussions around crowdsourcing, social media and the digital divide. Yahoo! scientists also provided demos on their latest projects, drawing a great deal of interest and attention from attendees.

You may have seen some of Yahoo! Labs’ thought provoking research in the news this week, whether it was focused on “Who Talks to Whom on Twitter” or if it’s possible to detect the credibility of Tweets.

For those who couldn’t make it to WWW2011, we checked in with Prabhakar Raghavan, Head of Yahoo! Labs, from the event:
[]
For a full recap of all the activities from this week’s WWW2011, check out the Yahoo! Labs Web site and follow us on Twitter @YahooLabs. You can also see Tweets related to WWW2011 at #WWW2011.

This week, IBM’s supercomputer, Watson (named after IBM’s founder, Thomas J. Watson), took on two of the most championed Jeopardy! contestants of all time in an exhilarating $1 million Jeopardy! face-off between man and machine.

Watson defeated Jeopardy! defenders Ken Jennings and Brad Rutter, amassing $77,147 in winnings in a nail-biting three-night tournament that sparked interest around the field of artificial intelligence and data analytics.

IBM explained, that by matching the text in a question to the text in its vast memory, Watson can analyze and recite an accurate answer in less than three seconds. If there is no match in Watson’s “brain,” it takes a guess based on a confidence level that is calculated on probabilities.

So what makes Watson’s genius possible? A whole lot of storage, sophisticated hardware, super fast processors and Apache Hadoop, the open source technology pioneered by Yahoo! and at the epicenter of big data and cloud computing.

Hadoop was used to create Watson’s “brain,” or the database of knowledge and facilitation of Watson’s processing of enormously large volumes of data in milliseconds. Watson depends on 200 million pages of content and 500 gigabytes of preprocessed information to answer Jeopardy questions. That huge catalog of documents has to be searchable in seconds. On a single computer, it would be impossible to do, but by using Hadoop and dividing the work on to many computers it can be done.

In 2005, Yahoo! created Hadoop and since then has been the most active contributor to Apache Hadoop, contributing over 70 percent of the code and running the world’s largest Hadoop implementation, with more than 40,000 servers. As a point of reference, our Hadoop implementation processes 1.5 times the amount of data in the printed collections in the Library of Congress per day, approximately 16 terabytes of data.

We’ve been doing it because we think it’s a game-changer for the Internet. Hadoop is critical to Yahoo!’s business, delivering personalized experiences to our more than 630 million users worldwide. Yahoo! Mail uses Hadoop to fight spam, the Yahoo! home page content is personalized with Hadoop and a suite of our personalization technologies. What these have in common is that they require processing huge amounts of data very quickly and reliably on large numbers of computers, mirroring Waston’s requirements to win Jeopardy! And just like Hadoop was critical to Watson’s success, it has a fundamental and direct impact on Yahoo!’s performance and bottom-line.

So if you’re ever on Jeopardy!…”It is the technology behind every click on Yahoo! and the IBM supercomputer that accomplished the impossible….”

“What is Apache Hadoop!?”

Originally launched in November 2007, Yahoo! and its M45 program are providing universities the opportunity to conduct research otherwise impossible without the power and speed of a supercomputing resource, which consists of approximately 4,000 processors.

Hadoop, the open source technology at the epicenter of big data and cloud computing, is the core data analysis technology used across Yahoo!, and is used by all the universities participating in the M45 research initiative. Yahoo! benefits from university contributions to the Hadoop code base as well as through insights from cutting edge research initiatives conducted on Yahoo’s M45 supercomputer.

Examples of academic research conducted on the M45 include two of the world’s largest knowledge acquisition research projects – Never Ending Language Learning System (NELL) at Carnegie Mellon University and KnowItAll at the University of Washington.

One of the hottest research trends at universities today is the merging of mobile computing and cloud computing. By expanding the M45 platform to support both mobile and cloud computing research, Yahoo! is enabling top tier universities to tackle some of the industry’s most critical computing challenges.

Additional university research projects facilitated on Yahoo!’s M45 supercomputer include

• Carnegie Mellon – performing research in large-scale graph mining (graphs with billions of nodes), text search and analysis, statistical natural language processing, analysis of media Internet traffic, statistical machine translation, learning to read the Web and file systems research.
• Cornell – exploring the use of advanced methods from computer science to help solve environmental and broader sustainability challenges.  As part of its Citizen Science research program, Cornell researchers are exploring methods to enable a smart phone application to enter bird observation data into the cloud.
• Purdue – combining the power of mobile and cloud computing to develop a context-aware navigation system for blind and visually impaired people.  Researchers also plan to investigate topics in cloud data privacy, information retrieval and the automatic off-loading of computation from mobile devices to the cloud.
• Stanford – merging large-scale cloud computing techniques and advanced statistical machine learning methods to analyze the vast amount of text, image and network data now available, on the web and elsewhere.  The goal is to achieve a new level of understanding of the semantics latent in various media, attaining systems with greater artificial intelligence.
• University of California at Berkeley – analyzing social networks and studying population genetics, as well as testing new architectures for collecting traffic data from GPS-equipped mobile phones and estimating traffic conditions in real-time, analyzing climate-change satellite data, prototyping new scientific applications and improving cluster scheduling and reliability.
• University of Massachusetts Amherst – investigating efficient inference of the relations among documents and passages in a large collection of books.
• University of Michigan at Ann Arbor – profiling and understanding MapReduce job execution, optimizing energy consumption and performance for heterogeneous MapReduce workloads, performing large-scale language model analysis and investigating how to offer a geo-distributed service that can be easily accessed through mobile phones for content distribution, computation offloading and other services.
• University of Washington – studying scalable scientific data management and large-scale knowledge acquisition, and building a large knowledge base that can be queried both by Web and mobile phone.

Yahoo! is also a founding member of the Open Cirrus Cloud Computing Testbed and the Open Cloud Consortium, both facilitating scientific research in the cloud.  Combined, these programs bring the collective wisdom of many of the world’s top scientific researchers to Yahoo!’s industry-leading research portfolio at Yahoo! Labs.

With Halloween weekend about to kick off, there’s no better way to get in the mood to deal with the zombies (and other ghouls) trick or treating at your door than to command your very own Zombie army.

Here’s how it works:

• Within the world of the game, the zombie apocalypse is upon us and the world is being divided and conquered by warlords, each leading their Shambling Horde of zombies.
• Each player can move their zombie army into different territories on a fictitious island.
• As the two armies begin to meet in those territories, skirmishes ensue.
• Each player allocates his or her zombies into three groups during a skirmish and the winner is determined by matching the three different groups against each other.
• It’s a best two-out-of-three contest and the winner of the skirmish is able to spawn additional zombies to continue their march.
• The battle ends when one player’s Shambling Horde reaches the other’s command post.

Sounds fun, right? We think it is. But it’s also grounded in some classical economics and game theory that should be familiar to anyone who’s had a finite amount of anything (budget, advertising dollars, even candy) that they had to spread around in a competitive environment.

Just to take a common example from the business world, say you have $1 million to spend on advertising for a new product and you knew one of your competitors was getting ready to launch a very similar product around the same time. Where would you spend your advertising money? In microeconomics this situation is often associated with Colonel Blotto’s problem. So as you can see, even though your Zombie horde may be perfectly virtual, the decisions you make in the game and the strategies you evolve to deal with each skirmish are something with a range of real world applications.

Why is Yahoo! Labs developing a game like Shambling Hordes? Well, first of all, because Zombies are awesome. And second of all, because using technology and new experiences to look at thorny intellectual problems are a big part of the experimentation we do in Labs.

We’re hoping to understand how you – the users – play the game and interact. How quickly do people shift strategies or learn new approaches? What lessons could be applied to online advertising marketplaces? To other new interactive social games? Like most of our experiments in the Labs sandbox, Shambling Hordes is an open-ended pursuit. We don’t know exactly where it will go, but we do know we’ll learn a lot.

Also, Zombies.

– The Yahoo! Labs Team

Like the first two years, the quality of the presentations and papers was fantastic, with the audience and participation continuing to grow. Several members of the Yahoo! leadership team spoke at Tech Pulse on a variety of topics. Today we thought it would be fun and interesting to share two short video interviews with this year’s featured guest speakers, Shree Nayar, chair of computer science at Columbia University, and James Surowiecki, writer for The New Yorker and author of “The Wisdom of Crowds.” Both talks captured the imaginations of the audience, and we hope after checking out these short clips, you’ll be as interested as we are to learn more about their work.

Shree Nayar presented on the emerging field of computational cameras. With our ability to capture more and more of the physical world with digital cameras and advanced optics, our ability to explore and refine that world computationally is extending how we think about reality and imagination. Nowhere was that potential more compelling than in Shree’s discussion of his Big Shot camera project, which gives kids hands-on access to the wonder and science of digital imaging.
James Surowiecki first introduced the idea of collective intelligence in his book “The Wisdom of Crowds” published in 2004. Since then, he’s continued to delve deeper into the phenomenon as additional research and experiments have proliferated – with the Web providing an amazing platform for additional, large-scale investigation. James has also gone on to investigate the downside of the “Wisdom of Crowds” by covering finance and business at The New Yorker, with the housing bubble of the past few years an example of crowds acting somewhat less than wise.

All in all, it was another great Tech Pulse and we’re looking forward to continuing the tradition next year.

-The 2010 Tech Pulse Team

Our own luminary, Carol Bartz, will give a keynote speech at the awards ceremony. She will talk about the rise of the insight-driven enterprise; the importance of science, technology, and data to our business; and the power of intense personalization — how we make our products increasingly relevant and improve consumer experiences.

In addition, our top techies will lead discussions about the world’s largest-scale technology challenges, the art of conducting social science research with data from hundreds of millions of people, and the potential for open source technology to change lives, among other topics. Here is a sample of just a few of the talks on the docket at Grace Hopper:

• Elizabeth Churchill, a Yahoo! Labs principal research scientist, will be presenting the results from her recently published paper “Anger Management: Using Sentiment Analysis to Manage Online Communities.”
• Barbara Poblete, also a Yahoo! Labs scientist, will present an introductory workshop on Hadoop, discussing how it can be leveraged for gleaning insights from social data.
• Julia Lee, director of engineering for Yahoo! Mail, will talk about innovating on one of the world’s largest Web mail services, Yahoo! Mail.
• Avni Khatri (front end software engineer) and a team of colleagues will facilitate the open source Codeathon for Humanity event through the Sahana Software Foundation, dedicated to humanitarian projects. During this hands-on event, conference attendees will build software that directly helps those in need.
• Fay Hellal, director of global advertising product marketing invites Kakul Srivastava, vice president of product management for communication products at Yahoo!, on stage for a chat about Women Thriving in Leadership.
• Mani Soni Abrol, an engineer from Yahoo! India, joins colleagues from IBM India and Intel Labs India for a panel on Women in Computing in India: Experiences With Boundaries. They’ll discuss the gender gap and the challenges faced by Indian women in technical industries, including research and academia.

We think technology remains the most rewarding career for women because ultimately it’s about envisioning a better future. This week we celebrate all the women technologists who are making a difference in our world. And to those embarking on their careers in technology: Welcome aboard!

You can learn more about these challenge areas and why they’re essential to what Yahoo! does in our blog series here on Yodel Anecdotal this past spring. Different Yahoo! Labs scientists weighed in on topics as diverse as Green Computing, Privacy and Security, Economics and Social Systems, Advertising, Web Information Management, and Machine Learning.

The 2010 KSC winners received a small amount of seed funding to help support their research and the opportunity to interact with Yahoo! scientists in their area.  The summit is a forum for winners and Yahoo! scientists to discuss research trends, present their work, and jointly develop innovative approaches to these challenges.

Preston McAfee, Yahoo!’s Chief Economist, and Raymie Stata, Yahoo!’s Chief Technology Officer, opened today’s session by providing an in-depth overview of what makes Yahoo! run and why it’s fascinating to scientists, giving students rare insight into Internet-scale challenges and data.

We sat down with two of this year’s winners: Moira Burke of Carnegie Mellon University and Scott Duke Kominers of Harvard University, to learn a bit about their research.

Here’s Moira:

And here’s Scott:

The purpose of the KSC Program is to help fuel innovation and invent the new sciences that advance the Internet. It’s a lofty purpose that requires great minds and open collaboration. Based on this year’s summit, there’s no doubt that these winners will play a major role in the future of the Internet and have clearly raised the bar for the program heading into 2011.

Congratulations on these tremendous accomplishments, and all the best to everyone!

Jamie Lockwood

Academic Relations Program Manager

Yahoo! Labs’ Academic Relations

There’s not enough room here to go into all of the insights Yahoo! Labs has published and presented this year – that’s what our website is for, and we highly recommend you check it out  – but to give you an idea, we recently discussed more than 25 papers at the ACMSIGIR and ACMKDD international research conferences in July. You’re not familiar with information retrieval and knowledge discovery? No worries. Here are a few highlights that will give you a feel for our work:

• Automatic Construction of Travel Itineraries Using Social Breadcrumbs: Every minute, thousands of images are uploaded to Flickr. In this study, the images and associated meta-data (like time stamps and geo-codes) became “social breadcrumbs” that Yahoo! Labs scientists used to create a roadmap to some of the world’s most popular intra-city travel itineraries, including Barcelona, London, Paris, New York, and San Francisco.

• Twitter Under Crisis: Can We Trust What We RT?: Yahoo! scientists examined Twitter usage during the 2010 Chilean earthquake, focusing on users’ differentiation of false rumors and legitimate news. The conclusion Yahoo! Labs scientists found was that users questioned the alleged rumors, almost self-regulating the truth of each other’s assertions, which shows that social networks may have the potential to correct their own mistakes.

These are just a couple, but I hope they pique your interest. There’s a lot to learn out there, and what we collectively know about the Web is changing every day.

In addition to publishing research papers and working with the academic community, Yahoo! Labs works very closely with product and engineering teams here at Yahoo! to continuously innovate and make sure our products – like Yahoo! Mail, Messenger, the Yahoo! Homepage, Search and many more – are using the latest scientific ideas and breakthroughs. So even if you’re not familiar with the latest algorithms for calculating the relevance of a website, there’s no doubt: Almost every second you spend with Yahoo! is one that’s been improved, in some way, by Yahoo! Labs.

For example, we use a technology we call content optimization that was developed in Yahoo! Labs. This helps to decide which stories will be featured in the Today Module on our homepage. All kinds of variables are factored in, from which stories seem to be gaining in popularity around the Web at that time, to what topics might be relevant to your local area and what news is about to take off, based on the years of experience and expertise of our editors. All of these factors and more get computed constantly at Yahoo!, and with that data we’re customizing your homepage, serving as many as 32,000 different versions  every five minutes. You get to see the page that’s most interesting to your world, and so does everyone else — it’s all based on science.

If you’re curious about the identities of these mad scientists working to make your Yahoo! experience better and more intuitive, here’s an up-close view from Science Week:

As you can see, they’re normal folks just like you and me. They’re just really obsessed with the science of the Web.

You can see more photos from Science Week by checking out our Flickr stream. And you can track all the innovations and interesting projects from Yahoo! Labs by following our Twitter feed.

-Prabhakar Raghavan

Yahoo! Chief Scientist and Head of Yahoo! Labs

At last year’s conference held in Madrid, Spain, Yahoo! presented 14 refereed papers and Ricardo Baeza-Yates, head of Yahoo! Labs Barcelona, gave a keynote on mining Web 2.0 data for Search. This year, Yahoo! raised the stakes again, with 21 papers accepted at the 2010 event (out of a total of 105 accepted papers for the entire event), as well as a variety of posters, demos and panels, including a featured discussion on the future of Search and the core technologies of the Web.

As usual, the Yahoo! booth at the conference attracted a lot of interest from attendees. Some of the highlights include demonstrations we are giving of some experimental projects to gather data and test our theories. Because not everyone can attend WWW, we thought it would be fun to give everyone a quick look-in from the booth with some short clips of our scientists and their demos.

First up, is Predictalot, a combinatorial prediction market game we debuted to test the “wisdom of the crowds” during March Madness and that will be making a global return to prominence for World Cup 2010 in South Africa:

Next up is Statler, a tool for developing a model of engagement for how users interact with content over time. In this case, Statler is analyzing public Twitter feeds around major events, like the 2008 Presidential debates and Barack Obama’s inauguration. The goal of Statler and the associated research is to be able to predict which items are likely to resonate over time, rather than spike in popularity before quickly fading away.

Our next demo, Ranking Entity Facets, is a technical name for a technology we’ve deployed at Yahoo! Search to make it easier for people to explore all kinds of people, movies, locations and other “entities” related to what they’re searching for at any given time. What makes this capability exciting is that it relies on the science and technology powering Yahoo!’s Web of Things platform where Yahoo! is able to mine all kinds of content and data sources to synthesize a Web page for you with all the information you need on a topic, rather than just a list of Web links.

And then there is Wrapper Induction, developed by our data mining group in Labs, which is all about extracting valuable information from Web pages.

For a full recap of all the events at WWW2010 this week, make sure to check out the Yahoo! Labs Web site and follow us on Twitter @YahooLabs. You can also see all Tweets related to WWW2010 at #WWW2010 and follow our Flickr stream.

Lin Koh
Yahoo! Labs

P.S. If you want to check out what fun is, here’s a cryptogram we shared with folks at WWW as part of a booth contest. Perhaps you’d like to take a crack at solving it too.

Or, to put it another way, we built a spoof website called ZOMG where you could put you and your friends into celebrity news stories “mad-libs” style and send them around to everyone you know.

We had a blast coming up with the storylines, designing the viral propagation and building the site over the last couple of days. And we hope everyone who had a chance to play with ZOMG enjoyed it as much as we did (we were helped by Carol, who got the game off to fun start herself).

We do a lot at Yahoo! Labs to try and understand the scientific phenomena underlying the Web. We’re always devising complex experiments, testing algorithms, machine learning and designing markets according to cutting edge economic concepts. Math, code and technical jargon abound at our lunches. But eccentricities aside, in our hearts and minds, we’re just like everyone else. We long to imagine ourselves as being “seen about town” and caught up in celebrity intrigue with Megan Fox. Well ok, some of us do.

All that being said, we did in fact learn a few things yesterday with ZOMG. We were curious about how transient flash interests like April Fool’s jokes disperse on the Web. What are the patterns of influence or rules of distribution, for example? We don’t have a full picture yet, but here are some very preliminary data (and watch for more in time, at the Labs website):

• By mid-afternoon, we had crossed 14,000 user-created stories
• The three top story categories were Baby Bumps, Seen About Town and American Idol

We’re starting to get used to this April Fool’s tradition and there’s no doubt the Web is a great playground to try stuff out on. And since we can technically pass off all of our jokes as sincere inquiries into the inner-workings of Internet science and culture, you’d better look out.

Wait until next year,
-Prabhakar Raghavan, Yahoo! Labs

P.S. One more interesting fact: if anyone is curious about who got the most fooled by ZOMG, the big winner is Tim Morse, Yahoo! CFO. Last night he was ready to throw our CMO Elisa Steele under the bus when she sent him the pretend FCC Fining Carol story as an urgent FYI. Congratulations, Tim!

Here’s how it works. You can invest virtual points on almost any prediction you can think of about the men’s college basketball tournament, like “Duke will advance further than Texas,” or “Every final four team’s name will start with the letter U.” For example, you can invest one point predicting that “there will be exactly 2 upsets in the Elite Eight” and, according to the current Predictalot odds, you’ll win ten points if you’re right.

You can keep your predictions and see how they turn out, or you can sell them at any time, even while you’re following the action on the court. How does that work? Think of your predictions like stocks that can be priced, sold and traded (for virtual points, of course) just like you’d buy and sell stocks, options and futures on the financial exchanges of the world. Predictalot offers hundreds of millions of such predictions, constantly computing prices for them all.

Throughout the tournament, the prices of all the predictions in the system will go up and down based on what’s actually happening in the tournament and on what everyone else thinks those predictions are worth. Based on the value of the predictions you invest in you’ll accumulate points and climb your way to the top of the Predictalot leader board.

In technical terms, Predictalot is what is called a combinatorial prediction market. Some of us at Yahoo! Labs, along with academics like George Mason University professor Robin Hanson, have been studying and writing about them for quite some time, and dreaming about trying them out in the real world. We built the first version of Predictalot during an internal Yahoo! Hack Day. And this year we used the Yahoo! Application Platform to finally build a public version of the game. Like any Yahoo! App, you can install Predictalot to play right on the Yahoo! homepage. (Note that we’re not special in that regard: anyone can develop a Yahoo! App that’s available to millions of Yahoo! users — there’s good sample code, it supports YUI and OpenSocial, and it’s easy to get started. Seriously, try it.)

After some fits and starts, a few late nights, and eventually a few all nighters, we’re proud and excited to go live with Predictalot, version 1.0. I can’t rave enough about the talent and dedication of the research engineers who gave the game a professional look and feel and some real production speed. They turned a very pie-in-the-sky idea into a reality. We have a lot feature ideas in mind for future versions, but the core is in place and we hope you enjoy the game.

With 9.2 quintillion outcomes, or nearly as many as the number of individual insects on Earth, Predictalot is to our knowledge the largest prediction market ever built. We hope that with your participation it will prove a great test to the limits of what the wisdom of crowds can produce.  We’d also like to pave the way for the serious use of combinatorial prediction market technology in a host of other arenas down the road.

Smarter markets will be extremely valuable in a variety of applications, especially online, where markets and exchanges decide prices and allocations for all kinds of goods and services, including advertising inventory. And more and more, the matching of buyers with sellers – which is the core function of any market – is requiring sophisticated algorithms, machine learning and optimization technology. Predictalot attempts to help us better understand these dynamics while also giving us the great satisfaction of picking Duke to go all the way.

(Duke alum) David Pennock and the Predictalot Team:
Mani Abrol, Janet George, Tom Gulik, Mridul Muralidharan, Sudar Muthu, Navneet Nair, Abe Othman, Daniel Reeves, Pras Sarkar

P.S. If you’d like to learn more about the technology hidden in the engine and the scientific factors at play, read our post about Predictalot on the Yahoo! Research website. And don’t forget to also check out all of Yahoo!’s fun tourney experiences.

On January 27, we announced the kick-off of our 2010 Key Scientific Challenges Program.  To highlight the scientific challenge areas included in the program, we launched a series of guest blog posts on Yodel Anecdotal. Our most recent post covered Web Information Management.

Another big challenge our Yahoo! research scientists are continually examining is machine learning. In this entry, John Langford from Yahoo! Labs shares some thoughts on how Yahoo! is driving research into machine learning and why it’s a fascinating field.

When I wake up in the morning, I can’t resist checking my email and browsing the Internet to see if anything has come up. Then I get to work thinking, writing, searching, finding, and learning various things, all using an Internet that’s powered by machine learning in dozens of ways. When I go to sleep at night, I smile because I know that in addition to using machine learning throughout my day, I’ve also done my part to advance machine learning technology, many others have done likewise, and that by doing so we’re making a major impact on people’s lives.

Even though machine learning has such a broad influence on the Internet, it can be quite difficult to recognize. This is primarily because machine learning’s benefits are often hidden — they are the spam emails you don’t see, the uninteresting news articles you don’t see, and the irrelevant search results you don’t see, just to name a few. In this sense, machine learning is like an invisible hand. It’s also sometimes easier to recognize the flaws in a machine learning system – like “Why did my email end up in my friend’s spam folder?” – than it is to notice its benefits. But despite these quirks, machine learning is one of the best technologies we have for solving some of the biggest problems on the Web.

The problem of spam is representative of why machine learning is so effective. Spammers are constantly changing and adapting their strategies and technology to evade even the most capable filters. Machine learning attacks this problem by aiming to build an automatic system able to stay ahead of the game and continually refine itself in response to its environment. We haven’t completely achieved that goal yet, but progress is steady. Machine learning systems can always get better, learn more, work faster and in ever more ways, because people will always want less spam and more interesting and relevant news articles.
Naturally, this reality means we’re constantly running into both the empirical and theoretical boundaries of machine learning and statistics. How do I learn from so much data that we can’t fit it on a machine? How do we extract evidence of what the best decision was? What if the best decision changes? How do I minimize the need to know the best decision? How do I effectively use the incredibly large quantities of information available on the Internet? And how do I fit it all together in an automatic way that is useful to someone? And how do you know it’s useful?

Good answers to these questions can improve the life of just about everyone, which is the core reason why Yahoo! is sponsoring the Key Scientific Challenges Program. If you are a graduate student working on these questions, you understand how exciting and challenging this field is. And if you aren’t, consider the satisfaction associated with changing fields. :-)

John Langford

Yahoo! Labs

During each of these hack events, Yahoo! developers teach passionate CS, HCI and engineering students about the latest and greatest open tools and technologies and web programming tips in a series of tech talks, then spend 24 hours with them to help them build their dream hack for the competition held at the end of the week. In true Yahoo! style, the all night contest is filled with fun, free food, schwag, prizes, Wii games and plenty of caffeine!

At the end of the season, the winners from each university will be flown by Yahoo! to our University Hack Showdown, which is normally held in conjunction with Yahoo!’s Open Hack Day, where they will hack it out for street cred, cool prizes and the opportunity to get their hack noticed by web industry judges, including VCs, web entrepreneurs, Yahoo! execs and technology gurus.

“The close of 2009 marked the third year of the Hack U™ program and it’s been amazing to watch the innovation, spirit and quality of the hacks continue to grow, said Jamie Lockwood, program manager for Hack U™. “The students used to just mash up as many open technologies as they could barely thinking about UI or what problem the hack solves, but today we see ideas and prototypes coming together that are more and more sophisticated and in some cases almost ready to go to market.”

The winning Hack U™ students from last season had no problem holding their own against the experienced developers that showed up at the Yahoo! Open Hack event in NYC, winning half of the categories including best overall hack.

The program has definitely caught the attention of Yahoo!’s internal open tech leaders, as well as the external web community. The Hack U™ winners from Toronto even snagged an interview at Y Combinator with their hack idea.

You can read more about the program and view some of the other hacks that have been developed at universities across the country and the world (we had our first India Hack U event at IIT earlier this year!) on the Yahoo! Developer Network site.

Hack U events have been held at some of the best engineering universities around the country including UC Berkeley, Stanford, Georgia Tech, Carnegie Mellon, University of Washington and University of Illinois and we’re always on the lookout for other universities that want to be part of the growing Hack U community We also want to keep the technology topics fresh, timely and up to speed with the latest web industry innovations so feel free to send us requests for specific speakers or ideas for suggested topics.

Check out the website for more information, updates and details about the upcoming 2010 schedule.

– Yahoo! Academic Relations and Hack U™ Team

Key Scientific Challenges, Entry #3: Microeconomics and Social Systems

On January 27 we announced the kick-off of our 2010 Key Scientific Challenges Program.  To highlight the scientific challenge areas included in the program, we launched a series of guest blog posts earlier this month on Yodel Anecdotal. Read our previous post on privacy and security, “Data, Data Everywhere, but How to Keep it Safe.”

Another big challenge are Yahoo!’s research scientists are continually examining is microeconomics and social systems. In this entry, Sharad Goel from Yahoo! Labs shares some thoughts on how Yahoo! is tackling the new opportunities for research into the social sciences that the Web is making possible and why it’s a fascinating field.

What do your friends really know about you? How much do they influence your decisions? How often do we stray from the cultural herd? How do groups organize to solve complex problems?

Answers to such fundamental questions about social behavior have often eluded us. With microscopes we peered into the intangibly small building blocks of life, and with telescopes we found our place in an unimaginably expansive universe. But without the tools to faithfully document human activity—a challenge that by comparison seems so palpable—we had no way to investigate the inner workings of our own communities. Now with an explosion of information on every aspect of our everyday existence—from what we buy, to where we travel, to whom we know—we can measure what until quite recently was thought unmeasurable. In the Microeconomics and Social Systems Group at Yahoo! Labs, we are using this proliferation of data to explore how societies function. It’s a fascinating area of study that is just beginning to shed light on new layers of human behavior, making it a perfect fit for the Key Scientific Challenges Program.

In a recent study that’s garnered some attention, for example, we asked, “How eccentric are people?” Looking at consumer preferences across movies, music, and web browsing, we came to the surprising conclusion that ordinary people have pretty extraordinary tastes. In particular, we found that typical Netflix and Yahoo! Music users regularly watch movies and listen to songs that are not even available in the largest brick-and-mortar retailers. This result not only challenges stereotypes about people blindly following the herd, but also highlights the importance of offering consumers broad selection. That is, specialty products may dramatically boost user satisfaction by providing buyers the convenience of “one-stop shopping” for both their mainstream and niche interests.

In other work, we used web search queries to forecast the commercial success of movies, songs, and video games. Weeks, sometime even months, before a movie opens or a video game is released, one can find traces of pent up consumer demand in the search query logs. We found that these telltale signs of early interest are remarkably good predictors of future success. The catch? Although the search logs do reflect user intent, more mundane indicators, such as production budgets and reviewer ratings, perform equally well at forecasting sales. Thus, the benefit of web search as a prediction tool may have less to do with its superiority over other methods than with its generality, low cost, and real-time nature.

At a time when we are drowning in data, at Yahoo! Labs we’re asking a simple question: what can you do with it? The answer is limited only by our imaginations.

The Key Scientific Challenges Program focuses on a variety of scientific issues that we believe are central to developing a better understanding of the Web and the foundational technologies that will accelerate innovation. Over the next couple of months, we’ll be offering a series of guest blog posts here on Yodel Anecdotal that offer a quick overview of these scientific challenge areas from a Yahoo! (or Yahoo!s) who have expertise in the field and can explain how their research can have an impact on making the Web more engaging, relevant and powerful.

Today’s our maiden voyage, and the first guest blog post comes from Scott Noteboom. Scott is one of the foremost experts on building data centers that makes it possible for Yahoo! to run some of the fastest, most popular Web sites in the world.  He’s also leading the charge to make Yahoo!’s data centers as efficient as possible, and we’ve asked him to talk about a new challenge area we just added to the Key Scientific Challenges Program this year, green computing.

Thanks,
The Key Scientific Challenges Team at Yahoo! Labs

Key Scientific Challenges, Entry #1: Green Computing
By Scott Noteboom

Increasing demand for the Web services and applications that have become such a central part of our lives (like email and Web video, just to name a few) is also causing a steady rise in the need for more and more computing power to make all of theses services work and work well.  When we talk about computing power, we’re often talking specifically about data centers, buildings that house thousands and thousands of servers working 24-hours-a-day to make sure that the Web sites you want are there on your screen and on your phone when you want them.

Of course, while the performance of servers is improving almost everyday, making all kinds of new innovations possible, they also require a lot of power. They need so much power that a 2007 joint report by the Environmental Protection Agency and the Lawrence Berkeley National Laboratory revealed that energy consumption from data centers had doubled between 2000 and 2006 and was expected to double again by 2011. The report further calculated that there would be huge energy and money losses without increases in the efficiency of running and cooling these centers (because servers produce a lot of heat).

Recognizing both the problem and the opportunity that this presents, Yahoo! has embraced its role as a leading environmentally sustainable company by addressing efficiency in our data centers head on.  We announced in June 2009 that we’d be committing to a significant reduction in the carbon intensity of our data centers by 2014. And to accomplish this goal, we’ve developed several innovations, including a new high-efficiency data center design that reduces our cooling load and increases utilization in our data centers, increasing the productivity of our servers, and squeezing as much productivity as possible out of every last kilowatt-hour. These efforts have gotten us some recognition and hopefully will inspire more innovation because making the world’s data centers more efficient is also an important element to tackling global warming.

Everyone wants to see innovation on the Web and what it makes possible for us in our everyday lives continue to flourish, and that’s where innovation behind the scenes in data centers is so important. By making data centers more efficient, we can make sure that future innovations that make the Internet more and more powerful for all of us will also be environmentally sustainable.

This is by no means a pipe dream, which is why we’re so excited to be part of the Key Scientific Challenges team recruiting some of the best young minds in the world to learn from our experiences and accelerate their own research and ideas. A study by The Climate Group entitled Smart 2020 predicts that, while greenhouse gas emissions from the Internet industry will rise to approximately 1.3 gigatons of CO2, the combined impact of smart grid, smart logistics, smart buildings and videoconferencing could reduce emissions by approximately 7.8 tons.

We’re very excited about the next big ideas that will continue to bring radical improvements in energy efficiency and truly green computing. If you’re interested in learning more about the Key Scientific Challenges Program visit the site, and if you’ve got an idea for research into green computing, get working on a proposal. Submissions are due March 5th.

Looking back, 2009 was our most successful year to date. Through a variety of programs, we brought together a lot of smart folks and important ideas. As we head into 2010, we want to share our excitement about these programs and events.

Hack U:
Our University Hack Day competitions (Hack U) brought Yahoo!’s open technology, top developers, and hack spirit to campuses for technical talks and a 24-hour student programming competition. Students from 11 universities in the U.S., Canada, and India built more than 300 hacks. Winners from each campus participated in the New York Open Hack Day, ultimately winning 6 of the 11 categories.

Yahoo! Big Thinkers Series:
As part of our Big Thinkers distinguished speaker series, Yahoo! brought the brightest minds in the academic community to our campus for talks, and we’ve made those lectures available on the Web for all to see and enjoy. We had talks on

• Global climate change with Kenneth Arrow, Nobel Laureate and Professor Emeritus of Economics at Stanford University
• The psychology of the Web with Sandy Pentland, MIT
• Getting around the “you don’t know what you don’t know” syndrome with Ion Stoica, University of California, Berkeley
• How computational thinking can transform education with Jeannette Wing of the National Science Foundation
• Calculating the largest prime number ever found with Ron Graham, University of California, San Diego
• How the Internet and the science of computing are on the verge of driving a revolution in medical and biological science with Leroy Hood, Institute of Systems Biology

Next year’s Big Thinkers series should be just as captivating — we’ve got a great lineup for 2010.

Webscope:
Our Webscope™ program offers a reference library of 24 interesting and scientifically useful datasets we’ve made available for noncommercial use by academics and other scientists. More than 880 academic researchers have used the datasets, resulting in 28 technical papers, journal articles and theses so far. Very few companies have the resources and global scale to help academics and students interact with the types of real-world datasets it takes to spark innovation, but Yahoo! is one of them and it’s really paid off.

Key Scientific Challenges (KSC):
Our Key Scientific Challenges program partnered with 21 graduate students in 2009. We gave them scholarships, plus the opportunity to work closely with Yahoo!’s scientists on solving some of the biggest challenges the Web offers. The winners attended the KSC Graduate Student Summit, where they presented the fruits of their labor to fellow students and other Yahoo! researchers. It was the first of what will surely be many peer-reviewed conferences for these talented folks.

Cloud Computing Research:
Our M45 Cloud Computing Cluster is a 4,000-processor testbed being used in academia for the advancement of cloud computing research and education. Faculty members at Carnegie Mellon University have written 40 technical publications based on research performed on M45, and the cluster is now being used by three additional universities. Yahoo! is also part of both the Open Cirrus testbed and the Open Cloud Consortium.

Global Impact – Yahoo! Days in Haifa:
In November we held two very successful “Yahoo! Days” at major Israeli universities – Tel Aviv University and Technion, the Israel Institute of Technology. Yahoo! scientists from around the world joined the event, which featured a keynote by Prabhakar Raghavan, the head of Yahoo! Labs. Pictures from the Tel Aviv and Technion events are available on Flickr.

Netflix Prize-winner Yehuda Koren’s Campus Tour:
Yehuda Koren, a research scientist with Yahoo! Labs in Israel, was part of a team that spanned countries, time zones, and companies, collaborating over a three-year period to win the Netflix Prize, one of the most well-publicized and interesting machine learning contests ever conceived. It’s a story with surprises, twists and turns, game-playing, late nights, and computational brute force. There’s also deep science behind it all — science that will drive future innovation on the Web. Naturally, Yahoo! thought it was the kind of story that students and faculty at some of the world’s best universities would like to hear in person. Yehuda has visited seven universities since the award was announced in September 2009, including MIT, Harvard, Carnegie Mellon, Columbia, Rutgers, Princeton, Penn, and GA Tech. And he’ll be back at in 2010.

Purple Footprints:
And, of course, we continued to sponsor our campus seminar series, leaving “purple footprints” at Carnegie Mellon University, MIT, and University of Illinois at Urbana-Champaign. All of those seminars have been recorded and are available online. Yahoos also presented more than 100 lectures, seminars, workshops and training sessions at campuses worldwide.

If you want to learn more about Yahoo! Academic Relations programs and how you can participate in 2010, please visit us.

Ron Brachman
Vice President
Yahoo! Labs

• “Integration of News Content Into Web Results” by Fernando Diaz at Second ACM International Conference on Web Search and Data Mining (WSDM)
• The work Yehuda Koren described in his paper, “Collaborative Filtering with Temporal Dynamics,” was a crucial contribution to his team. It won the Netflix prize well as the Best Paper award at the 15^th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)
• “Sources of Evidence for Vertical Selection,” by Jaime Arguello (a Yahoo! intern from Carnegie Mellon University), Fernando Diaz (Yahoo! Labs), Jamie Callan (Carnegie Mellon University), and Jean-Francois Crespo (Yahoo! Labs) at the 32^nd Annual ACM SIGIR Conference (SIGIR)
• “Generating Example Data for Dataflow Programs,” by Chris Olston, Shubham Chopra, and Utkarsh Srivastava at ACM SIGMOD/PODS Conference (SIGMOD)
• “From ‘dango’ to ‘japanese cakes’: Query Reformulation Models and Patterns,” by Francesco Bonchi and Carlos Castillo at the 2009 IEEE/WIC/ACM International Conference on Web Intelligence
• “On the Feasibility of Multi-Site Web Search Engines,” by Ricardo Baeza-Yates, Aris Gionis, Flavio Junqueira, Vassilis Plachouras, and Luca Telloli at the 18th ACM Conference on Information and Knowledge Management (CIKM)
• “Explore/Exploit Schemes for Web Content Optimization,” by Deepak Agarwal, Bee-Chung Chen and Pradheep Elango at IEEE International Conference on Data Mining (ICDM)

 
Yahoo! Labs also led the technical program at WSDM, ending up with a quarter of the total accepted papers at the event. And at the 18^th International World Wide Web Conference (WWW) in April 2009, Yahoo! Labs secured the most accepted papers from a single organization. Two of those papers — “Visual Diversification of Image Search Results,” by Reinier H. van Leuken (Universiteit Utrecht), Lluis Garcia (Yahoo! Labs), Ximena Olivares (Unversitat Pompeu Fabra), and Roelof van Zwol (Yahoo! Labs); and “Social Search in ‘Small-World’ Experiments,” by Sharad Goel, Roby Muhamad (Columbia University), and Duncan Watts — were nominated for the conference’s Best Paper award.

In addition, at SIGMOD, Bee-Chung Chen and Ashwin Machanavajjhala also earned Best Dissertation runner-up awards for their work on “Cube-Space Data Mining” and “Defining and Enforcing Privacy in Data Sharing,” respectively. 

Finally, wrapping up the year at CIKM, Yahoo! Labs topped 2008′s accomplishments and, in addition to the Best Paper award, received awards for Best Student Paper and runner-up Student Paper, and earned a record of 30 accepted papers. The awards for Best Student Paper and runner-up Student Paper went to Michalis Potamias, Francesco Bonchi, Carlos Castillo, and Aris Gionis for “Fast Shortest Path Distance Estimation in Large Networks;” and Matthijs van Leeuwen, Francesco Bonchi, Arno Siebes, and Borkur Sigurbjornsson for “Compressing Tags to Find Interesting Media Groups.” 

Though the year isn’t quite over yet, we at Yahoo! Labs are proud of our colleagues’ accomplishments, and we’re gearing up for what lies ahead in 2010. 

Prabhakar Raghavan
Head of Yahoo! Labs

This is the central problem posed by the Netflix Prize. Netflix is offering $1 million in prize money to anyone who can substantially improve (by more than 10 percent) the accuracy of its movie recommendation engine. While Netflix suggests movies based on your ratings history, the company isn’t satisfied with how well it can predict what you’ll like.

At Yahoo! Labs, this is just the kind of crazy difficult problem we love to take on. For scientists, it’s a pure challenge, requiring deep study and experimentation across a variety of fields, such as machine learning and data mining.

And for Yahoo! as a whole, these types of scientific problems also happen to be a critical element of what we most want to succeed at: connecting you with the content and information you most want in your life – even if you don’t know it yet.

That’s why we couldn’t be happier to pass along the news that Yehuda Koren, one of our scientists at Yahoo!’s Israel Lab, is part of the first qualifying team for the Netflix Prize.

Yehuda’s team, BellKor’s Pragmatic Chaos, reached first place on the Netflix Prize leaderboard on June 26, with an improvement of 10.05 percent. Achieving a more than ten percent improvement in the quality of movie recommendations is no drop in the bucket. It took Yehuda and his teammates three years to achieve and no other team has matched it yet.

Congratulations to Yehuda and his team. In the past few weeks alone, in addition to the Netflix Prize, Yehuda and his colleagues also received best paper prizes at two of the most important scientific conferences (ACM SIGMOD and ACM SIGKDD) for computer science and the Internet. Yahoo! researchers Christopher Olston, Shubham Chopra, Utkarsh Srivastava, Ashwin Machanavajjhala and Bee-Chung Chen, were also recognized for contributions to the science of how to better query and mine data, which will ultimately make it easier for you to get things done on the Web and beyond.

We may not yet have solved every problem the Internet has thrown our way, but at the very least, you should start feeling a lot more confident about those movies in your Netflix queue.

Prabhakar Raghavan
Head of Yahoo! Labs

Our research found that web searchers are regularly affronted by articles, blogs, facts, and pages that contain perspectives directly contradicting their own personal beliefs and values –- whether political, religious, economic, scientific, philosophical, etc. If consumers have the freedom in whether they navigate to the HuffingtonPost.com or FOXNews.com, why not extend that same choice to search? Until today, no other search engine could provide this level of personalization –- ensuring that consumers can search with the utmost confidence, knowing that they won’t be antagonized by their results.

Ideological Search, built on Yahoo! BOSS, is the result of extensive research conducted by virtually every scientist at Yahoo!. The team applied the latest research from the fields of sentiment analysis, intent detection, eye tracking, clustering and empathic reasoning to create this revolutionary service. We also found that adaptations were required within existing technologies to ensure ideologically-biased results. For example, Pig, the large-scale data processing environment, was not compatible with all beliefs, particularly among vegetarians. As a result we developed a sister codebase called Tofu, which proved to be more flexible and gelatinous, albeit less optimized.

To give Ideological Search a test drive, type in “global warming” or “stimulus package” and see for yourself. For more background on how we developed Ideological Search, visit the Yahoo! Labs site.

We hope this is the start of a more peaceful, conflict-free Yahoo! Search experience for you.

Try Ideological Search today!

Prabhakar Raghavan
Head of Yahoo! Labs