This week, IBM’s supercomputer, Watson (named after IBM’s founder, Thomas J. Watson), took on two of the most championed Jeopardy! contestants of all time in an exhilarating $1 million Jeopardy! face-off between man and machine.

Watson defeated Jeopardy! defenders Ken Jennings and Brad Rutter, amassing $77,147 in winnings in a nail-biting three-night tournament that sparked interest around the field of artificial intelligence and data analytics.

IBM explained, that by matching the text in a question to the text in its vast memory, Watson can analyze and recite an accurate answer in less than three seconds. If there is no match in Watson’s “brain,” it takes a guess based on a confidence level that is calculated on probabilities.

So what makes Watson’s genius possible? A whole lot of storage, sophisticated hardware, super fast processors and Apache Hadoop, the open source technology pioneered by Yahoo! and at the epicenter of big data and cloud computing.

Hadoop was used to create Watson’s “brain,” or the database of knowledge and facilitation of Watson’s processing of enormously large volumes of data in milliseconds. Watson depends on 200 million pages of content and 500 gigabytes of preprocessed information to answer Jeopardy questions. That huge catalog of documents has to be searchable in seconds. On a single computer, it would be impossible to do, but by using Hadoop and dividing the work on to many computers it can be done.

In 2005, Yahoo! created Hadoop and since then has been the most active contributor to Apache Hadoop, contributing over 70 percent of the code and running the world’s largest Hadoop implementation, with more than 40,000 servers. As a point of reference, our Hadoop implementation processes 1.5 times the amount of data in the printed collections in the Library of Congress per day, approximately 16 terabytes of data.

We’ve been doing it because we think it’s a game-changer for the Internet. Hadoop is critical to Yahoo!’s business, delivering personalized experiences to our more than 630 million users worldwide. Yahoo! Mail uses Hadoop to fight spam, the Yahoo! home page content is personalized with Hadoop and a suite of our personalization technologies. What these have in common is that they require processing huge amounts of data very quickly and reliably on large numbers of computers, mirroring Waston’s requirements to win Jeopardy! And just like Hadoop was critical to Watson’s success, it has a fundamental and direct impact on Yahoo!’s performance and bottom-line.

So if you’re ever on Jeopardy!…”It is the technology behind every click on Yahoo! and the IBM supercomputer that accomplished the impossible….”

“What is Apache Hadoop!?”

For hundreds of millions of people around the world, email is a critical part of their online communications, connecting them to the people who matter most to them. With that in mind, we are rolling out Yahoo! Mail Beta — a faster, safer, more personally meaningful communication experience – seamlessly across PC, mobile and tablet devices. Here’s what we focused on:

Blazing Speed: We’ve re-architected Yahoo! Mail Beta from the ground up to provide lightning-quick performance. It’s at least twice as fast as previous Yahoo! Mail versions, and significantly faster in key geographies. The Yahoo! Mail team has made several technological advancements to make this happen, from utilizing Yahoo!’s cloud technology to focusing on user-flow optimization. Check out our post on the YDN blog for more on the great technology that’s making this possible.

Reaching Everyone: There are even more ways to share your stories with the people who matter most to you in Yahoo! Mail Beta – no matter what device or what service they use. We’ve added Twitter integration alongside Facebook, as well as richer IM and SMS experieneces, offering you more ways to share directly from your inbox. You can tweet or retweet or send a text message out to friends while pulling together an email.  And you can also easily check out photos and videos from Flickr, Picasa, and YouTube via a new feature that automatically shows slideshows right in your emails when your friends send you links from these sites.

Simple yet Powerful Search: When you’ve got unlimited storage (as you do with with Yahoo Mail) – you have a lot of email!  Our new search interface helps you get the messages you need quickly and easily.  You have the power of sophisticated searches at your fingertips, allowing you to sort and refine by sender, attachment file, date, or folder location.

Stay Safe From Spam: We all know spam is incredibly annoying; it can also be malicious. That’s why we’re proud to be No. 1 in spam protection, blocking over 130 billion spam messages each month. And Yahoo! Mail Beta gives you an even better spam-free mail experience by leveraging the power of Hadoop technology to keep you one step ahead of the spammers.

Take it With You: With this new communications experience, we’re making sure your encounters will be equally fabulous whether you’re emailing from your laptop, iPhone, iPad or Android device. No more relearning how to get things done each time you switch from one connected device to another.

So wherever you are in the world, check out Yahoo! Mail Beta now – a single speedy, simple, safe, and seamless stop for all your online communications.  We’ll be making lots of enhancements and improvements during this beta period, and we can’t wait to hear your feedback.  For more details about Yahoo! Mail Beta, visit the Yahoo! Mail blog, and to find out about related policy updates, check out the Yahoo! Policy blog.

- Kakul Srivastava, vice president of product management, Communications and Communities

I’m honored to have the opportunity to stand in front of a thousand or so Hadoop technology leaders and developers to talk about how Yahoo! uses Hadoop and the cloud to deliver the right content and the right ads to the right people every second of every day. As the world’s largest user of Hadoop, there’s no doubt in our minds here at Yahoo! that it brings us significant value and is ready for mainstream, enterprise use. In fact, we believe Hadoop will play a big part in the future of the Web.

Everyone knows that the volume of content being published and shared on the Web is growing at a crazy pace. At Yahoo!, we want to help filter out the noise by bringing you the content that’s most personally relevant to you. But doing that requires us to make sense of this BIG data and do it superfast, which would simply be impossible to do if we didn’t have the major data processing power of Hadoop. In the amount of time it takes sunlight to reach the earth (8.3 minutes), Hadoop processes more than 300 million calculations with our data.

And that’s just one example of what Hadoop can do — we’re thankful for the passionate and brilliant Hadoop community and the supportive sponsors of this week’s Summit, all of whom are doing amazing things with this technology as well. Should be a great summit — for those of you who are just dying for more geeky details, follow #hadoopsummit on Twitter, or look for our photos on Flickr.

—Blake Irving, EVP, Chief Product Officer, Yahoo!

Follow Blake Irving on Twitter: @blakei

Starting this week, we will be bringing this vision to life in the Yahoo! products you use every day.  We are starting to integrate Facebook in a big way, a result of the agreement we first signed with Facebook last December. People who use Yahoo! and Facebook can now link their accounts to view and share updates with friends across both networks.  This means you can see your Facebook News Feed from your Yahoo! homepage, Yahoo! Mail, and other favorite Yahoo! sites and services.  When you create and share content on our properties – including Yahoo! News, Yahoo! Sports, Yahoo! Answers, and many of our entertainment sites, such as omg!, Yahoo! TV, and Yahoo! Music – you can share that content on Facebook.

Beyond Facebook, your activities on Yahoo! are another important way of bringing social experiences together. Globally, Yahoo! now has more 80 sites producing socially infused content, including three million photos and videos uploaded daily to Flickr, 1.1 million answers provided daily on Yahoo! Answers, and thousands of comments posted daily on articles across Yahoo! sites. These social actions – what we call Yahoo! Updates – can be found in many places across Yahoo!, within products you use and on Yahoo! pages you visit every day.

Our vision is to provide a central hub for people’s online lives, and we understand that requires a Web you can trust.  That’s why this week Yahoo! is making privacy settings even easier to use with a central dashboard.  Rolling out this week as a re-fresh of Yahoo! Profiles, Yahoo! Pulse is a dashboard that makes it simple to set and manage your activities and privacy on Yahoo!.

Expect more to come and until then, share, update your status, and tell us what you think.

Jim Stoneham, Vice President of Communities

Cody Simms, Senior Director, Yahoo! Social Platforms and Yahoo! Developer Network (YDN)

At last year’s conference held in Madrid, Spain, Yahoo! presented 14 refereed papers and Ricardo Baeza-Yates, head of Yahoo! Labs Barcelona, gave a keynote on mining Web 2.0 data for Search. This year, Yahoo! raised the stakes again, with 21 papers accepted at the 2010 event (out of a total of 105 accepted papers for the entire event), as well as a variety of posters, demos and panels, including a featured discussion on the future of Search and the core technologies of the Web.

As usual, the Yahoo! booth at the conference attracted a lot of interest from attendees. Some of the highlights include demonstrations we are giving of some experimental projects to gather data and test our theories. Because not everyone can attend WWW, we thought it would be fun to give everyone a quick look-in from the booth with some short clips of our scientists and their demos.

First up, is Predictalot, a combinatorial prediction market game we debuted to test the “wisdom of the crowds” during March Madness and that will be making a global return to prominence for World Cup 2010 in South Africa:

Next up is Statler, a tool for developing a model of engagement for how users interact with content over time. In this case, Statler is analyzing public Twitter feeds around major events, like the 2008 Presidential debates and Barack Obama’s inauguration. The goal of Statler and the associated research is to be able to predict which items are likely to resonate over time, rather than spike in popularity before quickly fading away.

Our next demo, Ranking Entity Facets, is a technical name for a technology we’ve deployed at Yahoo! Search to make it easier for people to explore all kinds of people, movies, locations and other “entities” related to what they’re searching for at any given time. What makes this capability exciting is that it relies on the science and technology powering Yahoo!’s Web of Things platform where Yahoo! is able to mine all kinds of content and data sources to synthesize a Web page for you with all the information you need on a topic, rather than just a list of Web links.

And then there is Wrapper Induction, developed by our data mining group in Labs, which is all about extracting valuable information from Web pages.

For a full recap of all the events at WWW2010 this week, make sure to check out the Yahoo! Labs Web site and follow us on Twitter @YahooLabs. You can also see all Tweets related to WWW2010 at #WWW2010 and follow our Flickr stream.

Lin Koh
Yahoo! Labs

P.S. If you want to check out what fun is, here’s a cryptogram we shared with folks at WWW as part of a booth contest. Perhaps you’d like to take a crack at solving it too.

We just announced with Microsoft that we have reached clearance for our search agreement, without restrictions, from both the U.S. Department of Justice and the European Commission.  We’ll now turn our attention to beginning the process of implementing the deal.

Here are some useful links to get more detailed information about today’s developments:

Search Alliance Official Website

Official Press Release

Yahoo! Search Blog

YDN Blog

Yahoo! Search Marketing Blog

Yahoo! Transition Center

Video: Yahoo! CEO Carol Bartz discusses Yahoo! & Microsoft’s new Search Alliance: Click here

Video: Yahoo! CEO Carol Bartz talks about Yahoo’s continued innovation in search: Click here

Official US DOJ Antitrust Division statement: Click here

We will continue to update Yodel with new information today and throughout the process.

Open is at the core of many Yahoo! initiatives and products. Over the past two years Yahoo! has been hard at work to change how we develop products and interact with our developer community. In addition to being avid supporters of Open Source, we participated and adopted community-based specifications such as OpenID and OAuth, and were a founding member of the OpenSocial and OpenID Foundations.  As you can see, we love Open.

But Open isn’t always easy.

Whenever we (or other companies) engage in a collaborative effort with a wider community, we are faced not only with technical challenges, but with the complex reality of intellectual property law. Patents, copyright, and trademarks are not what geek dreams are made of. This is why we have actively supported the creation of the Open Web Foundation, an organization dedicated to the creation of an open, free, and community-driven environment for the development of technical specifications:

The Open Web Foundation was founded to help developer communities collaborate and share technical innovation on the web, bringing to the world of formats and protocols the same successful grassroots approaches established by the open source community. Modeled after the Apache Software Foundation and Creative Commons, the Open Web Foundation seeks to facilitate the creation and implementation of specifications with legal agreements that make such work simple, safe, and sustainable.

For the past year, the Open Web Foundation Legal Committee has been hard at work on a new legal agreement for licensing of open specifications. While Open Source software enjoys a wide range of licenses for making software freely available, specifications and standards are usually licensed under a complex set of rules and conditions. These licenses are hard to read and spread over many pages full of terms even many lawyers don’t fully understand. There was also no suitable standalone agreement available for companies and communities to use when making their work available, forcing them into long and costly legal negotiation between the contributors.

Today, the Open Web Foundation is announcing the availability of the Open Web Foundation Agreement (OWFa), a reusable and straight-forward legal document, designed to be easily adopted by a wide range of specification communities and organizations.  Specifications made available under the Open Web Foundation Agreement may include everything from small ad-hoc formats sketched out among friends to large multi-corporation collaborations that ultimately grow into internationally recognized standards with the help of formal standards- setting organizations.

But what makes this agreement even more valuable today, is a commitment by a group of leading companies and individual contributors to apply it to a growing list of specifications. Today, Yahoo! is joining Facebook, Google, Microsoft, and others in making available the following specifications under the OWFa:

* OAuth Core 1.0 Revision A
* OAuth WRAP 0.9
* Simple Web Tokens 0.9

We are also releasing the Media RSS (mRSS) specification under the Open Web Foundation Agreement. Media RSS is an RSS extension used to syndicate rich media content (instead of just text). The specification is used when providers of media content want to share that content with a third party, usually a search engine like Yahoo! Video Search. We are also in the process of transitioning the mRSS specification to the capable hands of the RSS Advisory Board for future development.

This is just the first step in what we hope will be a new path for open collaboration and innovation on the web. The best innovation happens when we let our talented engineers and product managers solve problems. This is one less thing for them to worry about.

Eran Hammer-Lahav

Director of Standards Development, Yahoo!
President, The Open Web Foundation