I’m getting psyched for my first big developer event with Yahoo!. Tomorrow we’re hosting the 3rd Annual Hadoop Summit in Santa Clara, Calif. “What’s ‘Hadoop,’” you ask, and why is it so important as to have its very own summit? Hadoop is an open-source technology that lets companies like Yahoo! crunch unimaginable volumes of data — I’m talking about 16 terabytes every day. And why does that matter? Well, it’s what makes it possible for us to create highly relevant online and mobile experiences for the 600 million people around the globe who visit Yahoo! 11 billion times a month. In short, Hadoop is the magic behind every click at Yahoo!.

I’m honored to have the opportunity to stand in front of a thousand or so Hadoop technology leaders and developers to talk about how Yahoo! uses Hadoop and the cloud to deliver the right content and the right ads to the right people every second of every day. As the world’s largest user of Hadoop, there’s no doubt in our minds here at Yahoo! that it brings us significant value and is ready for mainstream, enterprise use. In fact, we believe Hadoop will play a big part in the future of the Web.

Everyone knows that the volume of content being published and shared on the Web is growing at a crazy pace. At Yahoo!, we want to help filter out the noise by bringing you the content that’s most personally relevant to you. But doing that requires us to make sense of this BIG data and do it superfast, which would simply be impossible to do if we didn’t have the major data processing power of Hadoop. In the amount of time it takes sunlight to reach the earth (8.3 minutes), Hadoop processes more than 300 million calculations with our data.

And that’s just one example of what Hadoop can do — we’re thankful for the passionate and brilliant Hadoop community and the supportive sponsors of this week’s Summit, all of whom are doing amazing things with this technology as well. Should be a great summit — for those of you who are just dying for more geeky details, follow #hadoopsummit on Twitter, or look for our photos on Flickr.

—Blake Irving, EVP, Chief Product Officer, Yahoo!

Follow Blake Irving on Twitter: @blakei

Measuring the Unmeasurable

Key Scientific Challenges, Entry #3: Microeconomics and Social Systems

On January 27 we announced the kick-off of our 2010 Key Scientific Challenges Program.  To highlight the scientific challenge areas included in the program, we launched a series of guest blog posts earlier this month on Yodel Anecdotal. Read our previous post on privacy and security, “Data, Data Everywhere, but How to Keep it Safe.”

Another big challenge are Yahoo!’s research scientists are continually examining is microeconomics and social systems. In this entry, Sharad Goel from Yahoo! Labs shares some thoughts on how Yahoo! is tackling the new opportunities for research into the social sciences that the Web is making possible and why it’s a fascinating field.

What do your friends really know about you? How much do they influence your decisions? How often do we stray from the cultural herd? How do groups organize to solve complex problems?

Answers to such fundamental questions about social behavior have often eluded us. With microscopes we peered into the intangibly small building blocks of life, and with telescopes we found our place in an unimaginably expansive universe. But without the tools to faithfully document human activity—a challenge that by comparison seems so palpable—we had no way to investigate the inner workings of our own communities. Now with an explosion of information on every aspect of our everyday existence—from what we buy, to where we travel, to whom we know—we can measure what until quite recently was thought unmeasurable. In the Microeconomics and Social Systems Group at Yahoo! Labs, we are using this proliferation of data to explore how societies function. It’s a fascinating area of study that is just beginning to shed light on new layers of human behavior, making it a perfect fit for the Key Scientific Challenges Program.

In a recent study that’s garnered some attention, for example, we asked, “How eccentric are people?” Looking at consumer preferences across movies, music, and web browsing, we came to the surprising conclusion that ordinary people have pretty extraordinary tastes. In particular, we found that typical Netflix and Yahoo! Music users regularly watch movies and listen to songs that are not even available in the largest brick-and-mortar retailers. This result not only challenges stereotypes about people blindly following the herd, but also highlights the importance of offering consumers broad selection. That is, specialty products may dramatically boost user satisfaction by providing buyers the convenience of “one-stop shopping” for both their mainstream and niche interests.

In other work, we used web search queries to forecast the commercial success of movies, songs, and video games. Weeks, sometime even months, before a movie opens or a video game is released, one can find traces of pent up consumer demand in the search query logs. We found that these telltale signs of early interest are remarkably good predictors of future success. The catch? Although the search logs do reflect user intent, more mundane indicators, such as production budgets and reviewer ratings, perform equally well at forecasting sales. Thus, the benefit of web search as a prediction tool may have less to do with its superiority over other methods than with its generality, low cost, and real-time nature.

At a time when we are drowning in data, at Yahoo! Labs we’re asking a simple question: what can you do with it? The answer is limited only by our imaginations.