In light of this, with the roll out of the new Yahoo! Mail last May, we recently launched more advanced anti-phishing defenses and enhanced spam protection to help protect your inbox.  And we’re happy to report its success.  The Yahoo! team put together new technology that’s now blocking nearly 550 billion spam messages from hitting your inboxes each month – that’s approximately 1,800 emails for every Yahoo! Mail user.

We think the technology we’re using to keep your mailboxes safe is quite awesome and we wanted to show you how we do it, so today we’re going live with the Yahoo! Mail Visualization Project – a view of what no one has seen before using live data…how we use cloud computing and Apache Hadoop technology to filter spam and re-route email for the 300 million mail users we have across the globe.

We won’t bore you with the techy details in writing because it’s much more compelling to see…

So why is it important for us to reveal how Yahoo! Mail works? Because we’re proud to show the science and technology powering the products you use every day.  Through the visualization project, we can demonstrate the technology behind the Yahoo! Network and show the impact that the tech has on you, the consumers of Yahoo!’s great technology.  As a quick example, the anti-spam protection we mentioned earlier uses Hadoop to aggregate anonymous data from the billions of emails sent and received each day and as a result has helped us reduce spam reports by 65%.  Our technology analyzes all this anonymous data (with the help of Hadoop) and identifies spam patterns so we can then use really intelligent algorithms to predict future email patterns that will differentiate “good” and “bad” senders.

To give some perspective on how powerful our computing technology is, think about these fun facts:

• For every loving message from mom, Yahoo! Mail blocks 4 spam messages from the people she warned you about.
• In the time it takes you to read this, our Hadoop software (the brain behind Yahoo! Mail) has learned from more than 4 million user actions.
• In the time it takes sunlight to reach earth, Hadoop processes 332 million user actions, helping reduce spam in Yahoo! Mail.
• Hadoop processes 1 terabyte of data (equivalent to 4,000 years of continuous typing) in the time it takes the fastest human to run a mile.
• We add over 70 million Yahoo! Mail accounts per year – that’s the combined population of London, NYC, Paris and Tokyo.

With that we’ll leave you with the link to the Mail Visualization project.  It’s super interactive and ready for you to check it out for yourself: http://visualize.yahoo.com/

-Markus Weimer, Yahoo! Labs
-Andreas Neumann, Grid Architect

Editor’s Note: This article is part of a series entitled, “In My Own Words,” that gives employees the opportunity to share their own stories about Yahoo!. Feel like yodeling your own? Post your Yahoo! story on your social networks using the hashtag #myYahoostory.

Yahoo!’s impact at the event is huge.  We use knowledge discovery and data mining every day to make the Yahoo! experience deeply personal and relevant for our users and advertisers alike. For example, as we described earlier this month, we use these principles to personalize the Yahoo! homepage with 45,000 different variations of our “Today Module” every five minutes. If you’re interested in learning more about why data is so important to Yahoo!, check out this video interview with some of our leading scientists that recently appeared on Forbes.com too.

We have some of our best researchers and top talent joining scientists from Stanford, Rutgers and Princeton, as well as the research labs of other leaders in the field, to share insight and case studies around such things as social media analytics, Internet ad systems, behavioral analytics and Hadoop.

We have Yahoo! researchers flying in from across the globe, presenting a total of seventeen papers, two tutorials, and of course, the KDD Cup!

This year’s KDD cup competition centered on Yahoo! Music, which has amassed billions of user ratings for a variety of musical pieces.  The competition challenged scientists to work with the raw ratings that encode information on how songs are grouped, which hidden patterns link various albums, which artists complement each other, and which songs users would like to listen to. Check out the details here: external KDD Cup Website.

Here’s a photo of Here’s a photo of one of the winning teams (Team Commendo) claiming their prize on Sunday.

We’re proud to have such a great team of professionals representing Yahoo! and driving scientific leadership across the industry.

More than 400 women from across the leading tech companies will meet, network and share their wealth of tech experience, including diverse perspectives from women at different stages of their careers. Guests can participate in an informal Yahoo! panel discussion on ensuring the success of women in technology in the future, as well as a host of other fun events that will run throughout the night. There will be two photo booths, a “Women in History” Jeopardy game and an anagram puzzle station. Attendees will also be able to experience Yahoo! products, such as Livestand, Connected TV, Hadoop and YSlow – all of which have had major contributions from Yahoo!’s female technologists.

So you may be asking, what is a girl geek?  We define girl geeks as women who are passionate about technology and are making a difference in their field each and every day.  The organization behind the dinner, Yahoo! Women in Tech, touts more than 600 members across the globe and focuses on empowering female tech professionals to succeed, as well as encouraging K-12 girls to pursue technical careers.  The group is thrilled to host this event again and showcase the incredible contributions and achievements of both women at Yahoo! and the wider industry.

We hope you can join us for an informative and enjoyable evening that celebrates your efforts and achievements in tech!

This week, IBM’s supercomputer, Watson (named after IBM’s founder, Thomas J. Watson), took on two of the most championed Jeopardy! contestants of all time in an exhilarating $1 million Jeopardy! face-off between man and machine.

Watson defeated Jeopardy! defenders Ken Jennings and Brad Rutter, amassing $77,147 in winnings in a nail-biting three-night tournament that sparked interest around the field of artificial intelligence and data analytics.

IBM explained, that by matching the text in a question to the text in its vast memory, Watson can analyze and recite an accurate answer in less than three seconds. If there is no match in Watson’s “brain,” it takes a guess based on a confidence level that is calculated on probabilities.

So what makes Watson’s genius possible? A whole lot of storage, sophisticated hardware, super fast processors and Apache Hadoop, the open source technology pioneered by Yahoo! and at the epicenter of big data and cloud computing.

Hadoop was used to create Watson’s “brain,” or the database of knowledge and facilitation of Watson’s processing of enormously large volumes of data in milliseconds. Watson depends on 200 million pages of content and 500 gigabytes of preprocessed information to answer Jeopardy questions. That huge catalog of documents has to be searchable in seconds. On a single computer, it would be impossible to do, but by using Hadoop and dividing the work on to many computers it can be done.

In 2005, Yahoo! created Hadoop and since then has been the most active contributor to Apache Hadoop, contributing over 70 percent of the code and running the world’s largest Hadoop implementation, with more than 40,000 servers. As a point of reference, our Hadoop implementation processes 1.5 times the amount of data in the printed collections in the Library of Congress per day, approximately 16 terabytes of data.

We’ve been doing it because we think it’s a game-changer for the Internet. Hadoop is critical to Yahoo!’s business, delivering personalized experiences to our more than 630 million users worldwide. Yahoo! Mail uses Hadoop to fight spam, the Yahoo! home page content is personalized with Hadoop and a suite of our personalization technologies. What these have in common is that they require processing huge amounts of data very quickly and reliably on large numbers of computers, mirroring Waston’s requirements to win Jeopardy! And just like Hadoop was critical to Watson’s success, it has a fundamental and direct impact on Yahoo!’s performance and bottom-line.

So if you’re ever on Jeopardy!…”It is the technology behind every click on Yahoo! and the IBM supercomputer that accomplished the impossible….”

“What is Apache Hadoop!?”

While most of us are satisfied with using pi to just two decimal places (3.14) to find the area of a circle, Nicholas is one in a long line of mathematicians and engineers who wants a bigger slice of pi. And so, we now know: The Two Quadrillionth Bit of π is 0!
Yes, you read that right; “Two Quadrillionth.” Wow.

The circumference of the visible universe can be calculated using a mere 39 or 40 decimal places, so what are Yahoos like Nicholas doing out at those outer reaches? For Nicholas, this was a project ideally suited to flex the muscles of Hadoop, the open source technology at the epicenter of big data and cloud computing.  The computation took 23 full days and required 1000 different machines using Hadoop.

Eric Baldeschwieler, Yahoo!’s vice president of Hadoop engineering, explains how this relates to our business. “We are a technology company with big data at our core, so it’s exciting to see a new world record that demonstrates the sheer power of distributed systems like Hadoop.”  Yahoo! engineers and scientists have been developing Hadoop since 2005 with a goal similar to Nicholas’ in mind – to push the limits of technology to better understand the complex algorithms and equations that make our business tick.

“The more we can tap our infrastructure to do heavy lifting, the more quickly we can innovate,” said Baldeschwieler.

Learning how to use such computing power, especially across distributed systems involving many computers, allows engineers to develop techniques for speeding up other calculations and algorithms. Faster data crunching translates into more exciting, relevant and fun Internet experiences for Yahoo! users.

Mostly, though, it’s just really, really cool. Congratulations, Nicholas and cheers to the power of Hadoop!

See the story by BBC News for additional details.

I’m honored to have the opportunity to stand in front of a thousand or so Hadoop technology leaders and developers to talk about how Yahoo! uses Hadoop and the cloud to deliver the right content and the right ads to the right people every second of every day. As the world’s largest user of Hadoop, there’s no doubt in our minds here at Yahoo! that it brings us significant value and is ready for mainstream, enterprise use. In fact, we believe Hadoop will play a big part in the future of the Web.

Everyone knows that the volume of content being published and shared on the Web is growing at a crazy pace. At Yahoo!, we want to help filter out the noise by bringing you the content that’s most personally relevant to you. But doing that requires us to make sense of this BIG data and do it superfast, which would simply be impossible to do if we didn’t have the major data processing power of Hadoop. In the amount of time it takes sunlight to reach the earth (8.3 minutes), Hadoop processes more than 300 million calculations with our data.

And that’s just one example of what Hadoop can do — we’re thankful for the passionate and brilliant Hadoop community and the supportive sponsors of this week’s Summit, all of whom are doing amazing things with this technology as well. Should be a great summit — for those of you who are just dying for more geeky details, follow #hadoopsummit on Twitter, or look for our photos on Flickr.

—Blake Irving, EVP, Chief Product Officer, Yahoo!

Follow Blake Irving on Twitter: @blakei

That’s why we’re becoming a platinum sponsor of the Apache Software Foundation, the non-profit that provides support for open source projects like the Apache HTTP server (which powers much of the Yahoo! network and has for more than 10 years), Lucene (the search software widely used on the Web), and Hadoop (which lets developers easily write and run applications that process massive amounts of data). All of these are community-driven efforts that will help define the future of web services.

We’re a big believer in giving back to this community After all, it could be the source of the next big breakthrough. You may recall that we recently unleashed a supercomputer, which researchers at Carnegie Mellon University are using as their distributed computing playground. We also have several engineers actively contributing to Apache’s Hadoop. In fact, we’re the project’s largest contributor and were so impressed with its potential, that we hired its founder, Doug Cutting, who’s also a VP at Apache.

We’ve been a huge open source beneficiary and hope our support of Apache will continue to improve the quality and expand the usage of critical software that will propel the entire industry.

Jay Kistler
VP, Systems, Tools and Services
Search & Advertising Technology Group