Originally launched in November 2007, Yahoo! and its M45 program are providing universities the opportunity to conduct research otherwise impossible without the power and speed of a supercomputing resource, which consists of approximately 4,000 processors.

Hadoop, the open source technology at the epicenter of big data and cloud computing, is the core data analysis technology used across Yahoo!, and is used by all the universities participating in the M45 research initiative. Yahoo! benefits from university contributions to the Hadoop code base as well as through insights from cutting edge research initiatives conducted on Yahoo’s M45 supercomputer.

Examples of academic research conducted on the M45 include two of the world’s largest knowledge acquisition research projects – Never Ending Language Learning System (NELL) at Carnegie Mellon University and KnowItAll at the University of Washington.

One of the hottest research trends at universities today is the merging of mobile computing and cloud computing. By expanding the M45 platform to support both mobile and cloud computing research, Yahoo! is enabling top tier universities to tackle some of the industry’s most critical computing challenges.

Additional university research projects facilitated on Yahoo!’s M45 supercomputer include

• Carnegie Mellon – performing research in large-scale graph mining (graphs with billions of nodes), text search and analysis, statistical natural language processing, analysis of media Internet traffic, statistical machine translation, learning to read the Web and file systems research.
• Cornell – exploring the use of advanced methods from computer science to help solve environmental and broader sustainability challenges.  As part of its Citizen Science research program, Cornell researchers are exploring methods to enable a smart phone application to enter bird observation data into the cloud.
• Purdue – combining the power of mobile and cloud computing to develop a context-aware navigation system for blind and visually impaired people.  Researchers also plan to investigate topics in cloud data privacy, information retrieval and the automatic off-loading of computation from mobile devices to the cloud.
• Stanford – merging large-scale cloud computing techniques and advanced statistical machine learning methods to analyze the vast amount of text, image and network data now available, on the web and elsewhere.  The goal is to achieve a new level of understanding of the semantics latent in various media, attaining systems with greater artificial intelligence.
• University of California at Berkeley – analyzing social networks and studying population genetics, as well as testing new architectures for collecting traffic data from GPS-equipped mobile phones and estimating traffic conditions in real-time, analyzing climate-change satellite data, prototyping new scientific applications and improving cluster scheduling and reliability.
• University of Massachusetts Amherst – investigating efficient inference of the relations among documents and passages in a large collection of books.
• University of Michigan at Ann Arbor – profiling and understanding MapReduce job execution, optimizing energy consumption and performance for heterogeneous MapReduce workloads, performing large-scale language model analysis and investigating how to offer a geo-distributed service that can be easily accessed through mobile phones for content distribution, computation offloading and other services.
• University of Washington – studying scalable scientific data management and large-scale knowledge acquisition, and building a large knowledge base that can be queried both by Web and mobile phone.

Yahoo! is also a founding member of the Open Cirrus Cloud Computing Testbed and the Open Cloud Consortium, both facilitating scientific research in the cloud.  Combined, these programs bring the collective wisdom of many of the world’s top scientific researchers to Yahoo!’s industry-leading research portfolio at Yahoo! Labs.

While most of us are satisfied with using pi to just two decimal places (3.14) to find the area of a circle, Nicholas is one in a long line of mathematicians and engineers who wants a bigger slice of pi. And so, we now know: The Two Quadrillionth Bit of π is 0!
Yes, you read that right; “Two Quadrillionth.” Wow.

The circumference of the visible universe can be calculated using a mere 39 or 40 decimal places, so what are Yahoos like Nicholas doing out at those outer reaches? For Nicholas, this was a project ideally suited to flex the muscles of Hadoop, the open source technology at the epicenter of big data and cloud computing.  The computation took 23 full days and required 1000 different machines using Hadoop.

Eric Baldeschwieler, Yahoo!’s vice president of Hadoop engineering, explains how this relates to our business. “We are a technology company with big data at our core, so it’s exciting to see a new world record that demonstrates the sheer power of distributed systems like Hadoop.”  Yahoo! engineers and scientists have been developing Hadoop since 2005 with a goal similar to Nicholas’ in mind – to push the limits of technology to better understand the complex algorithms and equations that make our business tick.

“The more we can tap our infrastructure to do heavy lifting, the more quickly we can innovate,” said Baldeschwieler.

Learning how to use such computing power, especially across distributed systems involving many computers, allows engineers to develop techniques for speeding up other calculations and algorithms. Faster data crunching translates into more exciting, relevant and fun Internet experiences for Yahoo! users.

Mostly, though, it’s just really, really cool. Congratulations, Nicholas and cheers to the power of Hadoop!

See the story by BBC News for additional details.

This is significant because data centers represent the majority of our energy consumption. Keeping Yahoo! running smoothly for more than 500 million people around the world calls for a lot of server power. So we’ve made it a priority to become a leader in designing and building data centers that are environmentally sustainable, investing millions to design facilities that make the best use of the energy we consume.

Here’s what makes us so proud of our future New York data center plans. First, it will be powered by one of the cleanest utilities in the country – fed predominantly by renewable hydroelectric power from Niagara Falls. And second, a record 90% of that energy will power the servers. To put that in context, the industry average is 50% or lower, with the other half dedicated mostly to keeping the servers cool.

For data center geeks, we expect our Buffalo Lockport, NY, data center design will have an annualized average PUE (power usage effectiveness) of 1.1 or better. To achieve that, we’ve come up with a unique building design that we call the Yahoo! Computing Coop (because it looks like something chickens live in), which is angled to take advantage of Buffalo’s microclimate, using 100% outside air to cool the servers.

We’ve been pushing green data center standards since we started building our own data centers two years ago. For example, our facilities in Washington are powered by zero-carbon wind and hydroelectric sources, and we use free cooling for most of the year, dropping energy consumption by 40-50%. As we build more capacity to meet demand, we’ll continue to focus on innovations and inventions that improve energy efficiency. And we’ve been sharing best practices to encourage the entire industry to put smarter policies in play.

And we’ll continue to push ourselves hard to lower our impact. Today we’re committing to reduce the carbon intensity of our data centers by at least 40% by 2014. In other words, we’ll decrease our average electricity consumption and greenhouse gas emissions from our data centers around the world. We’ll get there through a combination of innovative data center design, improving how we utilize our servers, cloud computing, and locating our data centers in areas where cleaner energy is available.

Reducing our carbon footprint has always been a priority and we’ve decided to focus all our energy and investment on that philosophy. We will no longer purchase carbon offsets as announced in 2007. Instead, we’ll focus our resources on reducing our carbon impact while helping the rest of the industry do the same. We believe creating highly-efficient data centers will have a greater long-term, direct impact on the environment and gives us the best opportunity to play a leadership role in addressing climate change.

So the next time you check your email, do a Yahoo! search, or get the latest environmental info on Yahoo! Green, you can feel good about putting some of the greenest data centers in the industry to work.

David Filo
Co-founder and Chief Yahoo

Academic research is facing new challenges in today’s Internet age. Universities often don’t have the equipment –- hardware and software -– to maintain in-depth research at Internet scale. Academic researchers are limited in the research they can conduct, and this, ultimately constrains the amount of large-scale Web innovations coming to the marketplace.

Here at Yahoo!, we believe in open and collaborative research as the best way towards building the next generation of the Web. As part of our dynamic Academic Relations program, we’re teaming up with academia, as well as other companies and governments across the globe, to invest in and pool together the large-scale computers that will let researchers conduct truly breakthrough work on cloud computing and data storage systems.

The HP/Intel/Yahoo! Cloud Research Testbed is a significant step in that it will not only allow researchers to run applications and data on large-scale supercomputers, they will be able to experiment and conduct research on a massive scale. It’s like letting them simulate a true Web environment and that’s exactly what you need to ensure which ideas will work in the wild.

Joining us in this first-ever large-scale international consortium are Infocomm Development Authority of Singapore (IDA), the University of Illinois at Urbana-Champaign (UIUC), and the Karlsruhe Institute of Technology in Germany, with contributions from the National Science Foundation (NSF) as well.

This is all part of Yahoo!’s overall focus and investment in cloud computing and data infrastructure. We recently announced the formation of the Cloud Computing and Data Infrastructure Group (CCDI), a new group dedicated to building out our next-generation cloud infrastructure.

In addition, in November 2007, Yahoo! deployed a supercomputing-class data center, called the M45, for cloud computing research; Carnegie Mellon University was the first institution to take advantage of this supercomputer. In March 2008, Yahoo! announced an agreement with Computational Research Laboratories (CRL) to jointly support cloud computing research in India. The CRL supercomputer is one of the world’s top five supercomputers and is the first supercomputer available to academic institutions in India.

I speak for our many research scientists when I say we’re excited about this open testbed and being able to collaborate with leaders who share our same vision. The HP/Intel/Yahoo! Cloud Research Testbed is a truly global research effort, and more partners and researchers will be invited to join and participate in the program when all of the systems are up and running later this year. The sky is the limit from here on out.

Prabhakar Raghavan
Head of Yahoo! Research

Photo from Nicholas_T