On April 10th, after a very sleepless 24-hour hacking period at the University of California, San Diego (UCSD), the fourth season of Yahoo!’s Hack U™ season came to an official close. For those of you who haven’t read about or participated in a Yahoo! hack event before, it’s a very “Yahoo!” event. Programmers and creative techies of all kinds get together for 24-hour sessions where the only goal is to stay awake and, hopefully, create a new application or Web experience (a “hack”) from scratch that can be demoed and judged by their peers at the end of the event.

Yahoo! hosts different types of hack events, some just for internal employees in different offices around the world (such as Spring Hack, which we’re holding today at our headquarters in Sunnyvale), and some that are open to all interested hackers who feel like taking on the challenge. We call these Open Hack Days, and they’ve also been held everywhere, from Brazil, England and India, to our most recent U.S. event in New York.

As you can imagine, students are always game for a hack day challenge and while many of them have participated in our Open Hack Days (even winning some), we also decided that they deserved a series of their own. And so “Hack U™” was born.

The Hack U™ program officially started in late 2006. The week of talks leading up to the student hack competition focuses on Web programming curriculum support, which isn’t typically taught in traditional university computer science departments. From the beginning, it was clear that the students were hungry for knowledge in this space, even if it meant they had to teach themselves.

The hands-on experience of hack events was so popular that it quickly gained momentum and is now a regularly requested event for our Academic Relations team, by both faculty and students alike. Today, the volunteer-based Hack U™ team, which includes Douglas Crockford, the well-known inventor of JSON and JavaScript guru, rotate campuses each year and keep a waiting list in order to accommodate all of the universities who want to take part in the event. Thankfully the tireless Yahoo! developers are passionate about teaching and hacking and believe that these endlessly inventive students hold the answers to a new generation of Web experiences.

At this year’s UCSD event, which closed out the spring season, David Vanoni and his teammates, all computer science students at UCSD, hadn’t slept in over 24 hours, but you wouldn’t have known it from watching the energetic pitch they gave to the judges about their “RockmyWorld” iPhone app. It was the second year David had participated in the Hack U™ competition and it was obvious from the moment he handed each of the judges their own iPhone and confidently walked them through the demo of their slick music locator app that he had come here to win. Luckily the judges agreed. David and his team are now part of an exclusive club along with the winning entries from seven other university hack teams at Carnegie Mellon; University of California, Berkeley; University of Washington; University of Texas, Austin, University of Illinois, Urbana-Champaign, University of Toronto and Georgia Tech.

Full Hack U™ recaps for this season, including more information about the winning hacks and hottest trends, visit the Yahoo! Developer Network. Here are the links from spring:

Georgia Tech
Texas at Austin
Bangalore
Hyderabad
University of Washington
UCSD

The next season of Hack U™ begins in fall 2010 and word has it there will be an entrepreneurial spin on the competitions. Stay tuned for more details and check out the developer network blog for more detailed campus overviews, pictures and the complete list of winning hacks.

-Jamie Lockwood, Academic Relations Yahoo! Labs

The Internet is 27 years old, give or take a few years and depending on whom you’re asking. But while the Web has changed a lot in our lives the last couple decades, by historical standards the science of the Internet is still relatively young.

Let me explain: it’s cliché to compare the Internet to Gutenberg’s invention of the printing press, but we forget that it took more than 200 years after the invention of the printing press until we experienced the rise of the novel as a popular art form. In other words, the Web we have now is not the Web we’ll have in the future. There’s a tremendous amount of innovation to come.

The question is: “What kind of innovation can we expect?”

At Yahoo! Labs, we believe that innovation in the Web experience of tomorrow will depend directly on the work being done behind the scenes today to create new scientific theories, models and disciplines for understanding the Internet. In fact, a core element of Yahoo! Labs’ charter is to invent the new sciences that will underlie the next generation Internet, from Green Computing, Privacy and Security, to Economics and Social Systems, Advertising, Web Information Management, and Machine Learning. And nowhere is that charter more evident than in our Key Scientific Challenges Program, which focuses on supporting the bright young minds at universities across the world who are thinking, researching and creating those new sciences.

Today we’re happy to announce that we have selected 23 students from 16 world-class universities as our 2010 Key Scientific Challenges winners. They were selected based on their research proposals into the 12 different scientific challenges we believe are critical to fueling innovation on the Web. Our outstanding 2010 winners are:

Computational Advertising
Yuanhua Lv, University of Illinois, Urbana-Champaign
Nan Li, Carnegie Mellon University
Green Computing
Hrishikesh Amur, Georgia Institute of Technology
Vijay Vasudevan, Carnegie Mellon University
Machine Learning & Statistics
Arthur Asuncion, University of California, Irvine
Ryan Tibshirani, Stanford University
Sameer Singh, University of Massachusetts
Sebastien Bratieres, Cambridge University
Microeconomics and Social Systems
Nikolay Archak, New York University
Moira Burke, Carnegie Mellon University
Ankur Mani, Massachusetts Institute of Technology
Yaw Anokwa, University of Washington
Scott Kominers, Harvard University
Pranav Dandekar, Stanford University
Search Experiences
Cristian Danescu-Niculescu-Mizil, Cornell
Aditya Parameswaran, Stanford University
Security and Privacy
Xin Hu, University of Michigan
Kai Wang, University of California, San Diego
Web Information Management
Yingyi Bu, University of Washington
Oktie Hassanzadeh, University of Toronto
Emanuele Coviello, University of California, San Diego
Rares Vernica, University of California, Irvine
Michael Wick, University of Massachusetts

Each of these students submitted to Yahoo! an idea and a research proposal that our scientists and leaders saw as a genuine contribution to their field and to an area we see as critical to laying a scientific foundation for future Web experiences. Over the next several months each of them will be able to learn from one of Yahoo!’s leading scientists in their specialized field, helping them take their research to the next level and learn what it’s like to crunch data, tackle technical hurdles and design experiments at true Internet scale, an opportunity rare in purely academic environments.

The 2010 winners, in addition to receiving $5,000 in unrestricted seed funding, will convene at Yahoo! headquarters in Sunnyvale, California in September for the exclusive Key Scientific Challenges Graduate Student Summit where they will spend two days with Yahoo! Labs scientists presenting and discussing their work.

Congratulations and good luck to our Key Scientific Challenge winners for 2010.

-Jamie Lockwood, Academic Relations Yahoo! Labs

The Invisible Hand of Machine Learning

Key Scientific Challenges, Entry #6: Machine Learning

On January 27, we announced the kick-off of our 2010 Key Scientific Challenges Program.  To highlight the scientific challenge areas included in the program, we launched a series of guest blog posts on Yodel Anecdotal. Our most recent post covered Web Information Management.

Another big challenge our Yahoo! research scientists are continually examining is machine learning. In this entry, John Langford from Yahoo! Labs shares some thoughts on how Yahoo! is driving research into machine learning and why it’s a fascinating field.

When I wake up in the morning, I can’t resist checking my email and browsing the Internet to see if anything has come up. Then I get to work thinking, writing, searching, finding, and learning various things, all using an Internet that’s powered by machine learning in dozens of ways. When I go to sleep at night, I smile because I know that in addition to using machine learning throughout my day, I’ve also done my part to advance machine learning technology, many others have done likewise, and that by doing so we’re making a major impact on people’s lives.

Even though machine learning has such a broad influence on the Internet, it can be quite difficult to recognize. This is primarily because machine learning’s benefits are often hidden — they are the spam emails you don’t see, the uninteresting news articles you don’t see, and the irrelevant search results you don’t see, just to name a few. In this sense, machine learning is like an invisible hand. It’s also sometimes easier to recognize the flaws in a machine learning system – like “Why did my email end up in my friend’s spam folder?” – than it is to notice its benefits. But despite these quirks, machine learning is one of the best technologies we have for solving some of the biggest problems on the Web.

The problem of spam is representative of why machine learning is so effective. Spammers are constantly changing and adapting their strategies and technology to evade even the most capable filters. Machine learning attacks this problem by aiming to build an automatic system able to stay ahead of the game and continually refine itself in response to its environment. We haven’t completely achieved that goal yet, but progress is steady. Machine learning systems can always get better, learn more, work faster and in ever more ways, because people will always want less spam and more interesting and relevant news articles.
Naturally, this reality means we’re constantly running into both the empirical and theoretical boundaries of machine learning and statistics. How do I learn from so much data that we can’t fit it on a machine? How do we extract evidence of what the best decision was? What if the best decision changes? How do I minimize the need to know the best decision? How do I effectively use the incredibly large quantities of information available on the Internet? And how do I fit it all together in an automatic way that is useful to someone? And how do you know it’s useful?

Good answers to these questions can improve the life of just about everyone, which is the core reason why Yahoo! is sponsoring the Key Scientific Challenges Program. If you are a graduate student working on these questions, you understand how exciting and challenging this field is. And if you aren’t, consider the satisfaction associated with changing fields. :-)

John Langford

Yahoo! Labs

Surfing the Data Wave: The Problem of Managing Events and Updates for Hundreds of Millions of Users

Key Scientific Challenges, Entry #5: Web Information Management

On January 27 we announced the kick-off of our 2010 Key Scientific Challenges Program.  To highlight the scientific challenge areas included in the program, we launched a series of guest blog posts earlier this month on Yodel Anecdotal. Read our previous post on online advertising, “The Art and Science of Advertising.”

Another big challenge our Yahoo!’s research scientists are continually examining is Web Information Management. In this entry, Brian Cooper from Yahoo! Labs shares some thoughts on how Yahoo! is driving research into information management and why it’s a fascinating field.

A Torrent of Data
Sites like Facebook and Twitter demonstrate that users really like to know what’s going on with their friends. At the same time, the popularity of blog readers and personalized RSS aggregators like MyYahoo! and iGoogle show that users also like up-to-date news. Although social networking and content aggregation seem like different applications, at the core they share a key mechanism: collecting the most recent content from a set of producers, and distributing it to a set of consumers. In the case of social networking, producers write status updates or post links, and consumers (their friends) get a list of the updates. In a news aggregator, the producers are news sites or blogs, and the consumers are the readers that want the content.

The core mechanism that takes content from producers and redistributes it to users is actually quite hard to get right, and is an example of the really tough problems that fall under the Key Scientific Challenge area on Web Information Management (and its sub-topic Data Management)

The main problem is the multiplicative explosion of updates. Consider a popular user of a site like Twitter. Ashton Kutcher, for example, has over 4.5 million followers as of February 18, 2010. Every time Ashton tweets, his words of wisdom have to be propagated to the feeds of all those millions of followers. Even if the average user has far less followers, their tweets may have to be propagated to hundreds of users or more. Suddenly, even a short 140-character message can consume a lot of server resources. It’s for this reason that Twitter notoriously found it difficult to scale and stay up at the same time. Some of the challenges in doing so have been described by Twitter engineers.

At Yahoo!, we’ve been aggregating and distributing content for a long time, and we’re increasingly distributing social updates too. Products such as Yahoo! Mail and Messenger show you recent updates and activities by your friends on Yahoo!. Plus, services like Yahoo! Updates provide tools for developers to feed a variety of different user activities and updates that happen on their sites into Yahoo!’s products (and vice versa). As we build our infrastructure to support both content and user updates, we run into many of the same scaling problems that other companies have been reporting. To address these challenges, we took a step back and looked at the whole architecture needed to support these “user event feeds”.

The Push or Pull Dilemma
I’d like to talk about one aspect in particular, as it directly addresses the intersection of building Internet-scale databases and dealing with social data. When distributing updates from producers to consumers, you have a fundamental choice: should you “push” updates to consumers, so that when a user logs in their feed is already constructed and ready to go? Or should you wait until a consumer logs in, and then “pull” updates from producers to construct the feed on the fly?

Pull ensures that we construct feeds only for users that actually log in, but it has a downside: the time a user spends waiting for their feed can be quite long because the system needs to collect and sort all of the updates from all of the people or news sources that the user follows. Many sites face this tradeoff between doing potentially unneeded work to always be ready on the one hand, and the strain of delivering on demand performance on the other. Digg recently reported moving from a pull to a push model for one part of their site, resulting in a huge increase in the amount of “pushed” data that had to be stored, but a significant decrease in response time for users.

Threading the Needle
At Yahoo! Labs, when we thought about this problem and tried to figure out whether push or pull was best, we realized something that should have been obvious all along: neither push nor pull is best.

Consider a producer named Alice that posts a new status update once an hour. We need to include her updates in the feeds we produce for her friends. Thus, when her friend Bob logs in, he should see some of Alice’s recent events. But we only really need to show Bob some of his friends’ events, because there is only room on the screen to show him 10 at a time. Now imagine that Bob only logs in once a week. In that week, Alice may have produced 80 or more updates. If we pushed all 80 to Bob’s feed, at least 70 will have fallen off the front page (replaced by newer Alice events or newer events from other friends) by the time Bob logs in, and all the resources (network bandwidth, disk I/O, CPU time) spent on pushing those 70 updates would be wasted. Multiply that waste by millions of users, and you can see where this is heading.

Alternatively, consider Carl, another one of Alice’s friends. Carl loves to check his feed and updates it every five minutes (he’s an over-sharer). In this scenario, a push model makes more sense: rather than querying the database every five minutes for Alice’s updates, we should push Alice’s events to a pre-computed page for Carl, and just serve him that same page over and over until a new Alice event arrives and updates the page. That way, Carl’s experience is great for him AND no servers meltdown!

The key insight is that we should do push for some producer/consumer pairs and pull for others. Deciding whether to do push depends on the relative frequency of the producer’s events
compared to the consumer’s logins, as well as the relative cost of push versus pull. We can minimize resource usage by only pushing for producer/consumer pairs where the consumer logs in frequently enough that the cost of pushing, plus serving the pre-computed page, is less than the cost of retrieving the event repeatedly from the producer’s queue.

It’s a challenging model to develop and put into place. It can be complicated by an almost endless array of variables, and the challenge of implementing this approach in the midst of hugely popular events, like Barack Obama’s inauguration, or Michael Jackson’s death, which caused huge surges in status updates, makes it even more difficult. It’s also what makes it utterly fascinating.

More details about this challenge, the techniques, arguments and the real-world experiments we’re developing at Yahoo! Labs will be available in our upcoming SIGMOD paper, “Feeding Frenzy: Selectively Materializing Users’ Event Feeds” written by Adam Silberstein, Brian Cooper and Raghu Ramakrishnan, as well as Jeffrey Terrace from Princeton, who was an intern here at Yahoo! Labs last summer. You’ll be able to find the paper on the Yahoo! Labs site later this year and take a look.

Brian Cooper

Yahoo! Labs

This week at the University of Washington we are thrilled to be kicking off the second half of the 2009-2010 Hack U™ season. The University of Washington hack will be followed by events at Georgia Tech, UT Austin and UC San Diego throughout March and April.

During each of these hack events, Yahoo! developers teach passionate CS, HCI and engineering students about the latest and greatest open tools and technologies and web programming tips in a series of tech talks, then spend 24 hours with them to help them build their dream hack for the competition held at the end of the week. In true Yahoo! style, the all night contest is filled with fun, free food, schwag, prizes, Wii games and plenty of caffeine!

At the end of the season, the winners from each university will be flown by Yahoo! to our University Hack Showdown, which is normally held in conjunction with Yahoo!’s Open Hack Day, where they will hack it out for street cred, cool prizes and the opportunity to get their hack noticed by web industry judges, including VCs, web entrepreneurs, Yahoo! execs and technology gurus.

“The close of 2009 marked the third year of the Hack U™ program and it’s been amazing to watch the innovation, spirit and quality of the hacks continue to grow, said Jamie Lockwood, program manager for Hack U™. “The students used to just mash up as many open technologies as they could barely thinking about UI or what problem the hack solves, but today we see ideas and prototypes coming together that are more and more sophisticated and in some cases almost ready to go to market.”

The winning Hack U™ students from last season had no problem holding their own against the experienced developers that showed up at the Yahoo! Open Hack event in NYC, winning half of the categories including best overall hack.

The program has definitely caught the attention of Yahoo!’s internal open tech leaders, as well as the external web community. The Hack U™ winners from Toronto even snagged an interview at Y Combinator with their hack idea.

You can read more about the program and view some of the other hacks that have been developed at universities across the country and the world (we had our first India Hack U event at IIT earlier this year!) on the Yahoo! Developer Network site.

Hack U events have been held at some of the best engineering universities around the country including UC Berkeley, Stanford, Georgia Tech, Carnegie Mellon, University of Washington and University of Illinois and we’re always on the lookout for other universities that want to be part of the growing Hack U community We also want to keep the technology topics fresh, timely and up to speed with the latest web industry innovations so feel free to send us requests for specific speakers or ideas for suggested topics.

Check out the website for more information, updates and details about the upcoming 2010 schedule.

– Yahoo! Academic Relations and Hack U™ Team

Measuring the Unmeasurable

Key Scientific Challenges, Entry #3: Microeconomics and Social Systems

On January 27 we announced the kick-off of our 2010 Key Scientific Challenges Program.  To highlight the scientific challenge areas included in the program, we launched a series of guest blog posts earlier this month on Yodel Anecdotal. Read our previous post on privacy and security, “Data, Data Everywhere, but How to Keep it Safe.”

Another big challenge are Yahoo!’s research scientists are continually examining is microeconomics and social systems. In this entry, Sharad Goel from Yahoo! Labs shares some thoughts on how Yahoo! is tackling the new opportunities for research into the social sciences that the Web is making possible and why it’s a fascinating field.

What do your friends really know about you? How much do they influence your decisions? How often do we stray from the cultural herd? How do groups organize to solve complex problems?

Answers to such fundamental questions about social behavior have often eluded us. With microscopes we peered into the intangibly small building blocks of life, and with telescopes we found our place in an unimaginably expansive universe. But without the tools to faithfully document human activity—a challenge that by comparison seems so palpable—we had no way to investigate the inner workings of our own communities. Now with an explosion of information on every aspect of our everyday existence—from what we buy, to where we travel, to whom we know—we can measure what until quite recently was thought unmeasurable. In the Microeconomics and Social Systems Group at Yahoo! Labs, we are using this proliferation of data to explore how societies function. It’s a fascinating area of study that is just beginning to shed light on new layers of human behavior, making it a perfect fit for the Key Scientific Challenges Program.

In a recent study that’s garnered some attention, for example, we asked, “How eccentric are people?” Looking at consumer preferences across movies, music, and web browsing, we came to the surprising conclusion that ordinary people have pretty extraordinary tastes. In particular, we found that typical Netflix and Yahoo! Music users regularly watch movies and listen to songs that are not even available in the largest brick-and-mortar retailers. This result not only challenges stereotypes about people blindly following the herd, but also highlights the importance of offering consumers broad selection. That is, specialty products may dramatically boost user satisfaction by providing buyers the convenience of “one-stop shopping” for both their mainstream and niche interests.

In other work, we used web search queries to forecast the commercial success of movies, songs, and video games. Weeks, sometime even months, before a movie opens or a video game is released, one can find traces of pent up consumer demand in the search query logs. We found that these telltale signs of early interest are remarkably good predictors of future success. The catch? Although the search logs do reflect user intent, more mundane indicators, such as production budgets and reviewer ratings, perform equally well at forecasting sales. Thus, the benefit of web search as a prediction tool may have less to do with its superiority over other methods than with its generality, low cost, and real-time nature.

At a time when we are drowning in data, at Yahoo! Labs we’re asking a simple question: what can you do with it? The answer is limited only by our imaginations.

Data, Data Everywhere, but How to Keep it Safe:
Key Scientific Challenges, Entry #2: Privacy & Security

On January 27 we announced the kick-off of our 2010 Key Scientific Challenges Program. Earlier this week we launched what we hope will be a thought-provoking series of guest blog posts here on Yodel Anecdotal that offer a quick overview of these scientific challenge areas. Check out our first post on green computing.

Today’s entry is another incredibly important issue and difficult challenge posed by the Web – privacy and security. We’ve recruited Ashwin Machanavajjhala from Yahoo! Labs to share his point of view on why privacy and security are Key Scientific Challenges.

As a popular destination on the Web, millions of users interact with Yahoo! every day. They search for information online, read the news, click on ads, upload, tag and share photos, and so on. Each of these billions of individual actions sheds some light on a particular user’s interests – what they like, what they want to know more about, buy or do in their spare time. They are all valuable cues that can be used to customize someone’s Web experience in a way that’s personally relevant to them.

Ultimately, a personally relevant experience is at the root of what we’re trying to deliver with all of Yahoo!’s products. The information that comes our way is a prime asset for making our users happy. But at the same time, with all that information comes a huge responsibility. We need to take extreme care to make sure people’s privacy is not breached. This responsibility to earn and keep our users’ trust is not just a matter of Yahoo! policy (although that is critical too), it’s also a technical challenge that requires scientific innovation to continuously improve and maintain.

When it comes to the trust of our users, we’ve learned lessons on how to approach both the policy and technical elements of privacy and security.  On the policy side, we are extremely proud of our Data Anonymization Policy, which has received wide support and affirms our commitment to help protect our users’ privacy. Yahoo!’s policy both dramatically reduces the time we hold personal data and increases the scope of log data covered under the policy. Under the policy, we anonymize user log data, including deletion of total IP address, after 90 days with limited exceptions to fight fraud, secure systems and meet legal obligations.  We’re also expanding our commitment to include data on page views, page clicks, ad views, and ad clicks as part of this policy.

On the technical side, we’ve invested in giving our users the ability to understand and shape how we interpret what’s personally relevant to them by launching our Ad Interest Manager, a central place where Yahoo! visitors can see a concise summary of their online activity and make easy, constructive choices about their exposure to interest-based advertising served by the Yahoo! Ad Network.

And our scientists at Yahoo! Labs have also been active in academic privacy research, examining new mathematical definitions of user privacy and developing novel technologies for sharing and utilizing user information to improve Web experiences in a privacy-preserving manner. These experiments and research have been published widely, opening up new avenues of investigation on issues that are critical to Yahoo!, like protecting search log data and dealing with so-called vanity queries, to new privacy frameworks that can be used in any field where making data available broadly is a key objective in overall innovation, as it is with the World Bank, Census Bureau and medical institutions publishing important public health studies, for example.

These efforts are just the beginning, though, which is why privacy and security is a Key Scientific Challenge area, and we can’t wait to see what the word’s aspiring minds have to say on the topic.

Ashwin Machanavajjhala
Research Scientist
Yahoo! Labs

On Wednesday we announced the kick off our 2010 Key Scientific Challenges Program. It is a thought-provoking competition that encourages top graduate students to help invent the future of the Internet by working with Yahoo! Labs to investigate and test their ideas in the real world.

The Key Scientific Challenges Program focuses on a variety of scientific issues that we believe are central to developing a better understanding of the Web and the foundational technologies that will accelerate innovation. Over the next couple of months, we’ll be offering a series of guest blog posts here on Yodel Anecdotal that offer a quick overview of these scientific challenge areas from a Yahoo! (or Yahoo!s) who have expertise in the field and can explain how their research can have an impact on making the Web more engaging, relevant and powerful.

Today’s our maiden voyage, and the first guest blog post comes from Scott Noteboom. Scott is one of the foremost experts on building data centers that makes it possible for Yahoo! to run some of the fastest, most popular Web sites in the world.  He’s also leading the charge to make Yahoo!’s data centers as efficient as possible, and we’ve asked him to talk about a new challenge area we just added to the Key Scientific Challenges Program this year, green computing.

Thanks,
The Key Scientific Challenges Team at Yahoo! Labs

Key Scientific Challenges, Entry #1: Green Computing
By Scott Noteboom

Increasing demand for the Web services and applications that have become such a central part of our lives (like email and Web video, just to name a few) is also causing a steady rise in the need for more and more computing power to make all of theses services work and work well.  When we talk about computing power, we’re often talking specifically about data centers, buildings that house thousands and thousands of servers working 24-hours-a-day to make sure that the Web sites you want are there on your screen and on your phone when you want them.

Of course, while the performance of servers is improving almost everyday, making all kinds of new innovations possible, they also require a lot of power. They need so much power that a 2007 joint report by the Environmental Protection Agency and the Lawrence Berkeley National Laboratory revealed that energy consumption from data centers had doubled between 2000 and 2006 and was expected to double again by 2011. The report further calculated that there would be huge energy and money losses without increases in the efficiency of running and cooling these centers (because servers produce a lot of heat).

Recognizing both the problem and the opportunity that this presents, Yahoo! has embraced its role as a leading environmentally sustainable company by addressing efficiency in our data centers head on.  We announced in June 2009 that we’d be committing to a significant reduction in the carbon intensity of our data centers by 2014. And to accomplish this goal, we’ve developed several innovations, including a new high-efficiency data center design that reduces our cooling load and increases utilization in our data centers, increasing the productivity of our servers, and squeezing as much productivity as possible out of every last kilowatt-hour. These efforts have gotten us some recognition and hopefully will inspire more innovation because making the world’s data centers more efficient is also an important element to tackling global warming.

Everyone wants to see innovation on the Web and what it makes possible for us in our everyday lives continue to flourish, and that’s where innovation behind the scenes in data centers is so important. By making data centers more efficient, we can make sure that future innovations that make the Internet more and more powerful for all of us will also be environmentally sustainable.

This is by no means a pipe dream, which is why we’re so excited to be part of the Key Scientific Challenges team recruiting some of the best young minds in the world to learn from our experiences and accelerate their own research and ideas. A study by The Climate Group entitled Smart 2020 predicts that, while greenhouse gas emissions from the Internet industry will rise to approximately 1.3 gigatons of CO2, the combined impact of smart grid, smart logistics, smart buildings and videoconferencing could reduce emissions by approximately 7.8 tons.

We’re very excited about the next big ideas that will continue to bring radical improvements in energy efficiency and truly green computing. If you’re interested in learning more about the Key Scientific Challenges Program visit the site, and if you’ve got an idea for research into green computing, get working on a proposal. Submissions are due March 5th.

Over the years, Yahoo! Labs has partnered closely with many universities and colleges. As a company, we want to invent the sciences needed for the next-generation Internet. Working with the faculty, researchers, and students of the world’s great universities is one of the best ways to meet that goal.

Looking back, 2009 was our most successful year to date. Through a variety of programs, we brought together a lot of smart folks and important ideas. As we head into 2010, we want to share our excitement about these programs and events.

Hack U:
Our University Hack Day competitions (Hack U) brought Yahoo!’s open technology, top developers, and hack spirit to campuses for technical talks and a 24-hour student programming competition. Students from 11 universities in the U.S., Canada, and India built more than 300 hacks. Winners from each campus participated in the New York Open Hack Day, ultimately winning 6 of the 11 categories.

Yahoo! Big Thinkers Series:
As part of our Big Thinkers distinguished speaker series, Yahoo! brought the brightest minds in the academic community to our campus for talks, and we’ve made those lectures available on the Web for all to see and enjoy. We had talks on
 

• Global climate change with Kenneth Arrow, Nobel Laureate and Professor Emeritus of Economics at Stanford University
• The psychology of the Web with Sandy Pentland, MIT
• Getting around the “you don’t know what you don’t know” syndrome with Ion Stoica, University of California, Berkeley
• How computational thinking can transform education with Jeannette Wing of the National Science Foundation
• Calculating the largest prime number ever found with Ron Graham, University of California, San Diego
• How the Internet and the science of computing are on the verge of driving a revolution in medical and biological science with Leroy Hood, Institute of Systems Biology

 
Next year’s Big Thinkers series should be just as captivating — we’ve got a great lineup for 2010.

Webscope:
Our Webscope™ program offers a reference library of 24 interesting and scientifically useful datasets we’ve made available for noncommercial use by academics and other scientists. More than 880 academic researchers have used the datasets, resulting in 28 technical papers, journal articles and theses so far. Very few companies have the resources and global scale to help academics and students interact with the types of real-world datasets it takes to spark innovation, but Yahoo! is one of them and it’s really paid off.

Key Scientific Challenges (KSC):
Our Key Scientific Challenges program partnered with 21 graduate students in 2009. We gave them scholarships, plus the opportunity to work closely with Yahoo!’s scientists on solving some of the biggest challenges the Web offers. The winners attended the KSC Graduate Student Summit, where they presented the fruits of their labor to fellow students and other Yahoo! researchers. It was the first of what will surely be many peer-reviewed conferences for these talented folks.

Cloud Computing Research:
Our M45 Cloud Computing Cluster is a 4,000-processor testbed being used in academia for the advancement of cloud computing research and education. Faculty members at Carnegie Mellon University have written 40 technical publications based on research performed on M45, and the cluster is now being used by three additional universities. Yahoo! is also part of both the Open Cirrus testbed and the Open Cloud Consortium.

Global Impact – Yahoo! Days in Haifa:
In November we held two very successful “Yahoo! Days” at major Israeli universities – Tel Aviv University and Technion, the Israel Institute of Technology. Yahoo! scientists from around the world joined the event, which featured a keynote by Prabhakar Raghavan, the head of Yahoo! Labs. Pictures from the Tel Aviv and Technion events are available on Flickr.

Netflix Prize-winner Yehuda Koren’s Campus Tour:
Yehuda Koren, a research scientist with Yahoo! Labs in Israel, was part of a team that spanned countries, time zones, and companies, collaborating over a three-year period to win the Netflix Prize, one of the most well-publicized and interesting machine learning contests ever conceived. It’s a story with surprises, twists and turns, game-playing, late nights, and computational brute force. There’s also deep science behind it all — science that will drive future innovation on the Web. Naturally, Yahoo! thought it was the kind of story that students and faculty at some of the world’s best universities would like to hear in person. Yehuda has visited seven universities since the award was announced in September 2009, including MIT, Harvard, Carnegie Mellon, Columbia, Rutgers, Princeton, Penn, and GA Tech. And he’ll be back at in 2010.

Purple Footprints:
And, of course, we continued to sponsor our campus seminar series, leaving “purple footprints” at Carnegie Mellon University, MIT, and University of Illinois at Urbana-Champaign. All of those seminars have been recorded and are available online. Yahoos also presented more than 100 lectures, seminars, workshops and training sessions at campuses worldwide.

If you want to learn more about Yahoo! Academic Relations programs and how you can participate in 2010, please visit us.

Ron Brachman
Vice President
Yahoo! Labs