Posted March 18th, 2010 at 3:30 pm by Lucas Mast, Blog Editor
No Comments » / Filed in: Uncategorized
Photo: “Scott Beale/Laughing Squid”
Yahoo! Jellytalks is a monthly speaker series aimed at developers, designers and entrepreneurs. Each month a relevant speaker attends a Jelly (casual co-working space for collaborative work) and they are streamed live. Tune in online: http://jellytalks.yahoo.com/
Here’s how it works
- We’ll invite an interesting speaker to a local Jelly, have ‘em talk for 30-45 minutes.
- We’ll stream the talk live so Jellies around the world can tune in.
- After the talk, anyone watching can ask the speaker questions via live chat.
Through a Meebo chat in page and a Twitter feed the global audience can ask questions and receive answers by using the Twitter hash tag i#jellytalks
Gary Vaynerchuk will talk about personal branding, consumer branding and business equity. He will touch on how he used social tools to engage his audience and how you can do the same from yourself or your business.
March 19th, 2010 – 2:30pm EST
More on the speaker:
Gary Vaynerchuk is a New York Times bestselling author and American businessman who was born in Belarus and immigrated to the United States as a young child. Gary’s entrepreneurial instincts took over at a young age, when he owned a franchise of neighborhood lemonade stands and made $1,000 a weekend selling baseball cards. Much to his dismay, his father Sasha pulled Gary into the family business, a local liquor store called Shopper’s Discount Liquors. Before long, Gary recognized that consumers collected rare wines just like people collected baseball cards, and he was off to the races. Gary transformed himself into a wine expert, rebranded the store as Wine Library, launched a retail website in 1997, and by 2008 he had raised annual revenue from $4 million to $60 million. In 2006 Gary achieved one of his life-long goals when he was caricatured on the front page of the Wall Street Journal in an article about online wine sales.
Tagged: Gary Vaynerchuk, JellyTalks, New York Times, Wine, WSJ
Post a Comment
Posted March 17th, 2010 at 10:30 am by Lucas Mast, Blog Editor
No Comments » / Filed in: Uncategorized
In the past month, Yahoo! Sports has proven that it is the destination that sports fans turn to when news happens. Not only did the site surpass all of its competitors to become the #1 online destination for the Winter Olympics, it also set a traffic record, generating more than 39 million unique monthly users in February (comScore, February 2010).
And now, Yahoo! is announcing that it has entered an agreement to acquire Citizen Sports, a leading social media company. Home to the #1 fantasy football app on Facebook, Citizen Sports engages and unites fans across the globe around their favorite sports and teams.
As Yahoo! pursues our social strategy of enriching, aggregating and distributing social content from across the Web, Citizen Sports will strengthen how we go about doing this in the Sports category and beyond. The combination of these two powerhouses will make the Yahoo! experience more personally relevant, create new opportunities for users and advertisers across Yahoo!, mobile and social networks and thrill sports fans everywhere.
Yahoo! Sports’ best in class news and information will be integrated into Citizen Sports’ range of products and applications for football, hockey, soccer, baseball, racing, rugby, hockey and cricket. In addition, Yahoo! Sports users will be able to broadcast their allegiances, create or join a conversation with friends and fans and cheer for their teams through Citizen Sports’ applications.
So…what are you waiting for? The race is on. Go get your March Madness on with the Bracket Challenge by Citizen Sports app and Yahoo!’s Tourney Pick ‘Em!
– Jeff Kinder, SVP, Media Products & Solutions
Tagged: Bracket Challenge, Citizen Sports, Tourney Pick'em, winter olympics, yahoo! sports
Post a Comment
Posted March 12th, 2010 at 2:36 pm by Lucas Mast, Blog Editor
3 Comments » / Filed in: Uncategorized
We invite you to join a new experiment we cooked up at Yahoo! Labs called Predictalot, a game that takes NCAA tournament pick ‘em to entirely new extremes.
Here’s how it works. You can invest virtual points on almost any prediction you can think of about the men’s college basketball tournament, like “Duke will advance further than Texas,” or “Every final four team’s name will start with the letter U.” For example, you can invest one point predicting that “there will be exactly 2 upsets in the Elite Eight” and, according to the current Predictalot odds, you’ll win ten points if you’re right.
You can keep your predictions and see how they turn out, or you can sell them at any time, even while you’re following the action on the court. How does that work? Think of your predictions like stocks that can be priced, sold and traded (for virtual points, of course) just like you’d buy and sell stocks, options and futures on the financial exchanges of the world. Predictalot offers hundreds of millions of such predictions, constantly computing prices for them all.
Throughout the tournament, the prices of all the predictions in the system will go up and down based on what’s actually happening in the tournament and on what everyone else thinks those predictions are worth. Based on the value of the predictions you invest in you’ll accumulate points and climb your way to the top of the Predictalot leader board.
In technical terms, Predictalot is what is called a combinatorial prediction market. Some of us at Yahoo! Labs, along with academics like George Mason University professor Robin Hanson, have been studying and writing about them for quite some time, and dreaming about trying them out in the real world. We built the first version of Predictalot during an internal Yahoo! Hack Day. And this year we used the Yahoo! Application Platform to finally build a public version of the game. Like any Yahoo! App, you can install Predictalot to play right on the Yahoo! homepage. (Note that we’re not special in that regard: anyone can develop a Yahoo! App that’s available to millions of Yahoo! users — there’s good sample code, it supports YUI and OpenSocial, and it’s easy to get started. Seriously, try it.)
After some fits and starts, a few late nights, and eventually a few all nighters, we’re proud and excited to go live with Predictalot, version 1.0. I can’t rave enough about the talent and dedication of the research engineers who gave the game a professional look and feel and some real production speed. They turned a very pie-in-the-sky idea into a reality. We have a lot feature ideas in mind for future versions, but the core is in place and we hope you enjoy the game.
With 9.2 quintillion outcomes, or nearly as many as the number of individual insects on Earth, Predictalot is to our knowledge the largest prediction market ever built. We hope that with your participation it will prove a great test to the limits of what the wisdom of crowds can produce. We’d also like to pave the way for the serious use of combinatorial prediction market technology in a host of other arenas down the road.
Smarter markets will be extremely valuable in a variety of applications, especially online, where markets and exchanges decide prices and allocations for all kinds of goods and services, including advertising inventory. And more and more, the matching of buyers with sellers – which is the core function of any market – is requiring sophisticated algorithms, machine learning and optimization technology. Predictalot attempts to help us better understand these dynamics while also giving us the great satisfaction of picking Duke to go all the way.
(Duke alum) David Pennock and the Predictalot Team:
Mani Abrol, Janet George, Tom Gulik, Mridul Muralidharan, Sudar Muthu, Navneet Nair, Abe Othman, Daniel Reeves, Pras Sarkar
P.S. If you’d like to learn more about the technology hidden in the engine and the scientific factors at play, read our post about Predictalot on the Yahoo! Research website. And don’t forget to also check out all of Yahoo!’s fun tourney experiences.
Tagged: hack day, March Madness, NCAA, yahoo! labs, YUI
Post a Comment
Posted March 11th, 2010 at 4:39 pm by Lucas Mast, Blog Editor
No Comments » / Filed in: Uncategorized
The Invisible Hand of Machine Learning
Key Scientific Challenges, Entry #6: Machine Learning
On January 27, we announced the kick-off of our 2010 Key Scientific Challenges Program. To highlight the scientific challenge areas included in the program, we launched a series of guest blog posts on Yodel Anecdotal. Our most recent post covered Web Information Management.
Another big challenge our Yahoo! research scientists are continually examining is machine learning. In this entry, John Langford from Yahoo! Labs shares some thoughts on how Yahoo! is driving research into machine learning and why it’s a fascinating field.
When I wake up in the morning, I can’t resist checking my email and browsing the Internet to see if anything has come up. Then I get to work thinking, writing, searching, finding, and learning various things, all using an Internet that’s powered by machine learning in dozens of ways. When I go to sleep at night, I smile because I know that in addition to using machine learning throughout my day, I’ve also done my part to advance machine learning technology, many others have done likewise, and that by doing so we’re making a major impact on people’s lives.
Even though machine learning has such a broad influence on the Internet, it can be quite difficult to recognize. This is primarily because machine learning’s benefits are often hidden — they are the spam emails you don’t see, the uninteresting news articles you don’t see, and the irrelevant search results you don’t see, just to name a few. In this sense, machine learning is like an invisible hand. It’s also sometimes easier to recognize the flaws in a machine learning system – like “Why did my email end up in my friend’s spam folder?” – than it is to notice its benefits. But despite these quirks, machine learning is one of the best technologies we have for solving some of the biggest problems on the Web.
The problem of spam is representative of why machine learning is so effective. Spammers are constantly changing and adapting their strategies and technology to evade even the most capable filters. Machine learning attacks this problem by aiming to build an automatic system able to stay ahead of the game and continually refine itself in response to its environment. We haven’t completely achieved that goal yet, but progress is steady. Machine learning systems can always get better, learn more, work faster and in ever more ways, because people will always want less spam and more interesting and relevant news articles.
Naturally, this reality means we’re constantly running into both the empirical and theoretical boundaries of machine learning and statistics. How do I learn from so much data that we can’t fit it on a machine? How do we extract evidence of what the best decision was? What if the best decision changes? How do I minimize the need to know the best decision? How do I effectively use the incredibly large quantities of information available on the Internet? And how do I fit it all together in an automatic way that is useful to someone? And how do you know it’s useful?
Good answers to these questions can improve the life of just about everyone, which is the core reason why Yahoo! is sponsoring the Key Scientific Challenges Program. If you are a graduate student working on these questions, you understand how exciting and challenging this field is. And if you aren’t, consider the satisfaction associated with changing fields. :-)
John Langford
Yahoo! Labs
Tagged: Key Scientific Challenges, Machine Learning, yahoo! labs
Post a Comment
Posted March 10th, 2010 at 10:03 am by Lucas Mast, Blog Editor
No Comments » / Filed in: Sports, Uncategorized
![tourney10_logo[1]](http://farm5.static.flickr.com/4035/4421378648_85885a5d57.jpg)
March is here, which means college basketball tournament time, one of my favorite times of the year, when even the mildest of fans start keeping up with every single game. When the Virginia Techs and the Daytons are crossing their fingers every single week that they stay inside that coveted bubble. When Facebook and Twitter status updates explode every time a top-10 team gets stunned or a star player makes an amazing dunk.
Facebook and Twitter—those sure weren’t around when I was playing in the Championship game 20 years ago against Duke. I’ll never forget the thrill of that victory, all the memories from the entire season, playing alongside Stacey Augmon and Larry Johnson—broken jaw and all.
I’m not on the court anymore, but that’s one of the beautiful things about the NCAA Championship Tournament: With every correct pick, every upset we call, every favorite team that advances, we (the fans) get to share in the thrill. One million dollars certainly doesn’t hurt, either—which is what Yahoo! is offering one lucky fan for submitting the perfect bracket. No perfect bracket in the end? No problem. The best bracket in the KFC- sponsored Yahoo! Sports Tourney Pick ‘em contest wins $10,000 (just ask Michael Lemon—he’s been enjoying his winnings from last year), and there’s even a second-chance contest from TIAA-CREF for the majority of us that must reluctantly accept our busted brackets.
Starting today, on-the-go fans will have access to a great new mobile experience for staying on top of their bracket selections. Yahoo! has created a mobile site where fans can play the Yahoo! Sports Tourney Pick’em game right on their mobile device at m.yahoo.com/tourneypickem. In addition, they’ll be able to get live scoring, game stats, updated schedules, news, and expert analysis from the Yahoo! Sports editorial team.
And it’s even easier these days to get your friends involved (and we all know the real satisfaction comes in beating that one friend who is so sure he’s going win) with Facebook and Twitter integration. Plus, you can be a die-hard fan without having to invest all the time and research. Yahoo!’s Bracket Wizard and Scenario Generator help fans with those quick decisions and allow them to see at a glance which teams to cheer for along the way to ensure that big W.
One of the perks of being a member of the Yahoo! Sports team is that I get to pitch in and offer my best (and presumably spot-on) advice to the masses. For those last-minute updates, fans can tune in to Yahoo! Sports Fantasy Bracket Live, a live and interactive web-show from the folks who brought you the Emmy®–nominated Yahoo! Fantasy Football Live. Tune in on March 17 as Jason King, Brad Evans, Larry Beil and I give you a final rundown on all the immediate updates affecting your teams. Viewers can even send us their bracket, and there’s a chance we’ll help you out with it on our show.
I know you guys are as excited about the next few weeks as I am. If you haven’t already, you better get in the game soon. Tip-off is only 8 days away, and it can’t get here soon enough!
-Greg Anthony, Yahoo! Sports College Basketball Analyst
Tagged: College Basketball, facebook, greg anthony, March Madness, NCAA, Pick'em, twitter, unlv, yahoo! mobile
Post a Comment
Posted March 8th, 2010 at 10:17 pm by Priscilla Tan,
2 Comments » / Filed in: Guest Opinions, Profiles, Working at Yahoo!, Yahoo! Opinions, Yahoo! in Asia
PROFILE SERIES:
Starting this week, we’ll be bringing you a step closer to the Yahoos who make stuff happen behind the scenes as well as guests from outside of the company to share their views on various hot topics.
Joey Alarilla, a prominent (some say famous) Filipino blogger and journalist who recently joined Yahoo! as our social media editor for Southeast Asia is our first guest. Read on to find out more about his new role and how social media is playing a critical part in the Yahoo! Southeast Asia newsroom as well as the upcoming Philippine Presidential election campaigns.
Welcome to Yahoo! Can you give us a brief introduction to yourself, for example, what were you doing before joining the company?
Prior to joining Yahoo!, I was the head of the Multimedia Department of the leading online game publisher in the Philippines, Level Up! My projects there included launching our live blogging and live video streaming services and producing/hosting a weekly online show for our community.
I was a tech journalist for over a decade. In 2000, I was one of the pioneer editors who spun off the leading Philippine online news site INQUIRER.net (then called INQ7.net) from the Philippine Daily Inquirer newspaper. As the multimedia editor, I launched the site’s blog network and online video service, while also hosting and producing several podcasts and editing the hackenslash gaming news site. I was also the founding president of the Asian Gaming Journalists Association and a tech blogger for CNET Asia.
Could you tell us what a social media editor at Yahoo! in the Philippines does?
As the social media editor, I listen and interact with the community, not just on Yahoo! properties, but also on other social networks. My job includes understanding and monitoring what’s trending on different social networks; engaging the communities on these networks to exchange ideas and get feedback; and sharing these insights with the editorial team and our content partners.
We are also looking for more user generated content and offering platforms for our users to have their stories covered by Yahoo! and their views heard and read by more audience.
Sounds exciting! Tell us more about the social media scene in the Philippines / Southeast Asia
The social media scene is quite vibrant not just in the Philippines but across the region. It’s becoming more and more mainstream, with celebrities, journalists, politicians, and other personalities becoming more active on social networks.
One of the factors contributing to the growth of social media is the increasing popularity of the mobile Internet. You could say that social media is starting to become the new SMS. When you see people typing on their phones here, they might not be texting, but actually posting status updates on their social networks, or chatting with other online users.
While Twitter and Facebook are gaining popularity in this region, Meme from Yahoo! has a huge fan base and growing very quickly. We have introduced themed Memes for comic book lovers and cosplay and anime fans – two popular topics in the Philippines.
Can you elaborate more on why you think social media has started to become a vital part for news gathering?
News organizations have to evolve to keep pace with the changing habits of their audience. Many Internet users, especially the younger ones, have embraced social media.
Their primary source of news is no longer newspapers, television or even online news. It’s trusted users on their social networks. That explains why you have a personal newspaper service like The Twitter Tim.es, which displays news and blog links from the people you trust on Twitter – you can even check out my personal newspaper.
Increasingly, social media users are “out-scooping” the breaking news teams from mainstream media. Social media is one of the key channels for delivering news and other information. In the Philippines, in the wake of the massive flooding caused by tropical storm Ondoy (international codename: Ketsana), Filipinos relied on social networks to keep themselves informed and coordinate relief and rescue operations. Through social media, different rescue groups were able to harness the spirit of volunteerism and encourage more people to help out the flood victims.
The fact is that news organizations that fail to adapt are in danger of becoming irrelevant. They can no longer assume that users will come to them, but must instead make sure their content is available wherever their users may be.
Due to different factors, some news organizations have been slow to integrate social media into the newsroom. I’m happy to say, however, that the situation is changing. Many news organizations in the Philippines and in Southeast Asia are embracing social media. This is a new and exciting frontier for many journalists, and Yahoo! will be here to help our media partners with our own insights based on our experiences and best practices.
During the recent social media forum in the Philippines, you discussed how social media is shaping the political landscape in the Philippines; do you think the candidates really understand the benefits/ use of social media?
Social media is generating a lot of hype in the upcoming Philippine presidential elections this May. Pundits are saying this will be the first Philippine elections in which social media will potentially be a game changer.
It’s good to see that many candidates recognize the importance of social media in their campaigns – no doubt inspired by the example of US President Barack Obama. What they have to keep in mind, however, is that social media is just one aspect of the campaign, and that their success will depend on how well they integrate it with their overall strategy.
The Obama team was able to reach out via social media but the crucial step was in translating this into actual grassroots support i.e, volunteers going door to door, and voters trooping out on Election Day. Social media is not a silver bullet. It’s not the be-all and end-all of a campaign, and if candidates become shortsighted, they might end up fighting an online popularity contest instead of inspiring action among their constituents.
Moreover, I would like to see candidates focus less on bombarding followers with their messages, and instead concentrate on interacting with voters online and listening to what they have to say. Social media is a conversation, and it’s a chance for voters to truly make their voices heard.
Empowering the voters through social media is the main goal of Yahoo!’s Purple Thumb site for the Philippine elections. We also held a social media forum, to examine the impact of social media on the upcoming elections. It’s all about the community – about putting the spotlight on what ordinary people have to say about the election issues that concern them most.
Social media is your chance to be heard. Don’t let anybody tell you otherwise.
Tagged: elections, philippines, purplethumb, social media, trends, Working at Yahoo!
Post a Comment
Posted March 8th, 2010 at 9:07 am by Lucas Mast, Blog Editor
No Comments » / Filed in: Cool Stuff, Product Pulse, Sports
As Spring Training heats up, we are very excited to announce an exclusive deal with Major League Baseball making Yahoo! Sports Fantasy Baseball ‘10 the official fantasy baseball game of MLB.com.
The co-branded Yahoo! Sports Fantasy Baseball ‘10, will provide fans with the most relevant and engaging fantasy baseball experience on the Web. The combination of Yahoo!’s unmatched sports news and MLB.com’s in-game and post-game video highlights truly makes this the biggest and best game for passionate fantasy baseball fans.
Yahoo! Sports Fantasy Baseball ’10 players can purchase MLB.com’s new Fantasy Highlights package, giving them the opportunity to watch customized in-game and post-game video highlights of their fantasy players directly from their homepage. The unique feature, combined with Yahoo! Sports’ leading online news and insights, will provide players with the most comprehensive and interactive fantasy baseball experience anywhere, taking fans beyond the box score to watch the performance of their fantasy roster as it unfolds on the field.
Additionally, Yahoo! Sports Fantasy Baseball ‘10 will offer a new live online auction draft application and free live scoring, which gives fans the ability to view live stats and instant scoring for teams or matchups and get real-time league standings straight to their homepage.
Yahoo! is the partner of choice for content companies, working closely with major sports, entertainment, and media companies to deliver the world’s most relevant and engaging content to its consumers. The MLB is the third official league relationship we’ve entered into, with the NHL and PGA Tour fantasy games already leading in their respective online categories. And now, with the combination of Yahoo!’s unmatched sports news, MLB.com’s wealth of statistical research tools and never-before-available live video highlights at users’ fingertips, we know Yahoo! Fantasy Baseball ’10 will quickly become the hottest product serving the incredibly passionate fantasy baseball audience. Check it out for yourself!
Tagged: baseball, MLB, Yahoo! fantasy baseball, yahoo! sports
Post a Comment
Posted March 5th, 2010 at 2:31 pm by Lucas Mast, Blog Editor
2 Comments » / Filed in: Uncategorized
“Hollywood’s Biggest Night” is just around the corner and Yahoo! is ready to take you down the red carpet and get you through the evening with up-to-the-minute information and all the news, photos, and juicy gossip you can handle.The 82nd Academy Awards are at 8 ET / 5 PT Sunday, March 7, 2010 at the Kodak Theater in Hollywood and will be hosted by Steve Martin and Alec Baldwin. While the show will be broadcast on ABC, Yahoo! once again is THE place where you can weigh in on all things Oscar, from the fashion faux pas to the surprise winners and the heartfelt speeches.
We’ve dedicated an entire section of Yahoo! Movies to this year’s awards show with the idea that Yahoo!’s Coverage of the Oscars starts with you! Let your voice be heard and make sure to cast a vote in our poll: “What do you want to see most when you watch the Oscars?” As of right now, 43% of you want to see who wins the big awards. You can also weigh in on who you think deserves to win Best Picture, Best Actor, Best Actress, Best Supporting Actor, Best Supporting Actress, and Best Director.
Interested in seeing what the stars are wearing? We’ve got you covered with our Oscars Photo galleries including fashion photos from the red carpet, insider photos from the before and after parties, photos of the nominees, and photos from the many events taking place Sunday.
We also have a Featured Videos section, a News section, and an ongoing Oscars Blog.
Coverage of the 82nd Annual Academy Awards is already underway with both OMG! and Yahoo! News feeding into the Oscars frenzy.
Want to be the best informed on this year’s likely winners and losers? We’ve broken down the Academy Awards by category to help you get a leg up on your friends in picking who will win Oscar gold, in How To Win Your Oscars Pool.
Find out why Sacha Baron Cohen (aka Ali G/Borat/Brüno) went from being a presenter to sitting the award show out and watching from home!
Want an expert’s opinion? Who’s more suited to give Oscar picks than OUR favorite Oscar–namely Oscar the Grouch? Sesame Street’s seminal crab-apple took a few minutes to answer some questions on who he thinks the winners will be on Sunday. Check out Oscar the Grouch’s Oscar Picks.
Criticisms of this year’s hottest movies and highest contenders are outlined in Yahoo! News’ Controversies big and small dog Oscars contenders. Find out why there’s been outrage over Sigourney Weaver’s role in Avatar.
Red-carpet style predictions and fashion alerts are revealed in OMG!’s Glam Slam: Oscar Style!
Have your own opinion on the big winners and major upsets? Want to weigh in on some of the fashion wins and fails? Then we invite you to join the conversation on Facebook and Twitter.
On Facebook, be sure to check out the Yahoo! Movies fan page (Yahoo! Movies) where you can write captions to photos of your favorite celebs. We’ll be uploading fashion photos of the movie stars from the red carpet, and we’re asking you to submit your most witty captions via comment.
On Twitter, follow @YahooMovies for real-time info on Academy winners. Follow @YahooOMG for updates on fashion from the red carpet. Follow @YodelingMamas for a Mom-blogger’s point of view on the evening. And of course follow @Yahoo for an overall perspective.
See you on the red carpet–and may the best man, woman and picture win!
Tagged: Academy Awards, Alec Baldwin, Avatar, blind side, bridges, bullock, Celebs, clooney, facebook, hurt locker, inglorious basterds, Oscars, precious, Steve Martin, twitter, up, yahoo! movies
Post a Comment
Posted March 4th, 2010 at 5:47 pm by Lucas Mast, Blog Editor
No Comments » / Filed in: Uncategorized
What do Jennifer Lopez, live animals, giant bananas, and a guy sitting on a couch watching TV have in common? They all contributed to the success of Yahoo! TV’s Primetime in No Time, which just streamed its 500th episode!
But the real contributor to the success of PiNT has been you, dear viewer. Because of you, our popular TV recap show has even more reasons to celebrate…we’ve reached more than 365 million streams, 12 million of which were on one day for Jennifer Lopez’s fall during the American Music Awards. Note to celebrities: please fall down more often and feel free to interrupt Taylor Swift any chance you get. Man, I love LIVE TV! And you guys are spending an average of 6 minutes on our page. Our show is only 3 and a half to 4 minutes long. This is why we love you!!!
We’ve had some good times on PiNT. Remember when Donald Shultz from Wild Recon brought live animals on the set? And who doesn’t love the skits where I get to sing and dance. And of course, my personal favorite — the Steelers winning the Superbowl. It’s their 6th by the way!
But it wasn’t all good times, for me anyways. My least favorite moment was getting sucker punched, falling off of some giant bananas and more torture all in the name of good television as a guest on ABC’s hit show Wipeout. As you are reading this, know that I am still in pain but will continue my TV watching duties. Hopefully I will still be able to have children after my final crash.
Let’s not forget to thank the many special guests who visited the PiNT studio in Los Angeles including the Glee Kids who were really jealous of my voice; reality show contestants from Survivor (thanks for showering before coming in) and The Bachelor/ The Bachelorette; actor Harry Hamlin and comedienne Margaret Cho.
But back to you, dear viewer. Thank you for watching Primetime in No Time on Yahoo! TV, for following me and Yahoo! TV on Twitter (@franknicotero and @yahootv), and for making us THE most popular online video show.
-Frank Nicotero
Post a Comment
Posted March 3rd, 2010 at 1:29 pm by Lucas Mast, Blog Editor
No Comments » / Filed in: Uncategorized
Surfing the Data Wave: The Problem of Managing Events and Updates for Hundreds of Millions of Users
Key Scientific Challenges, Entry #5: Web Information Management
On January 27 we announced the kick-off of our 2010 Key Scientific Challenges Program. To highlight the scientific challenge areas included in the program, we launched a series of guest blog posts earlier this month on Yodel Anecdotal. Read our previous post on online advertising, “The Art and Science of Advertising.”
Another big challenge our Yahoo!’s research scientists are continually examining is Web Information Management. In this entry, Brian Cooper from Yahoo! Labs shares some thoughts on how Yahoo! is driving research into information management and why it’s a fascinating field.
A Torrent of Data
Sites like Facebook and Twitter demonstrate that users really like to know what’s going on with their friends. At the same time, the popularity of blog readers and personalized RSS aggregators like MyYahoo! and iGoogle show that users also like up-to-date news. Although social networking and content aggregation seem like different applications, at the core they share a key mechanism: collecting the most recent content from a set of producers, and distributing it to a set of consumers. In the case of social networking, producers write status updates or post links, and consumers (their friends) get a list of the updates. In a news aggregator, the producers are news sites or blogs, and the consumers are the readers that want the content.
The core mechanism that takes content from producers and redistributes it to users is actually quite hard to get right, and is an example of the really tough problems that fall under the Key Scientific Challenge area on Web Information Management (and its sub-topic Data Management)
The main problem is the multiplicative explosion of updates. Consider a popular user of a site like Twitter. Ashton Kutcher, for example, has over 4.5 million followers as of February 18, 2010. Every time Ashton tweets, his words of wisdom have to be propagated to the feeds of all those millions of followers. Even if the average user has far less followers, their tweets may have to be propagated to hundreds of users or more. Suddenly, even a short 140-character message can consume a lot of server resources. It’s for this reason that Twitter notoriously found it difficult to scale and stay up at the same time. Some of the challenges in doing so have been described by Twitter engineers.
At Yahoo!, we’ve been aggregating and distributing content for a long time, and we’re increasingly distributing social updates too. Products such as Yahoo! Mail and Messenger show you recent updates and activities by your friends on Yahoo!. Plus, services like Yahoo! Updates provide tools for developers to feed a variety of different user activities and updates that happen on their sites into Yahoo!’s products (and vice versa). As we build our infrastructure to support both content and user updates, we run into many of the same scaling problems that other companies have been reporting. To address these challenges, we took a step back and looked at the whole architecture needed to support these “user event feeds”.
The Push or Pull Dilemma
I’d like to talk about one aspect in particular, as it directly addresses the intersection of building Internet-scale databases and dealing with social data. When distributing updates from producers to consumers, you have a fundamental choice: should you “push” updates to consumers, so that when a user logs in their feed is already constructed and ready to go? Or should you wait until a consumer logs in, and then “pull” updates from producers to construct the feed on the fly?
Pull ensures that we construct feeds only for users that actually log in, but it has a downside: the time a user spends waiting for their feed can be quite long because the system needs to collect and sort all of the updates from all of the people or news sources that the user follows. Many sites face this tradeoff between doing potentially unneeded work to always be ready on the one hand, and the strain of delivering on demand performance on the other. Digg recently reported moving from a pull to a push model for one part of their site, resulting in a huge increase in the amount of “pushed” data that had to be stored, but a significant decrease in response time for users.
Threading the Needle
At Yahoo! Labs, when we thought about this problem and tried to figure out whether push or pull was best, we realized something that should have been obvious all along: neither push nor pull is best.
Consider a producer named Alice that posts a new status update once an hour. We need to include her updates in the feeds we produce for her friends. Thus, when her friend Bob logs in, he should see some of Alice’s recent events. But we only really need to show Bob some of his friends’ events, because there is only room on the screen to show him 10 at a time. Now imagine that Bob only logs in once a week. In that week, Alice may have produced 80 or more updates. If we pushed all 80 to Bob’s feed, at least 70 will have fallen off the front page (replaced by newer Alice events or newer events from other friends) by the time Bob logs in, and all the resources (network bandwidth, disk I/O, CPU time) spent on pushing those 70 updates would be wasted. Multiply that waste by millions of users, and you can see where this is heading.
Alternatively, consider Carl, another one of Alice’s friends. Carl loves to check his feed and updates it every five minutes (he’s an over-sharer). In this scenario, a push model makes more sense: rather than querying the database every five minutes for Alice’s updates, we should push Alice’s events to a pre-computed page for Carl, and just serve him that same page over and over until a new Alice event arrives and updates the page. That way, Carl’s experience is great for him AND no servers meltdown!
The key insight is that we should do push for some producer/consumer pairs and pull for others. Deciding whether to do push depends on the relative frequency of the producer’s events
compared to the consumer’s logins, as well as the relative cost of push versus pull. We can minimize resource usage by only pushing for producer/consumer pairs where the consumer logs in frequently enough that the cost of pushing, plus serving the pre-computed page, is less than the cost of retrieving the event repeatedly from the producer’s queue.
It’s a challenging model to develop and put into place. It can be complicated by an almost endless array of variables, and the challenge of implementing this approach in the midst of hugely popular events, like Barack Obama’s inauguration, or Michael Jackson’s death, which caused huge surges in status updates, makes it even more difficult. It’s also what makes it utterly fascinating.
More details about this challenge, the techniques, arguments and the real-world experiments we’re developing at Yahoo! Labs will be available in our upcoming SIGMOD paper, “Feeding Frenzy: Selectively Materializing Users’ Event Feeds” written by Adam Silberstein, Brian Cooper and Raghu Ramakrishnan, as well as Jeffrey Terrace from Princeton, who was an intern here at Yahoo! Labs last summer. You’ll be able to find the paper on the Yahoo! Labs site later this year and take a look.
Brian Cooper
Yahoo! Labs
Tagged: Digg, Key Scientific Challenges, twitter, Web Information Management
Post a Comment
RSS
E-mail
Twitter
