Cyber Monday into Resolution Tuesday

Posted November 27th, 2007 at 1:40 pm by Rich Riley, Online Channel Division

 61 Comments / Filed in: Behind the Scenes

By now you have probably heard about the problems that many of our small business merchant customers experienced yesterday. Unfortunately, the system outage occurred at one of the worst possible times, and despite our concerted efforts to fix the problem as it emerged on Monday, we know that we let our merchant partners and their customers down. The good news is that our systems are now operating normally, and our merchants are able to accept orders from their customers.

Here’s what happened:

• On Monday at 6:00AM PT, the systems that power our merchant stores experienced outages, and shoppers of those stores were met with either error messages or they were unable to complete the checkout process.
• These issues lasted until about 1:00PM PT when, despite slow performance, transactions began going through at a much higher rate.
• By 6:00 PM PT things were back to normal and the performance of our systems was at 100%.

We deeply regret the inconvenience this caused to both our merchants and their shoppers. Our customers’ expectations were not met, nor were our own. And we are moving mountains inside Yahoo! to find out why and how this happened, and to take steps to try to ensure it doesn’t happen again.

As for the future, rest assured that we are taking the necessary steps to prepare for the peak holiday selling season. We have technical and customer relations staff mobilized and ready to support our partners.

At Yahoo! Small Business, we know that our success and our customers’ success are interdependent, and yesterday’s issue reminds us that we need to continue to work even harder in the future.

Rich Riley
SVP, Online Channel Division, Yahoo!

Post a Comment Bookmark This Digg This

61 Comments Add your own

Rick | November 27th, 2007 at 2:26 pm

“And we are moving mountains inside Yahoo! to find out why and how this happened”. I sure hope that Yahoo has some better idea as to why and what happened. Frankly, communication from Yahoo during the outage was terrible. Yahoo fumbled technologically and in their response to the issue.

As more details are know, I expect Yahoo to be transparent on what the problems were and to learn from their worst “October surprise (OK, November)” yet. Tell us merchants what happened and actually listen to our repeated requests for no new functionality in Q4. Frankly, Yahoo just doesn’t get this point. Things like the new cross sell seem great, but they were clearly not ready for prime time and rolled out just a few weeks before Thanksgiving. That type of feature should be rolled out no later than end of August.

Craig | November 27th, 2007 at 2:27 pm

This outage cost us big time in terms of money, our time and customer goodwill. We had sent out a Cyber Monday discount newsletter early Monday morning and had tons of complaints via our toll-free number of customers trying to place orders. To try to deal with the outage we posted an embarrassing message on our website noting that our server was not working properly and offered free shipping to any customer who would give our toll-free number a call to place an order. We then had to allocate staff members away from their normal jobs to handle the increased call volume. I am very disappointed in Yahoo!. We pay Yahoo! a lot of money each month (when you consider the percentage they get of our sales) and I expect a lot better service. Yahoo! should immediately come up with a plan to compensate merchants for this disruption of service on the most highly publicized day of online shopping. I think that giving a refund of all fees that Yahoo! collected from each store damaged by this outage for the prior six months would be appropriate. These are just my thoughts. If some folks disagree feel free to comment. Craig Clark. Owner. http://www.PacificPillows.com

George | November 27th, 2007 at 2:51 pm

I don’t believe this is the “real” reason. Those Legacy Stores that still used the old checkout had 100% uptime and orders flowed fine all day.

Wayne Eskridge | November 27th, 2007 at 2:53 pm

Just telling us the time line of what happened isn’t very useful. We already know that as we watched it happen and suffered the lost business because of it.

If you want to gain back the confidence of your customer base you ought to be more specific about what happened, why it happened, and what you are doing to assure us that it won’t happen again. Contrition is justified but information would be more valuable. Hopefully, that will be forthcoming, but I suspect Yahoo is more likely to offer platitudes instead and hunker down till the storm passes.

From our perspective, this is the first time we have felt the need to consider finding ways to protect ourselves against the possibility that Yahoo may not intend to commit enough resources to keep this system running reliably.

Won | November 27th, 2007 at 3:11 pm

If you really care, please take the time to read the email I sent to my account manager (below) in the hope that she would pass it along to management.

The following quote is a perfect illustration of the wrongheaded mentality at Yahoo!:

“And we are moving mountains inside Yahoo! to find out why and how this happened, and to take steps to try to ensure it doesn’t happen again.”

You should have moved mountains BEFORE the holiday season to ensure that this didn’t occur instead of wasting everyone’s time with a half-baked and hasty “cross-sell” roll-out that still doesn’t work all that well. In fact, I seem to remember receiving an email from Yahoo! Merchant Solutions prior to the holiday season ensuring us that new server upgrades would result in a faster and more glitch-free shopping experience for our customers. At the time, I got a warm and fuzzy feeling, believing that Yahoo! actually did something useful to add value to its product. Boy, was I wrong.

> Date: Tue, 27 Nov 2007 00:18:32 -0800 (PST)
> From: xxxxxxxxxxxxxx
> Subject: RE: Outage
> To: xxxxx

Thanks, but the System Status page was not exactly illuminating — short on specifics and no suggested workarounds. In addition, I (and the national media apparently) was frankly appalled by Yahoo!’s extremely poor communication during what many in the press (including CNBC) described as a “meltdown.”

Forget the fact that Yahoo! should have been prepared to handle “heavy traffic” on Cyber Monday — “You’d like to think the company would have anticipated ‘heavy holiday traffic’ and could have kept its merchants up and running.” – Jim Goldman, CNBC — I thought that it was particularly egregious that Yahoo! failed to suggest a workaround that many merchants were actually implementing after reading about it in a non-Yahoo! web forum. Many merchants could have simply switched to v2 checkout and continued to receive orders. No thanks to Yahoo!, I implemented this workaround after 8 hours of downtime and at least $10,000 in lost revenue (not to mention irate customers and irreparable harm to our business reputation). Those who were unaware of this workaround were impacted far worse than we were. Even worse, I must also note that it took Yahoo! no less than 10 hours to post an apology on the System Status page.

My anger and frustration are not directed at you specifically, but at those who were responsible for managing and implementing the Yahoo! Store platform. My sincere hope is that you will communicate several of the points I made above to your management team.

I would also like to receive a more detailed explanation about what occurred today, who was responsible for causing the problem, and what specifically will be done to prevent a recurrence of the problem during this holiday season (and beyond). Please also give me a good reason(s) why I
shouldn’t switch to a different shopping cart provider at my earliest opportunity. Finally, what will Yahoo! do to “make things right” for its thousands of merchant customers? I suggest that, at a MINIMUM, Yahoo! waive all fees for this month. That would be a start.

wrote:

>
> We are still investigating. As soon as we have more information it will
> be provided here:
>
> http://updates.smallbusiness.yahoo.com
>
> Regards,

Rob Snell | November 27th, 2007 at 3:15 pm

Rich — Great note, but it’s a little late for most merchants. I posted this and some other comments on WebMasterWorld:

My biggest complaint about Yahoo! Small Business these days has been with the lack of proactive communication from Yahoo to merchants about the specifics when anything is on fire.

To give credit, my account rep replied to several emails on what was arguably the busiest day of his career, and gave me honest answers.

I do think Yahoo dropped the ball here on what was a textbook marketing/PR opportunity for merchant communication in a time of crisis. For example, there’s a 37-page thread on the Ystoretools Forum with over 17,000 page views. Holy cow! I’ve never seen a thread that long on YST!

I know all the product people had their hands full, but IMHO someone who was NOT an engineer should have stepped up and posted info about what was going on, even if it was to say we don’t know what’s going on, but every available resource was working on it.

…an apology and a token service credit would go a long way in placating thousands of merchants.

What’s frustrating is when the average retailer gets more information about Yahoo! and what’s going on with platform-wide outages from the New York Times, CNBC, and other national media than from Yahoo itself. When that happens, it looks like some folks at Yahoo! are more interested in covering their ass than in keeping merchants in the loop. — Rob

J.D. | November 27th, 2007 at 3:16 pm

Look we understand that mistakes happen, but this was catastrophic… and from the tone of the blog, it looks like you still don’t know what happened or you don’t want to admit it. That’s a huge problem and is telling about the people making decisions at Yahoo. That’s almost MORE troubling than the fact that you let the meltdown happen in the first place. It’s been a day and a half and you still don’t know what caused it? C’mon. That’s just incredible.

1. Not telling the whole truth about what happened is really poor form and erases my confidence in the people writing the spin at Yahoo.
2. Admitting that you messed up, but that you have no idea what happened, on the other hand is poor business and it erases a lot of the confidence I once had in the Yahoo platform.

Please just come clean and let us know what really happened. You owe us much much more than that.

Frank Bianco | November 27th, 2007 at 3:49 pm

I agree with some of the previous comments that Yahoo did a horrible job communicating with their customers on Cyber Monday. I was doing google searches trying to find articles from 3rd parties to find out what was going on as Yahoo only stated that they were continuing to investigate the issue. This was the “only” information provided by Yahoo from about 12pm est to 6pm est. Yahoo tech service had a phone message that indicated you should continue to hold for 30 minutes (average hold time when I called) if your issue was “unrelated” to their outage. Another words they were saying hold on we will talk to you if it’s not about this crash. We were totally in the dark & spent time considering shutting down PPC campaigns & comparison shopping engines ( which we should have as losses are in the thousands from PPC & CSE’s alone) which charge us by the click & are a huge waste of money if a customer cant complete a sale. What’s worse is I called billing today and the rep had no idea about the outage, transferred me to a supervisor who suggested we speak with Yahoos Legal Dep’t. if we expected to be compensated for our losses. This is the first time as well our company is considering a new host. I think Yahoo needs to be very specific on what’s happening and provide estimates as to when their system should start working in the event of outages as well Yahoo needs to compensate their customers for this loss on the most important day of the year. Do I expect yahoo to credit my account the 10k+ I lost yesterday NO, but you need to do “something” to take responsibility economically for these loses and gain trust & confidence back.

Mark Hawks | November 27th, 2007 at 3:59 pm

I agree that it happened at the most inopportune time. From the phone calls I received throughout the day, I know that I lost business because of it. I chose Yahoo as a host because I thought that uptime would not be an issue. There should certainly be redundancy for the”systems that power our merchant stores” and I agree with other replies that you should provide more information on what specifically happened. To me, this is not acceptable. I have been receiving Error 103 codes for the last week and a half and then to top it off my customers can’t even order something online for 15 hours! Another blow on the bruise. What took so long to fix? Why is there no redundancy? I have lost faith.

Jerry | November 27th, 2007 at 4:22 pm

Were we ever notified of the problems? NO
Were we ever contact, even by email about the problems?NO
Did we lose orders and income? YES
Will Yahoo compensate us? REMAINS TO BE SEEN
Will you ever contact us directly? DOUBTFUL

All of this is not very customer friendly.

Bob | November 27th, 2007 at 4:35 pm

I hope you get the gist of what is being said here, on YStore Forums, WebmasterWorld and news sites like CNBC and NBC.

This is incredibly SAD:
SAD – that one of the giants of the internet can let this happen at a criticle time like this.

SAD – That when something catastrophinc like this does happen, that one of the giants in the Iternet industry could treat their Loyal merchants so poorly, as if they do not even exist.

SAD – That there was not up front and immediate communication about the issue. Started as an error experienced by only a few merchants, went to a configuration problem and then to a system outage.

SAD – That Yahoo did not use the System Status link and the YStore Blog on the manager’s page to let your loyoa customers kow what was going on. The first post in the YStore Blog was today!

SAD – That Yahoo did not see fit to send emails to their Loyal Merchants about the problem, effects, workarounds, etc.

SAD – That Yahoo did not provide the workaround that was available, at least to some merchants. That should have been your answer to your loyal customers, not someone else’s.

SAD – That Yahoo is taking a large PR hit becasue of poor planning or poor implementation of a new functionlality late in the shopping season.

SAD – About the extensive Season & Year Ending losses by all the Loyal Yahoo Merchants that were effected. I sincerely hope it does not drive some folks out of business.

SAD – That the negative PR affects not only Yahoo (who has to take repsonsibility for this situation) and again your loyal merchants, who also depend on Yahoo’s name as a bell weather of good service and trustworthy stores.

Did I say this was SAD?

Lawrence Ames | November 27th, 2007 at 4:53 pm

One issue here is that Yahoo! is telling the media that the outage was due to increased traffic. This was simply not the case. Yahoo! was well aware of the increased traffic that Cyber Monday would bring, and their servers can certainly handle it.

The issue that caused the problem was their new version 3 Checkout cart and the ill-advised addition to new code regarding cross-sells right before the busiest shopping day of the year — code that was either not written or not tested properly.

Merchants who were able to switch back to the older version 2 cart were able to circumvent most of these problems.

Yahoo could have been more honest that it wasn’t increased traffic but bad coding that caused the outage.

Saying that the outage was due to increased traffic is disingenuous and puts the cause elsewhere than in their own offices.

kevin marks | November 27th, 2007 at 5:00 pm

I’m frustrated. We put a good amount of money into the development of our yahoo store only to continue to have problems. It was not just on Cyber Monday – but on many other days. What about fraud protection – you have Visa running with their verified by Visa program throughout the world yet when I call Yahoo they know NOTHING about it. So, we continue to deal with fraud issues. This was the best thing that could have happened – because now maybe senior mgmt is yelling – “What The Heck Is The Problem Guys”. On top of it, I’ve worked in many e-commerce companies and honestly, I suspect they truly don’t know exactly what caused the problem – but can only make an educated guess. If one does not properly load test functionality to the MAX before launching it – then how does one know what piece of functionality caused the problem? And on top it, when you have an old platform that desperately needs an upgrade – its like building additional floors on top of a weak infrastructure.

Yes, some compensation is in order – but I would not expect a dime from Yahoo…it will take a class action suit for it to be addressed unfortunately :(

Flavio | November 27th, 2007 at 5:04 pm

I will state the obvious:

The Yahoo! Store is our livelihood. We and our families depend on it. We don’t get paid salaries for our time. We wake up every day, prepare for the worst and hope for the best. Start campaigns, monitor pay-per-click and compete with the likes of Amazon. Our job is to provide the best customer service and profit as much as we can in an extremely competitive environment and we do believe and know that we provide much better customer service than the big guys. We have to fight an prove every second that we are trustworthy to our customers because we are not a house name. It’s a heck of a job but we do it. That’s why we own successful online businesses. We do it but it’s a very tough job. And we do it with limited resources. Both financial and human.

From a successful small online business owner standpoint it’s extremely difficult for us to look at all this and not be mad. If we can do it, why can’t a multi-billion dollar company? The only thing that comes to mind is dismiss, incorrect assumptions, careless execution. Actually all assumptions are DUMB in its core. That’s why we prepare. Y! should try and improve their corporate culture. If it’s not about the customers (a.k.a. we Y! Store owners) it’s not about anything else. Y! should try and change fast. I remember when we first opened our Y! Store in 2000 the “store owner manual” would recommend to have a phone line for customers to call us, preferably a toll free number. The funny thing was that it took Y! a couple of years to have their own toll free number. It has definitely improved since then but frankly, to let this ball drop on Cyber Monday is beyond comprehensible.

To this date the Web-hosting solution is still filled with glitches and it was introduced to us as part of the Merchant Solutions platform. In the beginning it was the only way to upload images and then the “lib” was brought back. But now the Web-hosting still needs to be used for custom solutions and it continues to be simply very unreliable and unstable. We had to move hosted integrated solutions to a third party. Our blog actually vanished but Y! told us that the Web-hosting is not part of Merchant Solutions. It’s all very frustrating.

Bob Rawley | November 27th, 2007 at 5:21 pm

I will not jump on the band wagon for lawsuits or compensation. It’s not like we will ever know how good or bad we would have done yesterday without the outage.

I would like to say your communication was POOR and I would also like to know how you intend to make sure it does not happen again.

It would be nice for you guys to poke into http://newforums.ystoretools.com and let us know what’s going on if you dont want to post official status up-dates.

Byron Fast | November 27th, 2007 at 6:31 pm

How Yahoo! handled this particular event is unfortunately par for the course for longtime Yahoo! Store owners.

As a web software service Yahoo Store is remarkably unprofessional when new releases arrive, although they have greatly improved in recent years. Throughout the history of Yahoo Store, Yahoo has quietly introduced releases which have resulted in unannounced shopping cart changes, broken connections to external systems, and mangled valuable page links required for search engine placement.

Yahoo! Store likely employs hundreds of thousands of people and has been a source of actual revenue for Yahoo among a giant collection of unprofitable acquisitions, yet little attention seems to be paid to how it is run. If Yahoo could simply behave like a professional software firm and manage the releases like software companies did in the bad old days of desktop applications they would seem a lot smarter.

Claiming that “heavy traffic” brought them down is great marketing spin but every Yahoo! Store owner knows that Yahoo recently introduced new cross-sell features and when software breaks the best hunch is to look at what changed last.

Either way, Yahoo! looked a little unprepared: it takes a whole day to add capacity? Or a whole day to roll the servers back to a previous version? Believing either is a little shocking to this store owner.

Sheri | November 27th, 2007 at 6:41 pm

I have been with Yahoo! for almost 6 years. This is unthinkable. I am a small business, so a really bad day in the holiday season is devastating….we received so few orders due to this checkout problem. That translates into lots of current and future business lost. It is no exageration to say that if this happens again in the holiday season this year, it could put me and others like me out of business…can you say lawsuit?
I think Yahoo’s inability to contact customers directly about this problem is inexcusable.
I am extremely upset about this and hope that Yahoo! will move forward and do these few things or I will be spending a large part of early 2008 looking for another host:
1)Define and tell us exactly what went wrong
2)Change engineering practices to cover a problem like this BEFORE it happens, not after.
3)Credit my account for 1 to 3 months payment to help cover a small amount of lost revenue and to start to gain credibility with me again.
4)Develop a better process for both honestly defining a problem and then to effectively communicate that to each and every customer directly (not through a meaningless summary on the “System Status” page)
5)Offer workarounds to customers when this happens…I understand a workaround was available yesterday, but was not communicated to customer by Yahoo itself.
6)Prevent this from happening again this holiday season or I will be moving away from Yahoo either because my business failed because of their errors or because I am quickly seeking another host for my business.

Louise | November 27th, 2007 at 7:10 pm

I was quite disappointed with Yahoo! for not letting merchants know there was an issue with customers being able to order. I know that we lost several customers as a result of this. Some of our customers are quite persistent and continued to try placing an order and also called to place their order. Fortunately we have an physical store as well and can process orders this way.

We did not get any order via the website until late afternoon, we are located in Minnesota.

Catheirne Seven | November 27th, 2007 at 7:22 pm

Imagine on Black Friday the mallsstore managers lost the keys and well the malls, best buy, toys r us and more just can’t open, for the biggest shopping day of the year.. (You can look in the window, but you can’t buy!)

Y! Do you think that the management would just let the merchant come to work and say nothing to them and they try and open the doors! let them look around with um… why aren’t we open? Hey look there is my merchandise but no one can buy it!

I would say a great % of Y! Merchant only support their business online, and they were closed for more than 12 hours? um… Yahoo please have your account reps call the merchants, at least if there shopping carts are down LET them know instead of them calling their Marketing Consultants and complain that they have no sales, and let their team do the research for you. – Sad… I am sorry merchants!

Anna | November 27th, 2007 at 7:35 pm

I am horrified that a company such as Yahoo could let something like what happened yesterday occur on one of the most important internet retail days of the year and it was totally inexcusable. Our legacy store site was down from about 10:30 AM until 10:15 PM and this cost us many thousands of dollars in lost business that we will never recover. Yahoo claimed it was because of increased traffic but I don’t buy that – I think they screwed up royally and that we should be compensated. Who is going to reimburse us for all the money we wasted paying search engine fees for nothing for the entire day on top of that??

The problem should have been repaired in a more timely manner and it was despicable that they had the nerve to give us a link to updates which were never updated. There were no updates whatsoever between 8:30 and almost 7 1/2 hours later at 3:56PM – and even that one was useless. That is totally ridiculous and unacceptable. Why didn’t anyone at Yahoo customer service have any information other than “our engineers are working on it” and possibly offer us any suggestions as to switching to a different shopping cart? I had no idea that this was even a possibility. There was no communication at all about anything!

To add insult to injury, it was an embarrassment having to put a notice on our site telling angry customers that we were having technical difficulties and we could not even accommodate the phone calls we received with questions all day about our shopping carts not working. We could not even process the orders ourselves and it put us into an extremely frustrating situation with our customers who were interested in ordering from our site all day long but who took their business elsewhere. We work so hard every day to do our part in trying to run our business and pay all of this money to Yahoo without having any protection against this happening to us.

Shame, shame on you Yahoo! You had better come up with some plan to compensate us for this horrific screw-up.

Mike | November 27th, 2007 at 8:02 pm

It is unfortunate that this happened on cyber monday but it can happen at anytime. We have found that the best way is to have multiple websites on different platforms. It is more difficult to manage but the market coverage is greater.

People are always trying to hack the internet and we are always aware that it can happen at anytime. We went through a 4 day power outage and our backup generators only lasted 8 hrs. Online business rely on our hosts and the power grid.

Thanks to Y for getting it fixed as quickly as they did. Yes we lost money but hopefull the customer will come back it’s not over yet.

DigMyPage | November 27th, 2007 at 8:31 pm

Thank God that I don’t have any Yahoo store. I used to have a number domains in Yahoo Small business. My experience was horrible.

Their server was slow to the point that most of the time it will time out. You have to perform the same operation to get one job done. It was a total waste of time.

The user interface was intolerable. I don’t know who designed that domain management site. Probably somewhere in Elbonia where developers thinks they are super intelligenct and customers are dumb. Or those people simply don’t know how to provide a good customer experience.

I have moved all my domains to Godaddy and I am very happy.

Kevin | November 27th, 2007 at 8:58 pm

The best is yet to come. According to comScore, yesterday should only be the 12th busiest shopping day of the year with heavier shopping days to follow. This gives Yahoo! a chance to shine and continue the amazing service that they have provided for years.

I also feel the frustration of yesterday, but I think that it is important to acknowledge the years of successful uptime and reliability of the Yahoo! Store.

John M. Rowell | November 27th, 2007 at 10:23 pm

I’m waiting for an honest explanation re why it happened and more important what is being done (already done I hope) to prevent it from happening again.

During the day I considered switching to the old checkout system but was frankly afraid to make any changes for fear things might get even worse.
Yahoo should have quickly communicated that this option was open.

I’m waiting for Yahoo’s financial compensation for our losses.

Steve D | November 27th, 2007 at 10:42 pm

Kevin- wow! Talk about a self-serving post. Not only are you a developer that relies, at least in part, on Y! store customers, but you actually post on your site:

“With 99.9% uptime you never have to worry about your site crashing, even if your site gets a sudden spike in traffic.”

Might want to remove the sentence.

I think Y! does have a fairly good record with uptime, but your thinly veiled post is shameful.

Howard | November 28th, 2007 at 5:26 am

I noticed while reading Yahoo Sports many banners for the new “Yahoo Sports Store” in place of banners that used to direct customers to Yahoo Shopping.

Is Yahoo in the retail business now? Shouldn’t their priority be pushing customers to Yahoo Shopping so they can get there .20+ per click?

Are they planning more retail ventures to cut into the profits of Yahoo Store Merchants?

They should focus on running the best and most efficient store/shopping platform instead of branching into retail.

Lets start a pool when the next outage comes. I say the fax and email order confirmations have glitches like they have every holiday season for 10 years.

Good Luck to all Merchants.

spenser | November 28th, 2007 at 5:59 am

The original post gives a reason for the outage but conflicts in timelines with the status notification page.

Although the details of the power outage and it’s impact are non-existent, I will hazard a guess for the benefit of those readers who are not familiar with data center environments.

It would seem that the power outages hit a critical backend component. If that is the case, then the “workarounds” referred to on the status page would be efforts to route around the missing component. That component is not a single server, but could be a group of servers deployed to provide that particular backend function. For all I know, it could have been a database server. But, I can’t know for sure. Only internal staff has that knowledge.

The reason the legacy systems continuued to function in some fashion could well be due to the functions being deployed on a different set of devices not affected by the power problems.

One observation though, and not unique to Yahoo! is that old dinosaurs of the industry always approached release code with the attitude that something *will* go wrong. As a consequence, a multitude of backout and contigency scenarios were always in place before signing off on code rollouts.

The attitude of newer entrants seems to be *of course it works, I’m brilliant!* This is more common in Web 2.0 than you might be comfortable with.

Well, hubris always has a cost. It’s just a question of who pays.

How would you like this attitude permeating throughout financial institutions as the old cobol code toads get pushed aside?

At the very least the geezers, not geeks, have the intestinal fortitude to stand up and say “uh, I fat fingered it, and here is exactly what happened …”

Nothing beats experience in this game. Not brains, education, enthusiasm nor money.

You need people with the battle scars to prove that they know what *not* to do.

Brian | November 28th, 2007 at 6:34 am

I was not notified by my account rep or Yahoo of a problem. I tried calling the 800 number and when I got a canned response, I knew the problem was systematic. I then went to the Yahoo store owners’ forum (a non-Yahoo forum) where fellow merchants help each other because frankly Yahoo does not. Someone proposed rolling back to the old cart as a workaround. I then contacted the person who does my Yahoo store for me and he helped me roll back the cart.

The pattern here is I am used to not getting any useful information from Yahoo. No clue was given as to what might have been wrong or a possible workaround. No proactive emails were sent out, just a blurb on the System Status page which is pretty well buried in the store manager. You could have put up a banner where you place all those ads in our store manager.

Two days after Meltdown Monday, I get an apologetic email from Yahoo. Still not a peep from my account manager (who could at least have sent out broadcast emails). No detailed explanation of what went wrong or what you have done/are doing to fix it. I’m back up and running on the new cart. But, I’m nervous about it because if it stops working again, I know I’ll hear about it from my customers before I hear about it from Yahoo.

Unfortunately, this is pretty typical for Yahoo’s customer support.

Nora | November 28th, 2007 at 7:33 am

I am appalled at the lack of communication from Yahoo during this issue. We run two stores on the Yahoo platform and watch our visitors in “real time” as they peruse the sites.

You can imagine how happy we were to see our traffic had doubled on Monday, as we watched customers browse the pages. But, after a few hours it seemed odd that no orders were coming in. We had a few call in orders (good thing we advertise our toll free on the site), but when we tried to place them online we had issues. I thought it was a temporary glitch at the time.

However, when someone from the office went to place an order with another Yahoo store and couldn’t, we knew something was up. That’s when I started searching forums/message boards and the internet for an answer.

It’s really sad that we had to search for this ourselves, rather than be notified that there was a problem. Thanks goodness for the YStoreTools forums, mentioned by Bob above. If it weren’t for the people there, we wouldn’t have been able to implement a fix (work around) that seemed to work for the time being.

We switched back to the “old cart” on our one store (didn’t have that option on the other) and within 15 minutes we had four new orders come in. But, that was around 4:00 PM (our time) and the damage (severe damage) had already been done. Orders began coming in steadily after that. I can’t even begin to imagine the amount of business we lost on Monday, not to mention the first time customers (who got a really bad first impression of our shopping site), who went on to competitors who don’t run on the Yahoo platform. I guarantee I lost many potential repeat customers to this outage.

Had Yahoo notified customers of the problem earlier and suggested this fix (switching to the old cart), perhaps it could have saved many merchants from a dismal CyberMonday. I can’t imagine why “increased holiday traffic” was a problem, when those with the “new cart” couldn’t take in orders, and those with the “old cart” could. That does NOT sound like a traffic problem, but like an infrastructure problem within Yahoo.

BTW, I took the incentive to contact all of my business contacts who also run Yahoo stores and didn’t even know there was a major Yahoo problem, but had noticed no orders coming in on Monday themselves. They simply thought it was a “slow” day for them. Many that I contacted were able to switch over to the “old cart” and begin seeing orders again.

Come on Yahoo…a simple email to your merchants to let them know what was going on would have been a nice step. Not everyone checks the “system outages” pages on a daily basis. Many Yahoo store owners are small to medium sized businesses, with people running them that wear many different “hats.” They don’t have the resources or time to keep checking your support pages for potential outages.

Is a simple email too much to ask when there is a problem? I think it would have gone a long way to soothing some pretty angry folks who may now leave come the new year.

Bob | November 28th, 2007 at 7:41 am

The communications isues were really bothersome, as has been stated many times here and elsewhere. This is absolutely the most important thing Yahoo could do when there is a major “outage”. One thing I can’t understand – Why at the end of the status updates, such as they were, did Yahoo place a link to this blog (Yodel.yahoo.com). One that I have never seen or heard about before – when they have the Official Y!Store Blog http://www.ystoreblog.com/ on the manager’s page.

They have what could be a good communication tool and then use one nobody ever goes to for Yahoo Store info to give an apology. I like the idea of apologizing to the Yahoo Population, but what about communicating with people that are paying for your services. Still have not had an official email from Yahoo apologizing. Small things can make a big difference at a time like this.

Greg Tice | November 28th, 2007 at 7:49 am

No need for me to add more comments on Y!’s incredibly poor communication. The only thing I want to bring to light is the fantastic job Rackspace did in communicating and truly accepting responsibility for the problems their customers encountered as a result of the shutdown of their DFW datacenter on November 12th. Checkout the link below, the first post is at the bottom of the page.

http://www.rackspace.com/information/announcements/datacenter.php

Brian | November 28th, 2007 at 7:59 am

I’m so disappointed and angry at yahoo. Cyber Monday was set to be my first great day online. I’ve worked feverishly over the last 6 months to build my store to get ready for this day and what do I get. And now I’m learning from comments on this that I could have easily made a switch to an older cart version to resolve the problem. Where was Yahoo tech support to tell me that. I made 3 phone calls this day and not once was I offered a solution even after asking. I’ve lost valuable time, opportunity for new customers and money. Thanks Yahoo

Joe | November 28th, 2007 at 8:10 am

While this event had unfortunate timing, I think it is important to note just how reliable the Yahoo! Store platform has been throughout the years, and to not let this one event cast a dark shadow on what has been at least a decade of unprecidented reliability.

Everyone makes mistakes, and the best we can do is to learn from them, pick up the pieces, and move on – allowing ourselves to all learn from the experience, and to prevent history from repeating itself.

From the perspective of a merchant, we learn how to better prepare ourselves for a crisis, and from Yahoo’s perspective, they learn how to better prepare their systems to handle growth, and they also learn what their customers expect from them moving forward.

As uncomfortable as this event was for everyone involved, it is now in the past. I think it is safe to say that Yahoo! will do whatever is in their power to make sure that history does not repeat itself.

“To err is human, to forvive is devine”. – Alexander Pope

Bill Long | November 28th, 2007 at 8:25 am

Yes, our stores also were down on Monday. No information from Yahoo, only the Ystoretools Forum for information. And, incredibly, ZERO inputs from Yahoo on that forum, the only technical forum in existance for Yahoo store owners!

I looked through Yahoo’s financials to see if I could determine what percentage of revenue and profits come from the Merchant Solutions side, but that information does not appear to be public.

So now I have this sinking feeling that all of us merchants are simply **not important enough to Yahoo Corporate** to warrant much attention, because that portion of the business does not contribute much to the bottom line. They have bigger fish to fry, like keeping Google from running over them. The occurance and handling of the Monday outage is consistent with that perception.

So, Yahoo, prove me wrong!

Rick | November 28th, 2007 at 9:13 am

With all due respect to Kevin at Venture Web Design, your Polyanna feel-good post is a joke. You’re a Yahoo small business partner and developer of Y stores. That post is just a shameless love letter to Yahoo and the platform that you depend on for your living. Sorry, Yahoo doesn’t deserve hugs and sloppy wet kisses today.

You develop Y stores, you didn’t feel any pain from the downtime at all. I’m sure your clients did, but you didn’t. So spare us the claptrap about Yahoo getting the chance to shine. They’re job is to provide a stable platform to accept orders, they’re not “shining” by doing what we pay them to do. Let’s see if they can recover from their stumble before we all hold hands and sing Christmas carols.

PJ Brunet | November 28th, 2007 at 9:35 am

There’s no reason to depend entirely on Yahoo. Offer multiple payment options, have a backup plan. Unless you just discovered the Internet you know that servers go down, there’s no such thing as 100% uptime with any one service, but you can protect yourself. Even if your own server fails, if you manage your own DNS you can redirect traffic to a backup server in the event of an emergency. I’m not defending Yahoo, I could care less about Yahoo, but keep in mind nobody forced you to use Yahoo.

Don Fatula | November 28th, 2007 at 10:17 am

Gentleman,

There are mistakes and errors in life, but to this magnatitude it is unforgiveable.
The negative ripple effect to thousands of small & large businesses all around the world is just un acceptable.
No matter how you look at it…A loss….is a loss….and we business owners LOST!!!
Don

Brett Dewey | November 28th, 2007 at 10:19 am

Accidents happen. They shouldn’t , but they do. And when they do they need to be corrected quickly and amends made. That’s how we treat our customers, and I hope how you treat yours.

Since it’s in your best interest to get the problem solved fast I assume ‘all day’ was as quick as it could be repaired. The apology is nice, and some measure of ‘amends’ would be nicer.

I’m sure we can find a way to forgive and move on from the one day disaster.

Just don’t do it again.

Bob | November 28th, 2007 at 12:03 pm

Soooo, I do agree that at some point we need to put the blaming and such aside, although I don’t know if I am quite ready yet or not. So, in looking ahead at what can be done:

Everyone should join the http://newforums.ystoretools.com/ which is where the most help was given for this situation. Not to mention great advice on just about any other aspect of owning and running a Yahoo Store.

Look for other other cart solutions that could be quickly linked to in the event this occurs again. Does anyone have any suggestions?

I have seen some other suggestions about other ecommerce solutions, but what about solutions if one wants to stay with the Yahoo store. Some make it sound easy, but not sure that it is.

Louis H. Bergeron | November 28th, 2007 at 12:21 pm

Your reply to the problem might be acceptable if this was your first Holiday Season, it is not! Yahoo has proved that it does not have the resources, personel and technology to support it’s customer base at peak shopping times.

Not one of our service providers outside of Yahoo had a problem. I know Yahoo will sweep it under the rug and 2 weeks from now they’ll be going “what loss of service?”.

Kevin | November 28th, 2007 at 12:31 pm

Rick & Steve D — We also felt the pain on Cyber Monday, as we own http://www.ScentsandSprays.com. We are in the same boat as you, and have a vested interest in the platform being stable and robust.

Instead of everyone screaming at the top of their lungs for a discount/refund, wouldn’t it make more sense to see improvements made to the platform? Would a $100 credit make you feel better, or would you rather see that pittance go towards the development of additional capacity and capabilities?

Just because Ventura is an Authorized Developer, we do not speak for Yahoo!. My post was simply my feelings on the situation, as the Yahoo! Store has been stable since 1997.

We, as store owners, should all work together and be invested in the success of the Yahoo! Store platform. It’s unfortunate that the attacks are free flowing, while the suggestions for improvement are few and far between.

I believe that banding together as a community of store owners, as opposed to an angry mob, will be more effective in getting the changes that we all hope for.

Pat | November 28th, 2007 at 12:40 pm

I also believe that Yahoo did a poor communication job, but it’s not the first time on that issue. It seems whenever I have a problem, no faxes for two hours, slow connectivity, etc.. when you call them, they basically act either like you are the only one experiencing the problem, or you are an idiot… but then 2-3 hours later, apparently enough people call in with the same problem for them to take it seriously. It’s very frustrating to get any kind of true help or communication from them, whether it’s a serious issue (Monday) or something seemingly trivial (20 minute delays on “instant” faxed orders. It would be really great to see them read, respond, listen, update, fix the problems that customers repeatedly bring to their attention.

Rick | November 28th, 2007 at 1:11 pm

Transparency is not enough when you lose thousands in sales. Perhaps Yahoo should consider reimbursing us for the outage. A free year wouldn’t even begin to touch the amount of sales we lost on Monday.

john | November 28th, 2007 at 1:48 pm

To sum it all up-Highest page hits all year-Very little income.
Thanks Yahoo.

Roger | November 28th, 2007 at 2:22 pm

Almost a year ago, I came over to Yahoo because I felt they were one of the most professional merchat solutions providers around. I had left eBay because their site was not reliable, their communications were terrible and they always made changes during the holiday selling season.

What a disapointment to find out that Yahoo is, in fact, not much better than the bay.

Brian | November 28th, 2007 at 3:22 pm

I’m not interested in a couple of months hosting fees (few hundred dollars at best) in compensation for the outage. What I’d like to see if Yahoo do a MUCH better job at communicating (both ways) with its merchants. Listen to the things we want to run our businesses and tell us when you’re having problems, in real time in real terms. Monday would have been a lot better with some better communication.

BTW, I had never heard of this blog until getting the email today that led me to a link that led me here. And, as a small store owner, I don’t check the status link often (frankly, I had forgotten it was there). Yahoo should have emailed us or put up a banner in the store manager. And, once they were aware of the work-around (a lot of their merchants figured it out on their own) they should have let everybody know immediately.

I can’t afford to leave Yahoo. I have too much invested in it. I can’t afford a back up solution. I am just a Mom & Pop shop totally dependent on Yahoo to provide a stable platform for my store. I’ve traded fancier features on other platforms for stability with Yahoo. But, in my almost 6 years on the platform, one thing Yahoo has been completely consistent with is poor communication. It could hardly be any worse.

longtime yahoo store user | November 28th, 2007 at 4:44 pm

just read about the “latest wave of downtime to come on 11/28″, what a travesty that we have to once again endure yet more downtime and yahoo still won’t offer us the ability to revert to the older checkout platform, which had no problems and older yahoo stores still have the option of switching to. what gives yahoo and when will you announce big time remunerations

Bob | November 28th, 2007 at 5:09 pm

Brian
What email was that?

This was my first time hearing abut this particular blog and I am surprised that Yahoo picked this blog to roll out a “Mea Culpa”. Also, there is a YStore Blog that occasionally has some info in it and is linked form the Manager’s page: http://www.ystoreblog.com/ .

I would have thought they would use that blog for informational purposes rather than “Yahodel” as it is already set up to share info, such as it is.

Sounds like some remedial training in communications is in order.

Rob | November 28th, 2007 at 6:03 pm

It’s just amazing to see an internet pioneer like Yahoo have something like this happen and not prep for it while smaller shopping cart providers had no down time.

I think after seeing this and after all my frustrations with Yahoo on lack of communication and canned responses from their representatives that I’m going to take my business elsewhere. I’m starting to do some research on companies that have more features and are more stable then Yahoo and actually care about the little people. I’d love to get some input from others but so far the best solution I’ve tried is Volusion. If anyone has any others they could recommend I added my email here so please contact me.

Lastly I’d like to remind Yahoo that communication is the most important thing and that they need to listen up fast. They lost the search game to Google and now they are going to loose the e-commerce business to competitors too and its all because they refuse to listen to request and are slow to make improvements to their system. Sometimes I think there is just one developer working there.

Yahoo fell off the horse, lets see them get back on and improve and if they don’t improve then i guess I’ll be leaving them.

Won | November 28th, 2007 at 6:16 pm

I think it’s totally self-serving and misleading for you to state that “[t]hese issues lasted until about 1:00PM PT.” You and I both know that between 1PM-7PM, most shopping carts were so slow that it look several minutes for someone to place a single item into his/her shopping cart. That’s like waiting for a cashier for 5 minutes to scan one item. Sure, there are always persistent customers who will see an order through, but you and I both know that between 1PM-7PM MOST orders were LOST. Saying that performance was just “slow” during this time is laughable — it was EXTREMELY painfully slow. I can’t overstate how misleading, self-serving and revisionist this so-called “mea culpa” is.

It’s this kind of arrogance that culminates in such catastrophes.

By the way, what on earth are you folks thinking when you decide to perform server maintenance smack in the middle of the busiest 2 weeks of the year? Have you considered AT ALL what might happen if something goes wrong with this untimely “maintenance”? I thought the checkout issues were now “resolved” and “closed.” If so, why do you need to perform any maintenance at all? — unless, of course, you are admitting that you were grossly unprepared for the holiday season to the point of reckless incompetence. Why were these server upgrades not made BEFORE the busiest shopping period of the year? And please spare me the ridiculous “heavy holiday traffic” excuse. You and I both know that’s totally bogus. From what I hear, Amazon, eBay, Google and your other more competent competitors were quite prepared for the “heavy holiday traffic.”

Finally, we’re all still waiting for a detailed explanation about what caused this catastrophe, not a self-serving and revisionist recounting of events. We all know what happened — PAINFULLY so. Tell us something we don’t already know — and you just might start to win back some of our trust.

bob | November 29th, 2007 at 9:01 am

I am VERY sick of my Yahoo store! I don’t know how they justify charging transaction fees when there are so many other carts out there. I am switching in January.

The problem is they use the same checkout URL for EVERY STORE. That’s like having one checkout lane for an entire shopping mall. My store was down for 11 hours on Monday. That’s inexcusable.

vance | November 29th, 2007 at 10:26 am

please tell me this is a joke…..many….many yahoo hosted sites are down again

C | November 29th, 2007 at 10:53 am

No joke… our sites were down too. Unbelievable. Yahoo – tell us what’s going on!

Tom | November 29th, 2007 at 2:45 pm

I work for a company that provides services to Yahoo merchant solutions users & while the issues Yahoo has been having didn’t directly affect us in any large way it had some serious impact on YMS users we support. Some are still working on getting through the email fallout which does expend valuable resources especially this time of year. By far the best smaller than Yahoo ecommerce solutions out there are storehost online commerce suite and volusion. Dropping your yahoo site may not be viable or advisable for many users but having an additional storefront on another platform isn’t a bad idea. Yahoo will learn from this and work to correct what went wrong and how it was communicated to its users, they have no choice.

Mario | November 30th, 2007 at 6:36 am

Since Monday we have been experiencing checkout issues. Since Tuesday we have had 4 customers email us saying they can not checkout and the store is quite slow.

If we received 4 emails, imagine how many customers did not email us and just abandoned their shopping cart. This is ridicilous that this issue is still happening.

Ed | November 30th, 2007 at 7:04 am

I have purposely waited a few days to post a comment because I wanted to see what Yahoo was going to do. At this time I am appalled at Yahoo’s customer service. I never received and e-mail telling me that there was any type of a problem. I would think that Yahoo would have notified its customers immediately upon finding a problem. I am still waiting for an apology I do not believe that “We deeply regret the inconvenience this caused to both our merchants and their shoppers” is an apology.

I have stayed with Yahoo for several years despite their hosting limitations and high fees because I believed they had the best possible uptime and resources to keep my store operating smoothly.

Unless Yahoo comes through with some good customer service I will be transferring my store after the first of the year.

Dawn | March 13th, 2008 at 10:38 am

The internet is very hard to make money on these day’s so any downtime is very hurtful!

Jay Stevens | July 15th, 2008 at 1:17 pm

I totally agree, any downtime will result in tons of cash lost for your site. Be sure to prepare for the holiday rush.

Thanks

Jay Stevens
BestCyberMondayDeals
CyberMonday Deals and Sales Found Here.

Jay Stevens | August 3rd, 2008 at 11:24 pm

I love Cyber Monday. Any day that can produce that many power outages online, is a great day to be a part of. I love it.

Thanks

Jay
Cyber Monday

Chrystal | November 2nd, 2008 at 10:20 pm

I am hoping it will not be a repeat of last year!

Linda | July 15th, 2009 at 10:57 am

I didnt even realize that this happened until now. I am looking into a Yahoo store. I have only heard great reviews about the cart and the functionalities. Sometimes things happen beyond our control.