I just posted this on our Yahoo! Store blog, but wanted to follow up here as well.

We’ve heard you and we’ve been hearing from our merchant partners since Monday, and I want to let you know what happened earlier this week and what we’re doing about it.

On Monday and for a very short time yesterday, unusual service interruptions affected transactions in Yahoo! Merchant Solutions/Yahoo! Stores. After an investigation, we learned that the specific servers affected were the Shopping Cart database servers that handle cart contents for each shopper. New servers were added to the Shopping Cart database during our pre-holiday planning to manage the projected increase in transaction volume. When the holiday traffic surged, the hardware and software settings caused these servers to experience delays when customers added items to the Shopping Cart to complete checkout. When delays exceeded time limits, customers saw error messages stating that the servers had timed out (were not responding).

On Thursday, the service interruption occurred between 10:13 a.m. and 10:25 a.m. PT. During this time, merchants’ sites could not be accessed by shoppers. After investigating the issue, we found that this interruption occurred due to a network configuration issue resulting from new server capacity that was added for the holiday shopping season.

Immediately upon identification of both of these issues, our team worked hard to improve the situation as quickly as possible, and systems are now performing as expected.

These service interruptions were unrelated to our October feature release.

We fixed the problems as quickly as possible, and we are continuing to make improvements to our systems, including the addition of new hardware and system maintenance conducted on November 28. We will continue to post information about ongoing activities at our system status page.

We have apologized to our merchant partners for the impact that this service interruption had on their business and their customers at this busy time of year. As a token of our commitment to them, we have communicated with them our intent to waive all sales transaction fees for the month of November.

The success of our merchants’ business is extremely important to us, and we are committed to work as hard as we can, over and above the efforts already underway, to avoid such issues in the future.

Rich Riley
SVP, Online Channel Division

By now you have probably heard about the problems that many of our small business merchant customers experienced yesterday. Unfortunately, the system outage occurred at one of the worst possible times, and despite our concerted efforts to fix the problem as it emerged on Monday, we know that we let our merchant partners and their customers down. The good news is that our systems are now operating normally, and our merchants are able to accept orders from their customers.

Here’s what happened:

• On Monday at 6:00AM PT, the systems that power our merchant stores experienced outages, and shoppers of those stores were met with either error messages or they were unable to complete the checkout process.
• These issues lasted until about 1:00PM PT when, despite slow performance, transactions began going through at a much higher rate.
• By 6:00 PM PT things were back to normal and the performance of our systems was at 100%.

We deeply regret the inconvenience this caused to both our merchants and their shoppers. Our customers’ expectations were not met, nor were our own. And we are moving mountains inside Yahoo! to find out why and how this happened, and to take steps to try to ensure it doesn’t happen again.

As for the future, rest assured that we are taking the necessary steps to prepare for the peak holiday selling season. We have technical and customer relations staff mobilized and ready to support our partners.

At Yahoo! Small Business, we know that our success and our customers’ success are interdependent, and yesterday’s issue reminds us that we need to continue to work even harder in the future.

Rich Riley
SVP, Online Channel Division, Yahoo!