I reported early on that the cause of the Sidekick disaster was a SAN upgrade that failed without proper back-ups in place. (see also: More Details On What Caused the Sidekick Disaster) Daniel Eran Dilger, who has chimed in several times on this situation, has confirmed this through another source as well. He has an excellent article detailing many of the factors involved in all of this mess. If you have the time, I recommend reading the article.

But for those that don’t have time, here’s a quick summary:
Why weren’t there backups?
– Confirmation from a source that the loss of data was caused by stopping a 6-day backup by order of Microsoft / Roz Ho against the recommendations of Danger engineers. Because the backup was started, they had to remove a backup from a couple months ago in order to make space available. So they were left with an incomplete backup, since it was only run for 2 of the 6 days necessary. The SAN upgrade proceeded and things apparently went wrong.

No big deal, Microsoft says they recoved “most if not all” of our data
– Daniel Eran Dilger is skeptical that this is actually the case. He believes that it is possibly just Microsoft in denial. He writes:

If the company has stumbled upon a novel recovery avenue or some unknown backup that somehow remained missing for nearly two weeks, then this is great news for Sidekick users and helps to wipe some of the egg from the company’s cloud computing services face, although the situation still remains as the worst datacenter failure to ever impact mobile users as well as one of the most absurd responses pertaining to lost data as well.
…
However, Microsoft is also well known for advertising bullshit it can’t deliver.
…
f Microsoft strings along users long enough, it will be able to pat itself on the back with a “mission accomplished” even if it ultimately never actually delivered anything. It’s like saying you’ll call somebody back after a date and then just waiting until they figure out that you’re not really interested. After two weeks, the party on the other end begins blaming itself for waiting around.

Final thoughts
I’m paraphrasing here:
– If Microsoft can deliver even most of most users’ data, that’s awesome but doesn’t make up for the fact this happened in the first place.
– If Microsoft can’t deliver the data and this is just public relations BS to keep the negative press away until mainstream media forgets about the whole issue, and things fade into the background, then this is ridiculous.

Further, with this announcement, even if the company has no real data to recover, it will have erected a plausible story for denying anything significant ever happened. Know somebody who actually lost their important Sidekick data? You’ll be able to write them off as “one of the few who didn’t benefit from Microsoft’s miraculous data recovery.” It will be their word against Microsoft’s PR. Nobody will have records of who was impacted and whose data was recovered apart from Microsoft and probably T-Mobile, and the provider will likely have its records sealed by court order when it gets its big SLA settlement from Microsoft.

His closing paragraph makes it very clear that we need to keep this issue documented and not let Microsoft weasel their way out of another situation. I’ve opened up SKFail.com as a forum for this. Please head over there and leave posts detailing your experience, whether you lost data, if you’ve had it restored, etc.

What started out as “the worst loss of consumer data” will probably now be called “Microsoft’s Worst PR Disaster”.

An update from Microsoft’s own Roz Ho was posted early this morning on the T-Mobile Sidekick forum. She says that they have recovered “most, if not all, customer data” and that it will all soon be restored according to plan. She also confirms that it was indeed a system failure that wiped out the database and back-ups.

So here’s hoping that the Sidekick Disaster will soon be over and everything will be back to normal. I’m still wondering confirmation of the technical details on how this all happened and why Microsoft was so quick to say that all the data was “almost certainly has been lost” early on. Oh and one last handy tip to Microsoft/Danger: Make an application so that Sidekick users can ACTUALLY backup their data themselves.

The full post from Microsoft:

Updated: 10/15/2009 1:00 AM PDT

Microsoft Confirms Data Recovery for Sidekick Users

Data Restoration to Begin as Soon as Possible for Affected Customers

Dear T-Mobile Sidekick customers,

On behalf of Microsoft, I want to apologize for the recent problems with the Sidekick service and give you an update on the steps we have taken to resolve these problems.

We are pleased to report that we have recovered most, if not all, customer data for those Sidekick customers whose data was affected by the recent outage. We plan to begin restoring users’ personal data as soon as possible, starting with personal contacts, after we have validated the data and our restoration plan. We will then continue to work around the clock to restore data to all affected users, including calendar, notes, tasks, photographs and high scores, as quickly as possible.

We now believe that data loss affected a minority of Sidekick users. If your Sidekick account was among those affected, please continue to log into these forums for the latest updates about when data restoration will begin, and any steps you may need to take. We will work with T-Mobile to post the next update on data restoration timing no later than Saturday.

We have determined that the outage was caused by a system failure that created data loss in the core database and the back-up. We rebuilt the system component by component, recovering data along the way. This careful process has taken a significant amount of time, but was necessary to preserve the integrity of the data.

We will continue working closely with T-Mobile to restore user data as quickly as possible. We are eager to deliver the level of reliable service that our incredibly loyal customers have become accustomed to, and we are taking immediate steps to help ensure this does not happen again. Specifically, we have made changes to improve the overall stability of the Sidekick Service and initiated a more resilient backup process to ensure that the integrity of our database backups is maintained.

Once again, we apologize for this situation and the inconvenience that it has created. Please know that we are working all-out to resolve this situation and restore the reliability of the service.

Sincerely,
Roz Ho
Corporate Vice President
Premium Mobile Experiences, Microsoft Corporation

We all knew it wouldn’t be long before lawsuits were filed over the Sidekick outage and data loss. As Microsoft/Danger still struggles to restore data and get things back up and stable, people are already filing lawsuits, “claiming negligence and false claims.”

A suit filed for a Bakersfield, CA man and “all others similarly situated” says that Danger failed to handle Sidekick user’s data and that they advertised in a misleading manner. He’s asking for monetary damages as well as the court to order Microsoft to fix the Sidekick service or offer a full refund. The attorney handling the case was quoted as saying: “We are hopeful that T-Mobile and the rest of the defendants will do the right thing, use this as an opportunity to redesign the system as a new standard for cloud computing storage, and provide full compensation for the data loss.”

Another class action law suit (PDF of filing) was filed for Maureen Thompson and again “all others similarly situated” against T-Mobile/Danger/Microsoft for the outage and loss of data. Same sort of thing.

And there’s yet another suit (PDF) filed by Oren Rosenthal against T-Mobile for the negligence, breach of contract, blah blah blah.

Should be interesting to see how these play out. If you hear of any others, let us know over on skfail.com‘s Lawsuit forum.

image via kcoury

Want more geeky details on what happened at Microsoft/Danger? The short of it is that the SAN took a nose-dive and took out the drives that could have repaired the data with it. A total of 800TB of data was lost. There was an off-site tape backup, a reasonable backup measure, but with the way the SAN died the entire RAID array needed rebuilding and 800TB of data is a lot of data to rebuild. There are more details on server moves in the info below.

The following is reportedly from “someone close to the action”:

“Here’s the actual scoop, from someone involved in the recovery:

Danger, purchased by Microsoft, was moved into a Verizon Business datacenter in Kent, WA a short while ago. While this had to do with the MS assimilation, it was done as a one for one move from Danger to a DC that MS uses heavily. (MS didn’t re-write, port, migrate to winblows, etc.) The backend service uses a variety of hardware, load balancers, firewalls, web and application servers, and an EMC SAN (Storage Area Network, think huge drive array connected with fiber.)

Well last Tuesday, the EMC SAN took a dump on itself. What I mean by that is the backplane let the magic blue smoke out. While usually in the heavy iron class of datacenter products like an EMC SAN this means you fail over to the redundant backplane and life continues on. Not this time folks. In the process of dying, it took out the parity drives. What does that mean? It means the fancy RAID lost it’s ability to actually be a RAID. How much data got eaten by this mega-oops? 800TB. Why wasn’t it backed up? It was, to offsite tape, like it’s supposed to. But when the array is toast, can’t just start copying shit back.

Apparently EMC has been on site since Tuesday, but didn’t actually inform Danger/MS that their data is in the crapper until Friday afternoon. On top of that, EMC has done nothing to bring in replacement equipment between Tuesday and Friday. (In the Enterprise support world, that’s fucking retarded, multi-million dollar support contracts are that expensive for a reason.)

So what’s being done? Well the good news is that the complex was slated to be migrated into the Verizon Business cloud services (not MS’s cloud per se, but it’s MS’s effort.) And as a part of that migration a newer shinier SAN array was in process of being implemented. But space isn’t ready for it on the datacenter floor, and you can’t just toss the EMC raid and place this one in it’s place, it’s a different vendor and is 2 racks instead of one. This means it’s being shoehorned into a different part of the datacenter than was originally planned, one that doesn’t have the necessary 3 phase power installed. So there’s a bit of work to be done. Not to mention the restoral of 800TB of backup data from offsite tape.

Time to restoral? Looking like Wednesday at the earliest with techs working all weekend.”

Sounds like they know what they’re talking about, but since we haven’t been able to confirm this directly ourselves, we’re keeping it labeled as a rumor.

UPDATE (2009-10-15 01:02 PST): We’ve confirmed that Danger does indeed have servers in a Verizon Business Data Center, however it appears to be one in California, NOT Kent, WA. If you want to confirm, do a traceroute on one of Danger’s web proxies and you’ll find it ends up at danger-gw.customer.alter.net (157.130.202.122), an IP owned by Verizon (MCI) that appears to be around the San Jose/Santa Clara area. It’s possible (although unlikely) that the web proxy servers are kept separate from the user data servers though.

T-Mobile reports:

Microsoft/Danger have now made the necessary fixes to their network to restore the ability to sync your data. This means that your contacts, calendar entries, to-do lists, tasks, etc. will now sync on the network, just like they did prior to this data disruption. However, you must power cycle your device following the steps below in order to begin the synching process.

But they do add the discaimer that things are still “unstable” and that you should back up all your data. Note that a power cycle is going to Menu -> Power Off, letting your Sidekick turn off on its own and then powering it back up. Do NOT do a hard reset as your data will be lost.

After hearing reports about people having posts deleted and accounts banned on the official T-Mobile Sidekick forum and on PoweredByDanger.com there was a pressing need to have a completely uncensored forum where users can discuss how this Sidekick disaster has impacted them and the experiences they’ve had with T-Mobile’s customer service reps. Enter the appropriately named Skfail.com.

We want to collect user’s experiences as well as news articles, statements from T-Mobile, etc. in one place that people can go. So head on over to the forums and speak up!

T-Mobile released their Monday statement. The short of it? If you lost your data, you’re getting $100. The long of it? Read below:

Updated: 10/12/2009 5:15 PM PDT
T-MOBILE STATUS UPDATE ON SIDEKICK DATA DISRUPTION, MON., OCT. 12

Dear valued T-Mobile Sidekick customers:

We are thankful for your continued patience as Microsoft/Danger continues to work on preserving platform stability and restoring all services for our Sidekick customers. We have made significant progress this past weekend, restoring services to virtually every customer. Microsoft/Danger has teams of experts in place who are working around-the-clock to ensure this stability is maintained.

Regarding those of you who have lost personal content, T-Mobile and Microsoft/Danger continue to do all we can to recover and return any lost information. Recent efforts indicate the prospects of recovering some lost content may now be possible. We will continue to keep you updated on this front; we know how important this is to you.

In the event certain customers have experienced a significant and permanent loss of personal content, T-Mobile will be sending these customers a $100 customer appreciation card. This will be in addition to the free month of data service that already went to Sidekick data customers. This card can be used towards T-Mobile products and services, or a customer’s T-Mobile bill. For those who fall into this category, details will be sent out in the next 14 days – there is no action needed on the part of these customers. We however remain hopeful that for the majority of our customers, personal content can be recovered.

via the Sidekick Forums

The interesting point is that T-Mobile knows who to send these $100 customer appreciation cards to. That means they know who lost data, which means they know what data was lost on the server. So the hope is that they were able to recover some user’s data and it wasn’t totally lost.

The plot thickens. There’s an extensive article over on AppleInsider that lays out more details regarding the possibles motivations responsible for the Danger Disaster at Microsoft. There are two theories: dogfooding and sabotage.

Dogfooding is a reference to the term “eating one’s own dogfood” or replacing competitor’s technology with your own for internal uses. AppleInsider’s insider source says

Danger’s Sidekick data center had “been running on autopilot for some time, so I don’t understand why they would be spending any time upgrading stuff unless there was a hardware failure of some kind,” wrote the insider. Given Microsoft’s penchant for “for running the latest and greatest,” however, “I wouldn’t be surprised if they found out that [storage vendor] EMC had some new SAN firmware and they just had to put it on the main production servers right away.”

But as the article points out, what could have been the motivation for Microsoft to change technologies when they were already scuttling the Danger product in the attempts to save the Pink Project? Microsoft was only running things as they were bound by contract with T-Mobile. The article goes on to say:

Instead, the fact that no data could be recovered after the problem erupted at the beginning of October suggests that the outage and the inability to recover any backups were the result of intentional sabotage by a disgruntled employee. In any other circumstance, Microsoft or T-Mobile would likely have come forward with an explanation of the mitigating circumstances, blaming bad hardware, a power failure, or some freak accident.

An act of sabotage “would explain why neither party is releasing any more details: for legal reasons dealing with the ongoing investigation to find the culprit(s),” one of the sources said. Due to the way Sidekick clients interact with the service, any normal failure should have resulted in only a brief outage until a replacement server could be brought up.

So, was it sabotage? I’ve been saying all along that the level of this screw up is so high you start to think that someone has to be intentionally screwing up this much. Whether that’s actually sabotage or just Microsoft’s apathy and ineptitude is unknown at this point.

On top of all this, AppleInsider points out the obvious: T-Mobile must be irate with Microsoft right about now. Surely in their contract there was a Service Level Agreement (SLA) that dictated a certain level of uptime. And with almost two weeks of outage plus an enormous loss of data this “is the worst possible violation of the SLA conceivable”. I’m just waiting for the lawsuits between T-Mobile and Microsoft/Danger to start.

We’re still waiting for the official word from Microsoft/Danger that they promised us “on Monday”, but we’re hearing a few people report that they magically got their data back. Posts over on the official T-Mobile forum (and an uncensorable thread started on the new skfail.com) are giving us some hope that maybe all this data isn’t truly lost.

tommyd2107 says:

I was just on my phone and when I got off my phone all my contacts returned. I do not know if this will last for long but the site of my contacts returning is encouraging.

dariahna says:

The same thing happened to me…I shut my phone off several times without removing the battery…voile’! My contacts returned!

generalblue says:

Got my contacts back too! My phone froze up on me and since I have already lost everything anyways, I just took the battery out and put it back in. Once my phone was on about 5 minutes later I checked my address book for some reason and they were all there. I saved all my contacts to my simcard.

Anyone else out there have any luck?

Another tip that is being reported on poweredbydanger.com that is unofficial, not 100% tested, try at your own risk, might just work, etc. is if you have an old device that has your contacts and info you might be able to do a reverse sync (i.e. device to server). Make sure your old device is turned off, plug it into the wall, remove the battery, insert your SIM, replace the battery, power up, and cross your fingers. With any luck it’ll update Danger’s servers with this info, and then you can switch back to your newer Sidekick which will pull it back down. This is at your own risk and should only be tried after you’ve backed up all the contents on that old device. If you do try this, let us know if it works for you.

We’re hearing a number of reports from people that T-Mobile is shouldering the brunt of this disaster (which is truly Microsoft’s fault) by letting customers upgrade to new phones. The reports in our comments are mixed, but it seems like some people are getting a new G1 or a new MyTouch 3G and are paying a discounted rate or even getting it free. What’s more important is that we’re hearing many of these customers are being given the data plan at the $19.99 rate many of us are grandfathered in for with the Sidekick. There are also reports that people are being let out of their contract if they have a Sidekick, but we’re also hearing that they may not be allowed to do that anymore. I guess it depends on who you talk to and when as I’m sure things at T-Mobile are changing rapidly.

I’m holding out hope, but if you’re calling T-Mobile to switch, let us know your experience!

UPDATE: “Free” seems to be more of a rumor than truth. It looks like the norm for most people is that they are getting to upgrade to a phone at the upgrade price you see on T-Mobile.com ($129.99 for a G1, $149.99 for a MyTouch, etc). You’ll probably be required to sign a contract, but should be able to keep the $19.99 data plan. This is even for people that had just renewed their contract with a purchase of the Sidekick LX 2009. Not sure if people out of contract are getting any special treatment on top of this to keep them as a T-Mobile customer.