We all knew it wouldn’t be long before lawsuits were filed over the Sidekick outage and data loss. As Microsoft/Danger still struggles to restore data and get things back up and stable, people are already filing lawsuits, “claiming negligence and false claims.”

A suit filed for a Bakersfield, CA man and “all others similarly situated” says that Danger failed to handle Sidekick user’s data and that they advertised in a misleading manner. He’s asking for monetary damages as well as the court to order Microsoft to fix the Sidekick service or offer a full refund. The attorney handling the case was quoted as saying: “We are hopeful that T-Mobile and the rest of the defendants will do the right thing, use this as an opportunity to redesign the system as a new standard for cloud computing storage, and provide full compensation for the data loss.”

Another class action law suit (PDF of filing) was filed for Maureen Thompson and again “all others similarly situated” against T-Mobile/Danger/Microsoft for the outage and loss of data. Same sort of thing.

And there’s yet another suit (PDF) filed by Oren Rosenthal against T-Mobile for the negligence, breach of contract, blah blah blah.

Should be interesting to see how these play out. If you hear of any others, let us know over on skfail.com’s Lawsuit forum.

image via kcoury

Want more geeky details on what happened at Microsoft/Danger? The short of it is that the SAN took a nose-dive and took out the drives that could have repaired the data with it. A total of 800TB of data was lost. There was an off-site tape backup, a reasonable backup measure, but with the way the SAN died the entire RAID array needed rebuilding and 800TB of data is a lot of data to rebuild. There are more details on server moves in the info below.

The following is reportedly from “someone close to the action”:

“Here’s the actual scoop, from someone involved in the recovery:

Danger, purchased by Microsoft, was moved into a Verizon Business datacenter in Kent, WA a short while ago. While this had to do with the MS assimilation, it was done as a one for one move from Danger to a DC that MS uses heavily. (MS didn’t re-write, port, migrate to winblows, etc.) The backend service uses a variety of hardware, load balancers, firewalls, web and application servers, and an EMC SAN (Storage Area Network, think huge drive array connected with fiber.)

Well last Tuesday, the EMC SAN took a dump on itself. What I mean by that is the backplane let the magic blue smoke out. While usually in the heavy iron class of datacenter products like an EMC SAN this means you fail over to the redundant backplane and life continues on. Not this time folks. In the process of dying, it took out the parity drives. What does that mean? It means the fancy RAID lost it’s ability to actually be a RAID. How much data got eaten by this mega-oops? 800TB. Why wasn’t it backed up? It was, to offsite tape, like it’s supposed to. But when the array is toast, can’t just start copying shit back.

Apparently EMC has been on site since Tuesday, but didn’t actually inform Danger/MS that their data is in the crapper until Friday afternoon. On top of that, EMC has done nothing to bring in replacement equipment between Tuesday and Friday. (In the Enterprise support world, that’s fucking retarded, multi-million dollar support contracts are that expensive for a reason.)

So what’s being done? Well the good news is that the complex was slated to be migrated into the Verizon Business cloud services (not MS’s cloud per se, but it’s MS’s effort.) And as a part of that migration a newer shinier SAN array was in process of being implemented. But space isn’t ready for it on the datacenter floor, and you can’t just toss the EMC raid and place this one in it’s place, it’s a different vendor and is 2 racks instead of one. This means it’s being shoehorned into a different part of the datacenter than was originally planned, one that doesn’t have the necessary 3 phase power installed. So there’s a bit of work to be done. Not to mention the restoral of 800TB of backup data from offsite tape.

Time to restoral? Looking like Wednesday at the earliest with techs working all weekend.”

Sounds like they know what they’re talking about, but since we haven’t been able to confirm this directly ourselves, we’re keeping it labeled as a rumor.

UPDATE (2009-10-15 01:02 PST): We’ve confirmed that Danger does indeed have servers in a Verizon Business Data Center, however it appears to be one in California, NOT Kent, WA. If you want to confirm, do a traceroute on one of Danger’s web proxies and you’ll find it ends up at danger-gw.customer.alter.net (157.130.202.122), an IP owned by Verizon (MCI) that appears to be around the San Jose/Santa Clara area. It’s possible (although unlikely) that the web proxy servers are kept separate from the user data servers though.

T-Mobile reports:

Microsoft/Danger have now made the necessary fixes to their network to restore the ability to sync your data. This means that your contacts, calendar entries, to-do lists, tasks, etc. will now sync on the network, just like they did prior to this data disruption. However, you must power cycle your device following the steps below in order to begin the synching process.

But they do add the discaimer that things are still “unstable” and that you should back up all your data. Note that a power cycle is going to Menu -> Power Off, letting your Sidekick turn off on its own and then powering it back up. Do NOT do a hard reset as your data will be lost.

After hearing reports about people having posts deleted and accounts banned on the official T-Mobile Sidekick forum and on PoweredByDanger.com there was a pressing need to have a completely uncensored forum where users can discuss how this Sidekick disaster has impacted them and the experiences they’ve had with T-Mobile’s customer service reps. Enter the appropriately named Skfail.com.

We want to collect user’s experiences as well as news articles, statements from T-Mobile, etc. in one place that people can go. So head on over to the forums and speak up!

T-Mobile released their Monday statement. The short of it? If you lost your data, you’re getting $100. The long of it? Read below:

Updated: 10/12/2009 5:15 PM PDT
T-MOBILE STATUS UPDATE ON SIDEKICK DATA DISRUPTION, MON., OCT. 12

Dear valued T-Mobile Sidekick customers:

We are thankful for your continued patience as Microsoft/Danger continues to work on preserving platform stability and restoring all services for our Sidekick customers. We have made significant progress this past weekend, restoring services to virtually every customer. Microsoft/Danger has teams of experts in place who are working around-the-clock to ensure this stability is maintained.

Regarding those of you who have lost personal content, T-Mobile and Microsoft/Danger continue to do all we can to recover and return any lost information. Recent efforts indicate the prospects of recovering some lost content may now be possible. We will continue to keep you updated on this front; we know how important this is to you.

In the event certain customers have experienced a significant and permanent loss of personal content, T-Mobile will be sending these customers a $100 customer appreciation card. This will be in addition to the free month of data service that already went to Sidekick data customers. This card can be used towards T-Mobile products and services, or a customer’s T-Mobile bill. For those who fall into this category, details will be sent out in the next 14 days – there is no action needed on the part of these customers. We however remain hopeful that for the majority of our customers, personal content can be recovered.

via the Sidekick Forums

The interesting point is that T-Mobile knows who to send these $100 customer appreciation cards to. That means they know who lost data, which means they know what data was lost on the server. So the hope is that they were able to recover some user’s data and it wasn’t totally lost.

The plot thickens. There’s an extensive article over on AppleInsider that lays out more details regarding the possibles motivations responsible for the Danger Disaster at Microsoft. There are two theories: dogfooding and sabotage.

Dogfooding is a reference to the term “eating one’s own dogfood” or replacing competitor’s technology with your own for internal uses. AppleInsider’s insider source says

Danger’s Sidekick data center had “been running on autopilot for some time, so I don’t understand why they would be spending any time upgrading stuff unless there was a hardware failure of some kind,” wrote the insider. Given Microsoft’s penchant for “for running the latest and greatest,” however, “I wouldn’t be surprised if they found out that [storage vendor] EMC had some new SAN firmware and they just had to put it on the main production servers right away.”

But as the article points out, what could have been the motivation for Microsoft to change technologies when they were already scuttling the Danger product in the attempts to save the Pink Project? Microsoft was only running things as they were bound by contract with T-Mobile. The article goes on to say:

Instead, the fact that no data could be recovered after the problem erupted at the beginning of October suggests that the outage and the inability to recover any backups were the result of intentional sabotage by a disgruntled employee. In any other circumstance, Microsoft or T-Mobile would likely have come forward with an explanation of the mitigating circumstances, blaming bad hardware, a power failure, or some freak accident.

An act of sabotage “would explain why neither party is releasing any more details: for legal reasons dealing with the ongoing investigation to find the culprit(s),” one of the sources said. Due to the way Sidekick clients interact with the service, any normal failure should have resulted in only a brief outage until a replacement server could be brought up.

So, was it sabotage? I’ve been saying all along that the level of this screw up is so high you start to think that someone has to be intentionally screwing up this much. Whether that’s actually sabotage or just Microsoft’s apathy and ineptitude is unknown at this point.

On top of all this, AppleInsider points out the obvious: T-Mobile must be irate with Microsoft right about now. Surely in their contract there was a Service Level Agreement (SLA) that dictated a certain level of uptime. And with almost two weeks of outage plus an enormous loss of data this “is the worst possible violation of the SLA conceivable”. I’m just waiting for the lawsuits between T-Mobile and Microsoft/Danger to start.

We’re still waiting for the official word from Microsoft/Danger that they promised us “on Monday”, but we’re hearing a few people report that they magically got their data back. Posts over on the official T-Mobile forum (and an uncensorable thread started on the new skfail.com) are giving us some hope that maybe all this data isn’t truly lost.

tommyd2107 says:

I was just on my phone and when I got off my phone all my contacts returned. I do not know if this will last for long but the site of my contacts returning is encouraging.

dariahna says:

The same thing happened to me…I shut my phone off several times without removing the battery…voile’! My contacts returned!

generalblue says:

Got my contacts back too! My phone froze up on me and since I have already lost everything anyways, I just took the battery out and put it back in. Once my phone was on about 5 minutes later I checked my address book for some reason and they were all there. I saved all my contacts to my simcard.

Anyone else out there have any luck?

Another tip that is being reported on poweredbydanger.com that is unofficial, not 100% tested, try at your own risk, might just work, etc. is if you have an old device that has your contacts and info you might be able to do a reverse sync (i.e. device to server). Make sure your old device is turned off, plug it into the wall, remove the battery, insert your SIM, replace the battery, power up, and cross your fingers. With any luck it’ll update Danger’s servers with this info, and then you can switch back to your newer Sidekick which will pull it back down. This is at your own risk and should only be tried after you’ve backed up all the contents on that old device. If you do try this, let us know if it works for you.

We’re hearing a number of reports from people that T-Mobile is shouldering the brunt of this disaster (which is truly Microsoft’s fault) by letting customers upgrade to new phones. The reports in our comments are mixed, but it seems like some people are getting a new G1 or a new MyTouch 3G and are paying a discounted rate or even getting it free. What’s more important is that we’re hearing many of these customers are being given the data plan at the $19.99 rate many of us are grandfathered in for with the Sidekick. There are also reports that people are being let out of their contract if they have a Sidekick, but we’re also hearing that they may not be allowed to do that anymore. I guess it depends on who you talk to and when as I’m sure things at T-Mobile are changing rapidly.

I’m holding out hope, but if you’re calling T-Mobile to switch, let us know your experience!

UPDATE: “Free” seems to be more of a rumor than truth. It looks like the norm for most people is that they are getting to upgrade to a phone at the upgrade price you see on T-Mobile.com ($129.99 for a G1, $149.99 for a MyTouch, etc). You’ll probably be required to sign a contract, but should be able to keep the $19.99 data plan. This is even for people that had just renewed their contract with a purchase of the Sidekick LX 2009. Not sure if people out of contract are getting any special treatment on top of this to keep them as a T-Mobile customer.

By now the word is out on the street. Microsoft/Danger has most likely lost everyone’s personal info including contacts, notes, calendar entries, to-dos, etc. The question remains: How did this happen? Microsoft is a big software company, they’re well versed in the enterprise world and should have systems in place that allow them to weather any sort of issue like this. Of course everyone (T-Mobile, Microsoft/Danger) hasn’t come out with any details on the cause of the failure, but we’ve got some theories and rumors floating around.

Currently the rumor with the most weight is as follows:
Microsoft was upgrading their SAN (Storage Area Network aka the thing that stores all your data) and had hired Hitachi to come in and do it for them. Typically in an upgrade like this, you are expected to make backups of your SAN before the upgrade happens. Microsoft failed to make these backups for some reason. We’re not sure if it was because of the amount of data that would be required, if they didn’t have time to do it, or if they simply forgot. Regardless of why, Microsoft should know better. So Hitachi worked on upgrading the SAN and something went wrong, resulting in its destruction. Currently the plan is to try to get the devices that still have personal data on them to sync back to the servers and at least keep the data that users have on their device saved.

We’ve heard this from what appears to be several sources and it seems to hold weight. Needless to say it all boils down to one thing: Microsoft did not have a working backup.

How this happens in today’s day and age is beyond belief. Hundreds of thousands of customers that generate millions of dollars in revenue means you back their stuff up, in triplicate. You test these backups regularly, and you move a copy off site that doesn’t get touched except in case of an emergency (i.e. right now). The head of the mobile division (and person in charge of what’s left of Danger) is Roz Ho, who has been at Microsoft for 18 years. You would think she’d know something about how to run a business.

What does this mean for the future of the Sidekick? Unless Microsoft pulls a miracle out of thin air the Sidekick is dead. People are already jumping ship to other phones with this news, and the exposure of how inept Microsoft is when it comes to the mobile world is huge. If Microsoft can’t continue to run Danger, a company that was ground-breaking and solidly built, how can we expect anything from the Windows Mobile department?

If you haven’t lost your contacts from a hard reset, start copying them to something else right now. Microsoft has officially announced that your contacts, calendar, and other data has “almost certainly has been lost as a result of a server failure at Microsoft/Danger”.

Sidekick customers, during this service disruption, please DO NOT remove your battery, reset your Sidekick, or allow it to lose power.

Updated: 10/10/2009 12:35 PM PDT

T-MOBILE AND MICROSOFT/DANGER STATUS UPDATE ON SIDEKICK DATA DISRUPTION

Dear valued T-Mobile Sidekick customers:

T-Mobile and the Sidekick data services provider, Danger, a subsidiary of Microsoft, are reaching out to express our apologies regarding the recent Sidekick data service disruption.

We appreciate your patience as Microsoft/Danger continues to work on maintaining platform stability, and restoring all services for our Sidekick customers.

Regrettably, based on Microsoft/Danger’s latest recovery assessment of their systems, we must now inform you that personal information stored on your device – such as contacts, calendar entries, to-do lists or photos – that is no longer on your Sidekick almost certainly has been lost as a result of a server failure at Microsoft/Danger. That said, our teams continue to work around-the-clock in hopes of discovering some way to recover this information. However, the likelihood of a successful outcome is extremely low. As such, we wanted to share this news with you and offer some tips and suggestions to help you rebuild your personal content. You can find these tips in our Sidekick Contacts FAQ. We encourage you to visit the Forums on a regular basis to access the latest updates as well as FAQs regarding this service disruption.

In addition, we plan to communicate with you on Monday (Oct. 12) the status of the remaining issues caused by the service disruption, including the data recovery efforts and the Download Catalog restoration which we are continuing to resolve. We also will communicate any additional tips or suggestions that may help in restoring your content.

We recognize the magnitude of this inconvenience. Our primary efforts have been focused on restoring our customers’ personal content. We also are considering additional measures for those of you who have lost your content to help reinforce how valuable you are as a T-Mobile customer.

We continue to advise customers to NOT reset their device by removing the battery or letting their battery drain completely, as any personal content that currently resides on your device will be lost.

Once again, T-Mobile and Microsoft/Danger regret any and all inconvenience this matter has caused.

I am speechless.

Original post