Looks like the last thing T-Mobile needs right now is happening. I’m hearing reports from several people that there is a nationwide outage of sorts for T-Mobile right now. It doesn’t affect outgoing calls to non-T-Mobile phones, but if someone is trying to call you they’re probably just getting a busy signal right now. So far this is all we know, we’ll let you know if anything changes. Hopefully this will be short-lived.

UPDATE 4:30pm Pacific: Just got off the phone with T-Mobile, they’re acknowledging the outage and say that it is sporadic as far as what it is affecting. Texting, voice, and even internet was mentioned. She also mentioned that rumor had it that other carriers were experiencing similar problems right now but couldn’t name them. Their engineers are on it and it shouldn’t last long they say.

Chris posted the following in a comment, but I thought it was worth highlighting in a dedicated post. His letter touches on some of the feelings a lot of us long-time Sidekick lovers are feeling I think:

Dear T-Mobile/Danger:

My sentiments as a long-time subscriber:

T-Mobile:

I’ve had your services since your spokesperson was a parrot and your name was Omnipoint. I even sold them as an Authorized Dealer when the pager business died. It was 1998, and you only offered a handful of brick-sized phones (remember the Ericsson 388?), and your flagship handset had a 3-color screen and was made by Siemens. I remember using prepaid coupons to pay my postpaid bill – Genius! Your network was small, your plans were so-so, but digital was superior, and you were my first choice.

You became Voicestream, and had a fancy, stylized logo and Jamie-Lee Curtis was your spokesperson. She gave James Earl Jones a run for his money and you began to look like a major player. There were plenty of companies to choose from, but I stayed faithful. I used all the newest handsets and loved the ease of switching my SIM, allowing me to try all the newest phones. My Nokia 3300 fell 100 times but never failed me. I would just put the shell back together, and I had all the colors. I composed my own ringers painstakingly from the composer before it became a billion-dollar business. AT&T was still TDMA back then, and you were the only GSM player for a while (It still amazes me how you let Cingular/Big Blue leapfrog you with your own technology, but I digress).

You gobbled up smaller carriers left and right, and soon you got your new name. T-Mobile. Your coverage maps still looked like daddy-long-legs, but with each revision, the pink spiders gained weight. You were determined to provide call quality over tower quantity, knowing the latter would come soon enough. I always had enjoyable experiences as a customer and got my whole family under your wing.

Flash forward to 2002. I had a two-way pager from Skytel, and life was good. I could actually check email without a COMPUTER! My last regular handset was a Motorola V70, a slick-looking phone that rotated instead of flipping open. But I lusted for something that could do it all. I bought an unlocked Motorola Accompli 009 online to combine my 2-way pager with a phone, and also had a color screen. Mandatory headset required though, and Bluetooth didn’t exist yet. That didn’t last long. I looked at my sheet of upcoming phones and saw some chunky, grey, calculator-looking thing with a full keyboard. And the screen did some funky, flip move that snapped open like I always wished the V70 had done. Was this the device I had been looking for?

Danger:

Ahhh, what say I say to my dear old friend. I picked up your ugly duckling of a device and never looked back. I had never been brand-loyal with handset manufacturers, but boy, did that change. In my entire history as a wireless customer, no device has matched the ingenuity of something so useful as the Jump button. I’ve multi-tasked with ease, and frequented the hiptop.com (later, poweredbydanger.com) forums daily anticipating each OTA update. I remember when AIM smileys and copy/paste were added. I posted so often under the name JustifydHomicide that I was a “Power User”. Appletech, The Gryphon, mwsmith, JHC – These people were the gurus of my Sidekick experience. I got a developer’s key and loved testing applications, learning the Menu+Shift shortcuts like the back of my hand.

You were always so ahead of the curve, single-handedly creating the catalog model for which Apple smugly gets credit, and for which all the other OS companies are still clamoring. I felt that I was using the latest and greatest technology, for a while that is. I anticipated each new hardware launch and recall ordering the Sidekick 2 at midnight of the launch. I’ve done the same for each new device, and even sprung for the Mr. Cartoon and LRG LE’s. I remained positive after your acquisition by Microsoft, even after they unceremoniously fired half of your staff, anxiously hoping that someday I’d see the convergence of the Sidekick with another favorite, the Zune.

When the G1 came out, I resisted temptation and waited to upgrade to the LX09 even though 3G hadn’t launched in my area yet. I was grandfathered into your $20 data, and was willing to forgo the latest handset and stay loyal to you. For the first time, however, I was thoroughly unimpressed by your new outfit. Sure, your new dress was sexy and sleek, but the new integration of social networking apps lacked functionality and slowed my typically reliable OS to a crawl. I became plagued with frequent resets and freezing and now stay logged out of apps unless I need them, opting to use the browser for Twitter rather than the especially-limited catalog version. It soon became clear why this app and others were provided to the end user for free.

I began to fall out of love with you after gazing at superior products. I even flirted with a few, impressed by better features and ease of use. I developed an infatuation for Android like a middle middle-aged man staring at a young home-wrecker. The company whose ingenuity I had loved and been loyal to had changed like an estranged wife of 6 years. It felt like you lost your touch, and looking at your renders of the Project Pink phones had me drawing up my divorce papers.

Then, there was the straw that broke the camel’s back,

October 2nd, 2009:

You know the particulars, so I won’t cover old news. You know what you did. You cheated on me. I had a trust in you that I was sure you wouldn’t break. My secrets were safe with you throughout our relationship, and then you walked out on me in the form of a data outage. It had happened before, but never like this. I couldn’t reach you for over a week, and when I heard from you, I realized that it was worse than I thought. You didn’t just fail me, but you also destroyed everything we built together. Taking over 1000 contacts with you when you left was like emptying the joint account and taking the kids. You didn’t even leave the pictures as a memento. How cruel could you be?

Apparently you had a change of heart, restoring some of my contacts, although you haven’t yet made me whole. I’m still waiting for you to drop off the rest of my stuff – Do you think you can get the calendar, bookmarks, and photos back to me soon? I’m starting to think it’ll all work out in the end; I’m getting comfortable with the idea that we won’t be together much longer. I noticed you took yourself off the market on t-mobile.com, too. Decided not to play the field?

Android looks better every day, but I like my ladies with keyboards, and G1’s getting wrinkles every day. I check in on my new love interests N900 and Bold 2 daily, and can’t wait to choose my new wife. T-Mobile at least had enough Loyalty (Plan) to allow me to forgive them, but you I cannot. The credits and new handset they offered me make me feel a little better.

T-Mobile:

With your bold launch of Project Dark, this could not have come at a more inopportune time. I applaud your willingness to adopt a new model and buck the tradition that has plagued the wireless climate in the U.S. I will not be converting to one of these new plans, as my current plan suits me better. I do wonder two things about your no-contract Even More Plus plans:

Will handsets purchased at full retail come unlocked?

Will customer be allowed to provide their own unlocked handset and sign up for Even More Plus?

In closing, thank you T-Mobile for always having a “Think Customer First” attitude, a credo of another large and successful company. Your customer service is unparalleled, and your philosophy as a company influenced me to work for your corporation during my 9-year wireless career, before moving on to a different industry. In this situation, you didn’t hesitate to concede your faults for Microsoft/Danger’s shortcomings. Although you were only indirectly responsible, you realized that ultimately YOU collect my bill, and I respect that. I won’t disclose what concessions you’ve offered me to retain my business, but I do beleive that I will continue to a be satisfied customer some time in November. I hope Microsoft makes you whole for your loss of revenue due to this debacle.

Cellphone Chris

What started out as “the worst loss of consumer data” will probably now be called “Microsoft’s Worst PR Disaster”.

An update from Microsoft’s own Roz Ho was posted early this morning on the T-Mobile Sidekick forum. She says that they have recovered “most, if not all, customer data” and that it will all soon be restored according to plan. She also confirms that it was indeed a system failure that wiped out the database and back-ups.

So here’s hoping that the Sidekick Disaster will soon be over and everything will be back to normal. I’m still wondering confirmation of the technical details on how this all happened and why Microsoft was so quick to say that all the data was “almost certainly has been lost” early on. Oh and one last handy tip to Microsoft/Danger: Make an application so that Sidekick users can ACTUALLY backup their data themselves.

The full post from Microsoft:

Updated: 10/15/2009 1:00 AM PDT

Microsoft Confirms Data Recovery for Sidekick Users

Data Restoration to Begin as Soon as Possible for Affected Customers

Dear T-Mobile Sidekick customers,

On behalf of Microsoft, I want to apologize for the recent problems with the Sidekick service and give you an update on the steps we have taken to resolve these problems.

We are pleased to report that we have recovered most, if not all, customer data for those Sidekick customers whose data was affected by the recent outage. We plan to begin restoring users’ personal data as soon as possible, starting with personal contacts, after we have validated the data and our restoration plan. We will then continue to work around the clock to restore data to all affected users, including calendar, notes, tasks, photographs and high scores, as quickly as possible.

We now believe that data loss affected a minority of Sidekick users. If your Sidekick account was among those affected, please continue to log into these forums for the latest updates about when data restoration will begin, and any steps you may need to take. We will work with T-Mobile to post the next update on data restoration timing no later than Saturday.

We have determined that the outage was caused by a system failure that created data loss in the core database and the back-up. We rebuilt the system component by component, recovering data along the way. This careful process has taken a significant amount of time, but was necessary to preserve the integrity of the data.

We will continue working closely with T-Mobile to restore user data as quickly as possible. We are eager to deliver the level of reliable service that our incredibly loyal customers have become accustomed to, and we are taking immediate steps to help ensure this does not happen again. Specifically, we have made changes to improve the overall stability of the Sidekick Service and initiated a more resilient backup process to ensure that the integrity of our database backups is maintained.

Once again, we apologize for this situation and the inconvenience that it has created. Please know that we are working all-out to resolve this situation and restore the reliability of the service.

Sincerely,
Roz Ho
Corporate Vice President
Premium Mobile Experiences, Microsoft Corporation

We all knew it wouldn’t be long before lawsuits were filed over the Sidekick outage and data loss. As Microsoft/Danger still struggles to restore data and get things back up and stable, people are already filing lawsuits, “claiming negligence and false claims.”

A suit filed for a Bakersfield, CA man and “all others similarly situated” says that Danger failed to handle Sidekick user’s data and that they advertised in a misleading manner. He’s asking for monetary damages as well as the court to order Microsoft to fix the Sidekick service or offer a full refund. The attorney handling the case was quoted as saying: “We are hopeful that T-Mobile and the rest of the defendants will do the right thing, use this as an opportunity to redesign the system as a new standard for cloud computing storage, and provide full compensation for the data loss.”

Another class action law suit (PDF of filing) was filed for Maureen Thompson and again “all others similarly situated” against T-Mobile/Danger/Microsoft for the outage and loss of data. Same sort of thing.

And there’s yet another suit (PDF) filed by Oren Rosenthal against T-Mobile for the negligence, breach of contract, blah blah blah.

Should be interesting to see how these play out. If you hear of any others, let us know over on skfail.com‘s Lawsuit forum.

image via kcoury

T-Mobile reports:

Microsoft/Danger have now made the necessary fixes to their network to restore the ability to sync your data. This means that your contacts, calendar entries, to-do lists, tasks, etc. will now sync on the network, just like they did prior to this data disruption. However, you must power cycle your device following the steps below in order to begin the synching process.

But they do add the discaimer that things are still “unstable” and that you should back up all your data. Note that a power cycle is going to Menu -> Power Off, letting your Sidekick turn off on its own and then powering it back up. Do NOT do a hard reset as your data will be lost.

After hearing reports about people having posts deleted and accounts banned on the official T-Mobile Sidekick forum and on PoweredByDanger.com there was a pressing need to have a completely uncensored forum where users can discuss how this Sidekick disaster has impacted them and the experiences they’ve had with T-Mobile’s customer service reps. Enter the appropriately named Skfail.com.

We want to collect user’s experiences as well as news articles, statements from T-Mobile, etc. in one place that people can go. So head on over to the forums and speak up!

T-Mobile released their Monday statement. The short of it? If you lost your data, you’re getting $100. The long of it? Read below:

Updated: 10/12/2009 5:15 PM PDT
T-MOBILE STATUS UPDATE ON SIDEKICK DATA DISRUPTION, MON., OCT. 12

Dear valued T-Mobile Sidekick customers:

We are thankful for your continued patience as Microsoft/Danger continues to work on preserving platform stability and restoring all services for our Sidekick customers. We have made significant progress this past weekend, restoring services to virtually every customer. Microsoft/Danger has teams of experts in place who are working around-the-clock to ensure this stability is maintained.

Regarding those of you who have lost personal content, T-Mobile and Microsoft/Danger continue to do all we can to recover and return any lost information. Recent efforts indicate the prospects of recovering some lost content may now be possible. We will continue to keep you updated on this front; we know how important this is to you.

In the event certain customers have experienced a significant and permanent loss of personal content, T-Mobile will be sending these customers a $100 customer appreciation card. This will be in addition to the free month of data service that already went to Sidekick data customers. This card can be used towards T-Mobile products and services, or a customer’s T-Mobile bill. For those who fall into this category, details will be sent out in the next 14 days – there is no action needed on the part of these customers. We however remain hopeful that for the majority of our customers, personal content can be recovered.

via the Sidekick Forums

The interesting point is that T-Mobile knows who to send these $100 customer appreciation cards to. That means they know who lost data, which means they know what data was lost on the server. So the hope is that they were able to recover some user’s data and it wasn’t totally lost.

The plot thickens. There’s an extensive article over on AppleInsider that lays out more details regarding the possibles motivations responsible for the Danger Disaster at Microsoft. There are two theories: dogfooding and sabotage.

Dogfooding is a reference to the term “eating one’s own dogfood” or replacing competitor’s technology with your own for internal uses. AppleInsider’s insider source says

Danger’s Sidekick data center had “been running on autopilot for some time, so I don’t understand why they would be spending any time upgrading stuff unless there was a hardware failure of some kind,” wrote the insider. Given Microsoft’s penchant for “for running the latest and greatest,” however, “I wouldn’t be surprised if they found out that [storage vendor] EMC had some new SAN firmware and they just had to put it on the main production servers right away.”

But as the article points out, what could have been the motivation for Microsoft to change technologies when they were already scuttling the Danger product in the attempts to save the Pink Project? Microsoft was only running things as they were bound by contract with T-Mobile. The article goes on to say:

Instead, the fact that no data could be recovered after the problem erupted at the beginning of October suggests that the outage and the inability to recover any backups were the result of intentional sabotage by a disgruntled employee. In any other circumstance, Microsoft or T-Mobile would likely have come forward with an explanation of the mitigating circumstances, blaming bad hardware, a power failure, or some freak accident.

An act of sabotage “would explain why neither party is releasing any more details: for legal reasons dealing with the ongoing investigation to find the culprit(s),” one of the sources said. Due to the way Sidekick clients interact with the service, any normal failure should have resulted in only a brief outage until a replacement server could be brought up.

So, was it sabotage? I’ve been saying all along that the level of this screw up is so high you start to think that someone has to be intentionally screwing up this much. Whether that’s actually sabotage or just Microsoft’s apathy and ineptitude is unknown at this point.

On top of all this, AppleInsider points out the obvious: T-Mobile must be irate with Microsoft right about now. Surely in their contract there was a Service Level Agreement (SLA) that dictated a certain level of uptime. And with almost two weeks of outage plus an enormous loss of data this “is the worst possible violation of the SLA conceivable”. I’m just waiting for the lawsuits between T-Mobile and Microsoft/Danger to start.

We’re hearing a number of reports from people that T-Mobile is shouldering the brunt of this disaster (which is truly Microsoft’s fault) by letting customers upgrade to new phones. The reports in our comments are mixed, but it seems like some people are getting a new G1 or a new MyTouch 3G and are paying a discounted rate or even getting it free. What’s more important is that we’re hearing many of these customers are being given the data plan at the $19.99 rate many of us are grandfathered in for with the Sidekick. There are also reports that people are being let out of their contract if they have a Sidekick, but we’re also hearing that they may not be allowed to do that anymore. I guess it depends on who you talk to and when as I’m sure things at T-Mobile are changing rapidly.

I’m holding out hope, but if you’re calling T-Mobile to switch, let us know your experience!

UPDATE: “Free” seems to be more of a rumor than truth. It looks like the norm for most people is that they are getting to upgrade to a phone at the upgrade price you see on T-Mobile.com ($129.99 for a G1, $149.99 for a MyTouch, etc). You’ll probably be required to sign a contract, but should be able to keep the $19.99 data plan. This is even for people that had just renewed their contract with a purchase of the Sidekick LX 2009. Not sure if people out of contract are getting any special treatment on top of this to keep them as a T-Mobile customer.

By now the word is out on the street. Microsoft/Danger has most likely lost everyone’s personal info including contacts, notes, calendar entries, to-dos, etc. The question remains: How did this happen? Microsoft is a big software company, they’re well versed in the enterprise world and should have systems in place that allow them to weather any sort of issue like this. Of course everyone (T-Mobile, Microsoft/Danger) hasn’t come out with any details on the cause of the failure, but we’ve got some theories and rumors floating around.

Currently the rumor with the most weight is as follows:
Microsoft was upgrading their SAN (Storage Area Network aka the thing that stores all your data) and had hired Hitachi to come in and do it for them. Typically in an upgrade like this, you are expected to make backups of your SAN before the upgrade happens. Microsoft failed to make these backups for some reason. We’re not sure if it was because of the amount of data that would be required, if they didn’t have time to do it, or if they simply forgot. Regardless of why, Microsoft should know better. So Hitachi worked on upgrading the SAN and something went wrong, resulting in its destruction. Currently the plan is to try to get the devices that still have personal data on them to sync back to the servers and at least keep the data that users have on their device saved.

We’ve heard this from what appears to be several sources and it seems to hold weight. Needless to say it all boils down to one thing: Microsoft did not have a working backup.

How this happens in today’s day and age is beyond belief. Hundreds of thousands of customers that generate millions of dollars in revenue means you back their stuff up, in triplicate. You test these backups regularly, and you move a copy off site that doesn’t get touched except in case of an emergency (i.e. right now). The head of the mobile division (and person in charge of what’s left of Danger) is Roz Ho, who has been at Microsoft for 18 years. You would think she’d know something about how to run a business.

What does this mean for the future of the Sidekick? Unless Microsoft pulls a miracle out of thin air the Sidekick is dead. People are already jumping ship to other phones with this news, and the exposure of how inept Microsoft is when it comes to the mobile world is huge. If Microsoft can’t continue to run Danger, a company that was ground-breaking and solidly built, how can we expect anything from the Windows Mobile department?