Life as a Sidekick user is starting to return to normal as T-Mobile/Microsoft/Danger slowly announce each piece of data that is restorable as they make progress. We’ve gotten back Address Books, Calendars, To Do Lists, Notes, and most recently Photos. I know a lot of people are still waiting on Bookmarks and I imagine that will be next in line. However one of the hallmarks of the Sidekick is still missing and we haven’t heard any word on when it will return.

I’m of course talking about the Download Catalog, aka the first App Store. The Catalog has been down for over a month and this sucks both for Sidekick users as well as Sidekick Developers. I’m sure T-Mobile is feeling the hurt too as a large source of their income from Sidekick users comes from selling applications, ringtones and themes.

While there are no other ways to get applications (save having a developer key), it is possible to get ringtones and themes while the catalog is down from unofficial sites. Themes for most Sidekicks can be found at Sidekick Themes and if you have a Sidekick LX 2009 you can get free ringtones at Skringers.com.

UPDATE (2009-11-11): We’re hearing through the grapevine that the Download Catalog may be up by Saturday if things go well! Keep your fingers crossed.

Danger/Microsoft has added one more thing to the list of data you can restore. If you had photos that were saved on the Danger servers you can get them back now on my.t-mobile.com. Here’s how:

Although we cannot directly restore photos to your Sidekick through the sync process, if you wish to receive a copy of your photos by e-mail, please enter and confirm your e-mail address below. Your photos will be sent to you via e-mail, one photo per e-mail. Depending on how many photos you have, this process may take some time. Once all your photos have been sent, we’ll send a confirmation e-mail to your Tmail account.

Please note: Don’t send these to your Tmail address as you could exceed your Sidekick’s e-mail quota.

Microsoft/Danger has started rolling out an OTA (Over-The-Air) update to all Sidekick users. This update will not make any changes to your device that you’ll be able to see. It’s intended to fix some behind the scenes stuff so that “all Catalog application data will now be more securely backed up in the Microsoft / Danger systems.” This is NOT the Sidekick LX 2009 OTA I posted about earlier, which I assume is on indefinite hold until everything gets sorted out. It’s assumed that this OTA is rolling out to all Sidekick devices. Let us know in the comments if you received the OTA, which device you’ve got, and when you got the update.

T-Mobile posted an update over on their forums and we’ve archived a copy over on skfail.com since the forum updates aren’t permanent.

Dear T-Mobile/Danger:

My sentiments as a long-time subscriber:

T-Mobile:

I’ve had your services since your spokesperson was a parrot and your name was Omnipoint. I even sold them as an Authorized Dealer when the pager business died. It was 1998, and you only offered a handful of brick-sized phones (remember the Ericsson 388?), and your flagship handset had a 3-color screen and was made by Siemens. I remember using prepaid coupons to pay my postpaid bill – Genius! Your network was small, your plans were so-so, but digital was superior, and you were my first choice.

You became Voicestream, and had a fancy, stylized logo and Jamie-Lee Curtis was your spokesperson. She gave James Earl Jones a run for his money and you began to look like a major player. There were plenty of companies to choose from, but I stayed faithful. I used all the newest handsets and loved the ease of switching my SIM, allowing me to try all the newest phones. My Nokia 3300 fell 100 times but never failed me. I would just put the shell back together, and I had all the colors. I composed my own ringers painstakingly from the composer before it became a billion-dollar business. AT&T was still TDMA back then, and you were the only GSM player for a while (It still amazes me how you let Cingular/Big Blue leapfrog you with your own technology, but I digress).

You gobbled up smaller carriers left and right, and soon you got your new name. T-Mobile. Your coverage maps still looked like daddy-long-legs, but with each revision, the pink spiders gained weight. You were determined to provide call quality over tower quantity, knowing the latter would come soon enough. I always had enjoyable experiences as a customer and got my whole family under your wing.

Flash forward to 2002. I had a two-way pager from Skytel, and life was good. I could actually check email without a COMPUTER! My last regular handset was a Motorola V70, a slick-looking phone that rotated instead of flipping open. But I lusted for something that could do it all. I bought an unlocked Motorola Accompli 009 online to combine my 2-way pager with a phone, and also had a color screen. Mandatory headset required though, and Bluetooth didn’t exist yet. That didn’t last long. I looked at my sheet of upcoming phones and saw some chunky, grey, calculator-looking thing with a full keyboard. And the screen did some funky, flip move that snapped open like I always wished the V70 had done. Was this the device I had been looking for?

Danger:

Ahhh, what say I say to my dear old friend. I picked up your ugly duckling of a device and never looked back. I had never been brand-loyal with handset manufacturers, but boy, did that change. In my entire history as a wireless customer, no device has matched the ingenuity of something so useful as the Jump button. I’ve multi-tasked with ease, and frequented the hiptop.com (later, poweredbydanger.com) forums daily anticipating each OTA update. I remember when AIM smileys and copy/paste were added. I posted so often under the name JustifydHomicide that I was a “Power User”. Appletech, The Gryphon, mwsmith, JHC – These people were the gurus of my Sidekick experience. I got a developer’s key and loved testing applications, learning the Menu+Shift shortcuts like the back of my hand.

You were always so ahead of the curve, single-handedly creating the catalog model for which Apple smugly gets credit, and for which all the other OS companies are still clamoring. I felt that I was using the latest and greatest technology, for a while that is. I anticipated each new hardware launch and recall ordering the Sidekick 2 at midnight of the launch. I’ve done the same for each new device, and even sprung for the Mr. Cartoon and LRG LE’s. I remained positive after your acquisition by Microsoft, even after they unceremoniously fired half of your staff, anxiously hoping that someday I’d see the convergence of the Sidekick with another favorite, the Zune.

When the G1 came out, I resisted temptation and waited to upgrade to the LX09 even though 3G hadn’t launched in my area yet. I was grandfathered into your $20 data, and was willing to forgo the latest handset and stay loyal to you. For the first time, however, I was thoroughly unimpressed by your new outfit. Sure, your new dress was sexy and sleek, but the new integration of social networking apps lacked functionality and slowed my typically reliable OS to a crawl. I became plagued with frequent resets and freezing and now stay logged out of apps unless I need them, opting to use the browser for Twitter rather than the especially-limited catalog version. It soon became clear why this app and others were provided to the end user for free.

I began to fall out of love with you after gazing at superior products. I even flirted with a few, impressed by better features and ease of use. I developed an infatuation for Android like a middle middle-aged man staring at a young home-wrecker. The company whose ingenuity I had loved and been loyal to had changed like an estranged wife of 6 years. It felt like you lost your touch, and looking at your renders of the Project Pink phones had me drawing up my divorce papers.

Then, there was the straw that broke the camel’s back,

October 2nd, 2009:

You know the particulars, so I won’t cover old news. You know what you did. You cheated on me. I had a trust in you that I was sure you wouldn’t break. My secrets were safe with you throughout our relationship, and then you walked out on me in the form of a data outage. It had happened before, but never like this. I couldn’t reach you for over a week, and when I heard from you, I realized that it was worse than I thought. You didn’t just fail me, but you also destroyed everything we built together. Taking over 1000 contacts with you when you left was like emptying the joint account and taking the kids. You didn’t even leave the pictures as a memento. How cruel could you be?

Apparently you had a change of heart, restoring some of my contacts, although you haven’t yet made me whole. I’m still waiting for you to drop off the rest of my stuff – Do you think you can get the calendar, bookmarks, and photos back to me soon? I’m starting to think it’ll all work out in the end; I’m getting comfortable with the idea that we won’t be together much longer. I noticed you took yourself off the market on t-mobile.com, too. Decided not to play the field?

Android looks better every day, but I like my ladies with keyboards, and G1’s getting wrinkles every day. I check in on my new love interests N900 and Bold 2 daily, and can’t wait to choose my new wife. T-Mobile at least had enough Loyalty (Plan) to allow me to forgive them, but you I cannot. The credits and new handset they offered me make me feel a little better.

T-Mobile:

With your bold launch of Project Dark, this could not have come at a more inopportune time. I applaud your willingness to adopt a new model and buck the tradition that has plagued the wireless climate in the U.S. I will not be converting to one of these new plans, as my current plan suits me better. I do wonder two things about your no-contract Even More Plus plans:

Will handsets purchased at full retail come unlocked?

Will customer be allowed to provide their own unlocked handset and sign up for Even More Plus?

In closing, thank you T-Mobile for always having a “Think Customer First” attitude, a credo of another large and successful company. Your customer service is unparalleled, and your philosophy as a company influenced me to work for your corporation during my 9-year wireless career, before moving on to a different industry. In this situation, you didn’t hesitate to concede your faults for Microsoft/Danger’s shortcomings. Although you were only indirectly responsible, you realized that ultimately YOU collect my bill, and I respect that. I won’t disclose what concessions you’ve offered me to retain my business, but I do beleive that I will continue to a be satisfied customer some time in November. I hope Microsoft makes you whole for your loss of revenue due to this debacle.

Cellphone Chris

How is Microsoft recovering all the user data without having a backup to turn to? Inside sources tell Daniel Eran Dilger that Sun (the storage vendor) and Oracle (makers of the database software) have sent in their best people to attempt to stitch the database back together. Why is it taking so long?

“The first thing to do is wheel in a big pile of new disk space, and copy the individual disks so there is a raw backup. This is like making a copy of a jigsaw puzzle one piece at a time. Then they would assemble the puzzle using the copied pieces, in case any pieces need to be re-made from the original.

“This is very hard, requires detailed inside knowledge of how SAN addresses and volume manager layouts fit together with Oracle tables. Finally, they need to start up the database on top of the assembled puzzle, and Oracle will do its own clean up to get into a consistent state.

“The next thing you do is a fresh backup (several days), before you allow any users access to it. So it’s not surprising that this would take over a week, even after it was possible to say that the data is recoverable.”

I’m assuming that Microsoft considers the data “recovered” if they’re letting users access it. But we’re hearing reports of people missing contacts, some phone numbers for individual contacts, and other weird behavior including not even being able to download their info from the Desktop Interface. How is the data recovery working out for you guys?

PCWorld has an article on Microsoft’s CEO, Steve Ballmer, and his attitude towards the Sidekick disaster. He was quoted as saying:

“It is something we are going to have to address and explain to customers our method and process and quality approach and what went wrong in that case and how we are making sure that it does not happen again”

“It is not clear there was data loss,” he said. “Initially we thought there was. We are working hard to get all the users’s data back in the Sidekick case. I think we believe we will get all user data back at this juncture.”

So rest assured, even the CEO of Microsoft is noticing this mess-up.

T-Mobile has posted instructions on how to download your contacts and restore them to your Sidekick. Basically you log in to your account at my.t-mobile.com and a new link will appear on the front page, “Restore your contacts”

Once you click on the link to Restore your contacts, it’s a 2 step process to fully restore contacts. Download the contacts to your computer, and then upload the downloaded file to your Desktop Interface. It will probably take a few minutes for them to appear on your device after this.

This is only the first phase of recovery, as T-Mobile, Danger and Microsoft have stated that the remaining PIM data (Calendar, Notes and ToDo’s) as well as Photos saved to the backend and not your SD card will be returning shortly. Perhaps through a similar method.

Let us know in the comments how the recovery goes and if you were in the group that lost all their contacts, or some of them. We’re curious to know!

I reported early on that the cause of the Sidekick disaster was a SAN upgrade that failed without proper back-ups in place. (see also: More Details On What Caused the Sidekick Disaster) Daniel Eran Dilger, who has chimed in several times on this situation, has confirmed this through another source as well. He has an excellent article detailing many of the factors involved in all of this mess. If you have the time, I recommend reading the article.

But for those that don’t have time, here’s a quick summary:
Why weren’t there backups?
- Confirmation from a source that the loss of data was caused by stopping a 6-day backup by order of Microsoft / Roz Ho against the recommendations of Danger engineers. Because the backup was started, they had to remove a backup from a couple months ago in order to make space available. So they were left with an incomplete backup, since it was only run for 2 of the 6 days necessary. The SAN upgrade proceeded and things apparently went wrong.

No big deal, Microsoft says they recoved “most if not all” of our data
- Daniel Eran Dilger is skeptical that this is actually the case. He believes that it is possibly just Microsoft in denial. He writes:

If the company has stumbled upon a novel recovery avenue or some unknown backup that somehow remained missing for nearly two weeks, then this is great news for Sidekick users and helps to wipe some of the egg from the company’s cloud computing services face, although the situation still remains as the worst datacenter failure to ever impact mobile users as well as one of the most absurd responses pertaining to lost data as well.
…
However, Microsoft is also well known for advertising bullshit it can’t deliver.
…
f Microsoft strings along users long enough, it will be able to pat itself on the back with a “mission accomplished” even if it ultimately never actually delivered anything. It’s like saying you’ll call somebody back after a date and then just waiting until they figure out that you’re not really interested. After two weeks, the party on the other end begins blaming itself for waiting around.

Final thoughts
I’m paraphrasing here:
- If Microsoft can deliver even most of most users’ data, that’s awesome but doesn’t make up for the fact this happened in the first place.
- If Microsoft can’t deliver the data and this is just public relations BS to keep the negative press away until mainstream media forgets about the whole issue, and things fade into the background, then this is ridiculous.

Further, with this announcement, even if the company has no real data to recover, it will have erected a plausible story for denying anything significant ever happened. Know somebody who actually lost their important Sidekick data? You’ll be able to write them off as “one of the few who didn’t benefit from Microsoft’s miraculous data recovery.” It will be their word against Microsoft’s PR. Nobody will have records of who was impacted and whose data was recovered apart from Microsoft and probably T-Mobile, and the provider will likely have its records sealed by court order when it gets its big SLA settlement from Microsoft.

His closing paragraph makes it very clear that we need to keep this issue documented and not let Microsoft weasel their way out of another situation. I’ve opened up SKFail.com as a forum for this. Please head over there and leave posts detailing your experience, whether you lost data, if you’ve had it restored, etc.

What started out as “the worst loss of consumer data” will probably now be called “Microsoft’s Worst PR Disaster”.

An update from Microsoft’s own Roz Ho was posted early this morning on the T-Mobile Sidekick forum. She says that they have recovered “most, if not all, customer data” and that it will all soon be restored according to plan. She also confirms that it was indeed a system failure that wiped out the database and back-ups.

So here’s hoping that the Sidekick Disaster will soon be over and everything will be back to normal. I’m still wondering confirmation of the technical details on how this all happened and why Microsoft was so quick to say that all the data was “almost certainly has been lost” early on. Oh and one last handy tip to Microsoft/Danger: Make an application so that Sidekick users can ACTUALLY backup their data themselves.

The full post from Microsoft:

Updated: 10/15/2009 1:00 AM PDT

Microsoft Confirms Data Recovery for Sidekick Users

Data Restoration to Begin as Soon as Possible for Affected Customers

Dear T-Mobile Sidekick customers,

On behalf of Microsoft, I want to apologize for the recent problems with the Sidekick service and give you an update on the steps we have taken to resolve these problems.

We are pleased to report that we have recovered most, if not all, customer data for those Sidekick customers whose data was affected by the recent outage. We plan to begin restoring users’ personal data as soon as possible, starting with personal contacts, after we have validated the data and our restoration plan. We will then continue to work around the clock to restore data to all affected users, including calendar, notes, tasks, photographs and high scores, as quickly as possible.

We now believe that data loss affected a minority of Sidekick users. If your Sidekick account was among those affected, please continue to log into these forums for the latest updates about when data restoration will begin, and any steps you may need to take. We will work with T-Mobile to post the next update on data restoration timing no later than Saturday.

We have determined that the outage was caused by a system failure that created data loss in the core database and the back-up. We rebuilt the system component by component, recovering data along the way. This careful process has taken a significant amount of time, but was necessary to preserve the integrity of the data.

We will continue working closely with T-Mobile to restore user data as quickly as possible. We are eager to deliver the level of reliable service that our incredibly loyal customers have become accustomed to, and we are taking immediate steps to help ensure this does not happen again. Specifically, we have made changes to improve the overall stability of the Sidekick Service and initiated a more resilient backup process to ensure that the integrity of our database backups is maintained.

Once again, we apologize for this situation and the inconvenience that it has created. Please know that we are working all-out to resolve this situation and restore the reliability of the service.

Sincerely,
Roz Ho
Corporate Vice President
Premium Mobile Experiences, Microsoft Corporation

We all knew it wouldn’t be long before lawsuits were filed over the Sidekick outage and data loss. As Microsoft/Danger still struggles to restore data and get things back up and stable, people are already filing lawsuits, “claiming negligence and false claims.”

A suit filed for a Bakersfield, CA man and “all others similarly situated” says that Danger failed to handle Sidekick user’s data and that they advertised in a misleading manner. He’s asking for monetary damages as well as the court to order Microsoft to fix the Sidekick service or offer a full refund. The attorney handling the case was quoted as saying: “We are hopeful that T-Mobile and the rest of the defendants will do the right thing, use this as an opportunity to redesign the system as a new standard for cloud computing storage, and provide full compensation for the data loss.”

Another class action law suit (PDF of filing) was filed for Maureen Thompson and again “all others similarly situated” against T-Mobile/Danger/Microsoft for the outage and loss of data. Same sort of thing.

And there’s yet another suit (PDF) filed by Oren Rosenthal against T-Mobile for the negligence, breach of contract, blah blah blah.

Should be interesting to see how these play out. If you hear of any others, let us know over on skfail.com’s Lawsuit forum.

image via kcoury