The following is reportedly from “someone close to the action”:

“Here’s the actual scoop, from someone involved in the recovery:

Danger, purchased by Microsoft, was moved into a Verizon Business datacenter in Kent, WA a short while ago. While this had to do with the MS assimilation, it was done as a one for one move from Danger to a DC that MS uses heavily. (MS didn’t re-write, port, migrate to winblows, etc.) The backend service uses a variety of hardware, load balancers, firewalls, web and application servers, and an EMC SAN (Storage Area Network, think huge drive array connected with fiber.)

Well last Tuesday, the EMC SAN took a dump on itself. What I mean by that is the backplane let the magic blue smoke out. While usually in the heavy iron class of datacenter products like an EMC SAN this means you fail over to the redundant backplane and life continues on. Not this time folks. In the process of dying, it took out the parity drives. What does that mean? It means the fancy RAID lost it’s ability to actually be a RAID. How much data got eaten by this mega-oops? 800TB. Why wasn’t it backed up? It was, to offsite tape, like it’s supposed to. But when the array is toast, can’t just start copying shit back.

Apparently EMC has been on site since Tuesday, but didn’t actually inform Danger/MS that their data is in the crapper until Friday afternoon. On top of that, EMC has done nothing to bring in replacement equipment between Tuesday and Friday. (In the Enterprise support world, that’s fucking retarded, multi-million dollar support contracts are that expensive for a reason.)

So what’s being done? Well the good news is that the complex was slated to be migrated into the Verizon Business cloud services (not MS’s cloud per se, but it’s MS’s effort.) And as a part of that migration a newer shinier SAN array was in process of being implemented. But space isn’t ready for it on the datacenter floor, and you can’t just toss the EMC raid and place this one in it’s place, it’s a different vendor and is 2 racks instead of one. This means it’s being shoehorned into a different part of the datacenter than was originally planned, one that doesn’t have the necessary 3 phase power installed. So there’s a bit of work to be done. Not to mention the restoral of 800TB of backup data from offsite tape.

Time to restoral? Looking like Wednesday at the earliest with techs working all weekend.”

Sounds like they know what they’re talking about, but since we haven’t been able to confirm this directly ourselves, we’re keeping it labeled as a rumor.

UPDATE (2009-10-15 01:02 PST): We’ve confirmed that Danger does indeed have servers in a Verizon Business Data Center, however it appears to be one in California, NOT Kent, WA. If you want to confirm, do a traceroute on one of Danger’s web proxies and you’ll find it ends up at danger-gw.customer.alter.net (157.130.202.122), an IP owned by Verizon (MCI) that appears to be around the San Jose/Santa Clara area. It’s possible (although unlikely) that the web proxy servers are kept separate from the user data servers though.

Dogfooding is a reference to the term “eating one’s own dogfood” or replacing competitor’s technology with your own for internal uses. AppleInsider’s insider source says

Danger’s Sidekick data center had “been running on autopilot for some time, so I don’t understand why they would be spending any time upgrading stuff unless there was a hardware failure of some kind,” wrote the insider. Given Microsoft’s penchant for “for running the latest and greatest,” however, “I wouldn’t be surprised if they found out that [storage vendor] EMC had some new SAN firmware and they just had to put it on the main production servers right away.”

But as the article points out, what could have been the motivation for Microsoft to change technologies when they were already scuttling the Danger product in the attempts to save the Pink Project? Microsoft was only running things as they were bound by contract with T-Mobile. The article goes on to say:

Instead, the fact that no data could be recovered after the problem erupted at the beginning of October suggests that the outage and the inability to recover any backups were the result of intentional sabotage by a disgruntled employee. In any other circumstance, Microsoft or T-Mobile would likely have come forward with an explanation of the mitigating circumstances, blaming bad hardware, a power failure, or some freak accident.

An act of sabotage “would explain why neither party is releasing any more details: for legal reasons dealing with the ongoing investigation to find the culprit(s),” one of the sources said. Due to the way Sidekick clients interact with the service, any normal failure should have resulted in only a brief outage until a replacement server could be brought up.

So, was it sabotage? I’ve been saying all along that the level of this screw up is so high you start to think that someone has to be intentionally screwing up this much. Whether that’s actually sabotage or just Microsoft’s apathy and ineptitude is unknown at this point.

On top of all this, AppleInsider points out the obvious: T-Mobile must be irate with Microsoft right about now. Surely in their contract there was a Service Level Agreement (SLA) that dictated a certain level of uptime. And with almost two weeks of outage plus an enormous loss of data this “is the worst possible violation of the SLA conceivable”. I’m just waiting for the lawsuits between T-Mobile and Microsoft/Danger to start.

tommyd2107 says:

I was just on my phone and when I got off my phone all my contacts returned. I do not know if this will last for long but the site of my contacts returning is encouraging.

dariahna says:

The same thing happened to me…I shut my phone off several times without removing the battery…voile’! My contacts returned!

generalblue says:

Got my contacts back too! My phone froze up on me and since I have already lost everything anyways, I just took the battery out and put it back in. Once my phone was on about 5 minutes later I checked my address book for some reason and they were all there. I saved all my contacts to my simcard.

Anyone else out there have any luck?

Another tip that is being reported on poweredbydanger.com that is unofficial, not 100% tested, try at your own risk, might just work, etc. is if you have an old device that has your contacts and info you might be able to do a reverse sync (i.e. device to server). Make sure your old device is turned off, plug it into the wall, remove the battery, insert your SIM, replace the battery, power up, and cross your fingers. With any luck it’ll update Danger’s servers with this info, and then you can switch back to your newer Sidekick which will pull it back down. This is at your own risk and should only be tried after you’ve backed up all the contents on that old device. If you do try this, let us know if it works for you.

I’m holding out hope, but if you’re calling T-Mobile to switch, let us know your experience!

UPDATE: “Free” seems to be more of a rumor than truth. It looks like the norm for most people is that they are getting to upgrade to a phone at the upgrade price you see on T-Mobile.com ($129.99 for a G1, $149.99 for a MyTouch, etc). You’ll probably be required to sign a contract, but should be able to keep the $19.99 data plan. This is even for people that had just renewed their contract with a purchase of the Sidekick LX 2009. Not sure if people out of contract are getting any special treatment on top of this to keep them as a T-Mobile customer.

Currently the rumor with the most weight is as follows:
Microsoft was upgrading their SAN (Storage Area Network aka the thing that stores all your data) and had hired Hitachi to come in and do it for them. Typically in an upgrade like this, you are expected to make backups of your SAN before the upgrade happens. Microsoft failed to make these backups for some reason. We’re not sure if it was because of the amount of data that would be required, if they didn’t have time to do it, or if they simply forgot. Regardless of why, Microsoft should know better. So Hitachi worked on upgrading the SAN and something went wrong, resulting in its destruction. Currently the plan is to try to get the devices that still have personal data on them to sync back to the servers and at least keep the data that users have on their device saved.

We’ve heard this from what appears to be several sources and it seems to hold weight. Needless to say it all boils down to one thing: Microsoft did not have a working backup.

How this happens in today’s day and age is beyond belief. Hundreds of thousands of customers that generate millions of dollars in revenue means you back their stuff up, in triplicate. You test these backups regularly, and you move a copy off site that doesn’t get touched except in case of an emergency (i.e. right now). The head of the mobile division (and person in charge of what’s left of Danger) is Roz Ho, who has been at Microsoft for 18 years. You would think she’d know something about how to run a business.

What does this mean for the future of the Sidekick? Unless Microsoft pulls a miracle out of thin air the Sidekick is dead. People are already jumping ship to other phones with this news, and the exposure of how inept Microsoft is when it comes to the mobile world is huge. If Microsoft can’t continue to run Danger, a company that was ground-breaking and solidly built, how can we expect anything from the Windows Mobile department?

# The source says that the details about the Project Pink phones are true and that they wouldn’t be surprised if the photos were an intentional leak, however the rumors about Microsoft working with Verizon and the unhappy employees are definitely from an unhappy (ex) employee.

# What about handset makers?

“Let’s start with the handset makers. Motorola has announced their intention to abandon WinMo for Android, and Palm has already abandoned WinMo in favor of their own WebOS. A few others, such as HTC, have been hedging their bets, announcing their support for Android, but not yet willing to publicly disown WinMo. Now the story has been leaked that Microsoft has been secretly working with Sharp on this (doomed) [Pink] project for over a year, cutting everyone else out of the picture completely.

”You figured out the likely consequences of this over a month ago so I won’t belabor the point other than to say that Sharp, who has been a terrific partner to Danger throughout the Hiptop/Sidekick era, was no doubt hoping that Pink would allow them to expand their phone offerings in the Americas, a dream which has now been dashed due to typical Microsoft blundering. I doubt that either Sharp or any of the other handset OEM’s will ever take Microsoft at their word again.

# Trying to build 4 versions of the Pink phones?

“the Pink planners decided to try to build both UMTS and CDMA phones in both form factors, for a grand total of four different SKUs. None of the other smartphone platform builders were foolish enough to try to build more than one phone at a time, and whoever made this decision didn’t take into account the added complexity involved in coding to support two different radio modules (from two different vendors, no less!) for two completely different radio technologies, not to mention validating and testing the RF performance of four different antennas (since each form factor and radio technology requires its own antenna design). At some point the UMTS project was placed on the back burner, and for all I know it may have already been cancelled, but the damage to the project has been done.”

# Pitting Verizon against T-Mobile

”The final operator who is going to be pissed is T-Mobile, who has been just as loyal of a partner to Danger as Sharp has been. I don’t know exactly what Microsoft has been telling them, but they have no doubt realized that they’ve been cut out of this deal in favor of their largest competitor. What’s worse is that apparently Microsoft has been lying to them this whole time about the amount of resources that they’ve been putting behind Sidekick development and support (in reality, it was cut down to a handful of people in Palo Alto managing some contractors in Romania, Ukraine, etc.).

“The reason for the deceit wasn’t purely to cover up the development of Pink but also because Microsoft could get more money from T-Mobile for their support contract if T-Mobile thought that there were still hundreds of engineers working on the Sidekick platform. As we saw from their recent embarrassment with Sidekick data outages, that has clearly not been the case for some time.

# So what happens now?

“The most likely possibility is that anything worth salvaging from Pink will get folded into the larger WiMo 7 project, where it should have been located in the first place. Another possibility would be another massive layoff and shutting down the Danger offices, but with the economic crunch, I doubt that Microsoft would choose to take the hit of handing out severance packages to all the affected engineers, and breaking the lease on the buildings.”

# And finally, what all of us wish would happen, and the reasons why it probably won’t:

”The smartest course of action (and therefore the one that Microsoft is least likely to take!) would be for them to recognize the value of the original Danger platform that they acquired, and to rebuild the Hiptop/Sidekick into a force to compete with Android (which is, after all, “Sidekick 2.0” in many respects). This won’t happen for three reasons: 1) Microsoft’s irrational hatred of Java, 2) Microsoft’s irrational love of Windows in all of its horrible flavors, and 3) all the Danger folks who loved the Sidekick platform have left or likely will leave soon, and Microsoft has no in-house expertise in Java or the Danger platform.

October has been a bad month for the Sidekick and the future of what’s left of the Danger team. Not only was there the Great Sidekick Data Outage of October ‘09, now we’re hearing that Project Pink, a Microsoft project that the remaining Danger team was put on, might not even be seeing the light of day.

MobileCrunch heard some interesting news from an anonymous source that seems to be legit. Here’s the bullet points:

# As I reported previously, most of the Danger team has been fired or left.
# If any of the Pink devices ship, supposedly they will not have third party application support, i.e. no App Store/Download Catalog.
# The Project Pink employees are not happy. Most of the team still uses Sidekicks or even iPhones and “hate the product”. And here’s the best quote from MC’s source: many feel that the division exists only to “challenge [the Windows Mobile 7 team] and upset them into competing.”
# The project is about 2 years behind schedule and some basic applications are getting cut in an attempt to someday launch this thing.
# The touchscreen on the “Turtle” is not a viable touchscreen as the UI contains too many things on the screen at once and your finger ends up covering half the screen.
# The design of the user interface was done by a 3rd party and now Microsoft is trying to actually build the thing with some difficulty.
# And finally, the worst news for the 2 remaining Project Pink fans: MobileCrunch’s source says that the project “is near death and probably will be canceled.”

Ouch, this is of course all unconfirmed rumors, but if any of it is true… our sympathies to the remaining Danger employees.

Trash can photo from http://www.flickr.com/photos/katmere/. Device images via Gizmodo

A little birdie told me that there’s a new OTA (Over-The-Air) Update that should be coming soon for all you Sidekick LX 2009 owners. It’s currently being tested by the super secret trial users. Once all the bugs get squashed it will start rolling out to the general public in the typical Sidekick fashion (randomly and only a few people at first). No word on exactly what bugs have been squashed, but I imagine it’s going to help with the crashing and freezing that many users (including yours truly) have reported on their Sidekick LX 2009.

Looks like prototype images of the two rumored Project Pink phones have leaked out of Microsoft. Gizmodo posted both the Pure and Turtle devices, confirmed they’ll be made by Sharp, and that they’re going to be aimed at the Sidekick market. So while these won’t be “Sidekicks” per-say, they are the devices that Microsoft expects you current Sidekick users to be interested in. So, what do you think?

via Gizmodo

It’s looking more and more like the next “Sidekicks” won’t really be the Sidekick we know and love. With the purchase of Danger by Microsoft, the team behind the Sidekick has been moved over to a project codenamed Pink. I’ve mentioned this project before, but 9to5mac has some new info on Pink. They’re reporting that the first two devices to come out of Project Pink will be landing in early 2010. The devices will be sliders, so the familiar screen flip will be gone, but they will be manufactured by Sharp like the current ‘kicks. There are two devices rumored to be coming out. Their code names are “Turtle” and “Pure”.

Turtle will be a “squircle” shaped device with a slide out keyboard. The image you see above is a supposed drawing of it. The slide out keyboard will also have a mic so you’ll need to have the keyboard open when you’re making a phone call. The Pure device is more along the lines of a standard candy-bar slider style device. So far no pics of Pure have leaked out yet.

They’re both rumored to be setup to use an App Store model of application delivery and I wouldn’t be surprised if behind the scenes it was the same technology running Danger’s Download Catalog. However probably most controversial news to die-hard Sidekickers: It’s looking like the new devices might be running on Windows Mobile 7. What do you think? Does Microsoft have a chance to jump back in the market or is it too late as everyone jumps ship to iPhones and Android devices.