It ain’t easy… being Pink. Techeye is reporting that Microsoft’s Project Pink phones are going to be delayed until sometime in April. The reason for the delay? “issues with the European models and roll out,” but that’s not to say there’s anything wrong with the phones which are already done and ready. There’s just some stuff to work out with carriers.

It’s looking like they’ll be available on Verizon here in the states and Vodafone in the UK. Originally there was talk about unveiling the Pink phones (Turtle and Pure) at CeBIT but now it’s looking like Microsoft is going to throw their own events to herald the arrival of the Pink phones and they’re willing to shell out big bucks to make sure it’s something significant..

Has the day finally come? T-Mobile appears to have pulled all of the Sidekick models from their site except for the Sidekick LX 2009 Carbon. And even the Sidekick LX 2009 Carbon is listed as “Temporarily Out of Stock”. Is this just a hiccup on T-Mobile’s site or does this mean the end of the Sidekick?

If you’re still dying for a Sidekick, there are other places to get them.

UPDATE: Confirmed via our sources that at least the Sidekick 2008 is discontinued.

UPDATE #2: The Sidekick LX 2009 Carbon has returned to T-Mobile’s site. More info on T-Mo’s forums.

UPDATE #3: It’s official, the 2008 and the LX 09 Orchid are gone according to an official post on the T-Mobile Forums

There are a number of tweets on Twitter over the past few months that specify the client as “Danger” and have a mysterious hashtag of #tmdp or #tmd. The posts started showing up with the #tmd tag but later change to the #tmdp tag.

Many of the tweets have “from Danger” listed as the client, which Twitter links to Microsoft. Usually tweets sent from the Sidekick say “from Sidekick” so this seems to indicate that this could be a new Twitter client from Danger. Whether this Twitter client means a new device, new operating system, the continuation of Project Pink, etc is anyone’s guess at this point.

Here’s some of the interesting tweets:

@JohnMatherly: Thanks to Roz for a #tmdp breakfast in Palo Alto
11:56 AM Dec 7th, 2009 from Danger

@JohnMatherly: We’re counting on ProdUS to be #tmdp
4:52 PM Jan 19th from Danger

@darylw: #tmdp Welcome new team members….we are excited to have you join us.
1:38 PM Dec 16th, 2009 from Danger.

@darylw: #tmdp wish i saw more posts from team members on twitter.com that says posted “n hours ago from Danger”
11:55 PM Nov 18th, 2009 from web

@pbernard: i think twitterfall is working #tmdp
4:22 PM Nov 4th, 2009 from DestroyTwitter

@pbernard: digging xbox twitter #tmd
8:44 PM Sep 24th, 2009 from Xbox LIVE

@pbernard: and we have PINK DONUTS for strat review #tmd
7:41 AM Sep 15th, 2009 from DestroyTwitter

@pbernard: #tmd ta-da.
1:42 PM Aug 27th, 2009 from DestroyTwitter

@terminal4: wow. it’s like stuff is kinda workin #tmdp
10:04 PM Dec 9th, 2009 from Danger.

@JRayZor: This is harder to use than binary mode ftp #tmd
12:08 AM Sep 4th, 2009 from web

@cpicoto: 2311 CD INT – Palo Alto, CA
11:46 AM Nov 11th, 2009 from Danger

@danblack: the #tmdp can fix itself sometimes.
11:37 AM Nov 20th, 2009 from Danger

@danblack: The #tmd is tasty today
3:44 PM Oct 15th, 2009 from Danger

@XuNami: Fun bug bash today. I’m so glad it came together #tmdp
7:45 PM Jan 8th from Danger.

Chris Pirich is one of those tweeting with the #tmdp hashtag and a “from Danger” client. Chris is the head of engineering for Pink.

Other Microsoft people in on the Twitter fun include:

• Pete Bernard: “involved in a number of initiatives regarding mobile and embedded devices.”
• Daryl Welsh: Director of Test – Premium Mobile Experiences at Microsoft
• Andrew Clinick: Group Program Manager at Microsoft
• Carlos Picoto: Director of Program Management at Microsoft
• Dan Black: is/was the Lead Developer Relation Manager for Xbox
• Namrata Bachwani: Release Manager at Microsoft
• Chris Sorensen: Senior Manager, Community and Social Media at Microsoft

- It seems like a lot of XBox people are now part of the mobile experience team. This may be the one thing that bodes well for Pink.
- #TMD is happening at Microsoft Studio G in Redmond.
- There are a lot of references to 4 digit numbers in #tmdp tweets. Could these be build numbers for an OS or firmware?
- @DarylW made a #tmd list

What do you guys think? What could TMDP stand for? Have you found any interesting tweets with this hash tag?

Want more geeky details on what happened at Microsoft/Danger? The short of it is that the SAN took a nose-dive and took out the drives that could have repaired the data with it. A total of 800TB of data was lost. There was an off-site tape backup, a reasonable backup measure, but with the way the SAN died the entire RAID array needed rebuilding and 800TB of data is a lot of data to rebuild. There are more details on server moves in the info below.

The following is reportedly from “someone close to the action”:

“Here’s the actual scoop, from someone involved in the recovery:

Danger, purchased by Microsoft, was moved into a Verizon Business datacenter in Kent, WA a short while ago. While this had to do with the MS assimilation, it was done as a one for one move from Danger to a DC that MS uses heavily. (MS didn’t re-write, port, migrate to winblows, etc.) The backend service uses a variety of hardware, load balancers, firewalls, web and application servers, and an EMC SAN (Storage Area Network, think huge drive array connected with fiber.)

Well last Tuesday, the EMC SAN took a dump on itself. What I mean by that is the backplane let the magic blue smoke out. While usually in the heavy iron class of datacenter products like an EMC SAN this means you fail over to the redundant backplane and life continues on. Not this time folks. In the process of dying, it took out the parity drives. What does that mean? It means the fancy RAID lost it’s ability to actually be a RAID. How much data got eaten by this mega-oops? 800TB. Why wasn’t it backed up? It was, to offsite tape, like it’s supposed to. But when the array is toast, can’t just start copying shit back.

Apparently EMC has been on site since Tuesday, but didn’t actually inform Danger/MS that their data is in the crapper until Friday afternoon. On top of that, EMC has done nothing to bring in replacement equipment between Tuesday and Friday. (In the Enterprise support world, that’s fucking retarded, multi-million dollar support contracts are that expensive for a reason.)

So what’s being done? Well the good news is that the complex was slated to be migrated into the Verizon Business cloud services (not MS’s cloud per se, but it’s MS’s effort.) And as a part of that migration a newer shinier SAN array was in process of being implemented. But space isn’t ready for it on the datacenter floor, and you can’t just toss the EMC raid and place this one in it’s place, it’s a different vendor and is 2 racks instead of one. This means it’s being shoehorned into a different part of the datacenter than was originally planned, one that doesn’t have the necessary 3 phase power installed. So there’s a bit of work to be done. Not to mention the restoral of 800TB of backup data from offsite tape.

Time to restoral? Looking like Wednesday at the earliest with techs working all weekend.”

Sounds like they know what they’re talking about, but since we haven’t been able to confirm this directly ourselves, we’re keeping it labeled as a rumor.

UPDATE (2009-10-15 01:02 PST): We’ve confirmed that Danger does indeed have servers in a Verizon Business Data Center, however it appears to be one in California, NOT Kent, WA. If you want to confirm, do a traceroute on one of Danger’s web proxies and you’ll find it ends up at danger-gw.customer.alter.net (157.130.202.122), an IP owned by Verizon (MCI) that appears to be around the San Jose/Santa Clara area. It’s possible (although unlikely) that the web proxy servers are kept separate from the user data servers though.

The plot thickens. There’s an extensive article over on AppleInsider that lays out more details regarding the possibles motivations responsible for the Danger Disaster at Microsoft. There are two theories: dogfooding and sabotage.

Dogfooding is a reference to the term “eating one’s own dogfood” or replacing competitor’s technology with your own for internal uses. AppleInsider’s insider source says

Danger’s Sidekick data center had “been running on autopilot for some time, so I don’t understand why they would be spending any time upgrading stuff unless there was a hardware failure of some kind,” wrote the insider. Given Microsoft’s penchant for “for running the latest and greatest,” however, “I wouldn’t be surprised if they found out that [storage vendor] EMC had some new SAN firmware and they just had to put it on the main production servers right away.”

But as the article points out, what could have been the motivation for Microsoft to change technologies when they were already scuttling the Danger product in the attempts to save the Pink Project? Microsoft was only running things as they were bound by contract with T-Mobile. The article goes on to say:

Instead, the fact that no data could be recovered after the problem erupted at the beginning of October suggests that the outage and the inability to recover any backups were the result of intentional sabotage by a disgruntled employee. In any other circumstance, Microsoft or T-Mobile would likely have come forward with an explanation of the mitigating circumstances, blaming bad hardware, a power failure, or some freak accident.

An act of sabotage “would explain why neither party is releasing any more details: for legal reasons dealing with the ongoing investigation to find the culprit(s),” one of the sources said. Due to the way Sidekick clients interact with the service, any normal failure should have resulted in only a brief outage until a replacement server could be brought up.

So, was it sabotage? I’ve been saying all along that the level of this screw up is so high you start to think that someone has to be intentionally screwing up this much. Whether that’s actually sabotage or just Microsoft’s apathy and ineptitude is unknown at this point.

On top of all this, AppleInsider points out the obvious: T-Mobile must be irate with Microsoft right about now. Surely in their contract there was a Service Level Agreement (SLA) that dictated a certain level of uptime. And with almost two weeks of outage plus an enormous loss of data this “is the worst possible violation of the SLA conceivable”. I’m just waiting for the lawsuits between T-Mobile and Microsoft/Danger to start.

We’re still waiting for the official word from Microsoft/Danger that they promised us “on Monday”, but we’re hearing a few people report that they magically got their data back. Posts over on the official T-Mobile forum (and an uncensorable thread started on the new skfail.com) are giving us some hope that maybe all this data isn’t truly lost.

tommyd2107 says:

I was just on my phone and when I got off my phone all my contacts returned. I do not know if this will last for long but the site of my contacts returning is encouraging.

dariahna says:

The same thing happened to me…I shut my phone off several times without removing the battery…voile’! My contacts returned!

generalblue says:

Got my contacts back too! My phone froze up on me and since I have already lost everything anyways, I just took the battery out and put it back in. Once my phone was on about 5 minutes later I checked my address book for some reason and they were all there. I saved all my contacts to my simcard.

Anyone else out there have any luck?

Another tip that is being reported on poweredbydanger.com that is unofficial, not 100% tested, try at your own risk, might just work, etc. is if you have an old device that has your contacts and info you might be able to do a reverse sync (i.e. device to server). Make sure your old device is turned off, plug it into the wall, remove the battery, insert your SIM, replace the battery, power up, and cross your fingers. With any luck it’ll update Danger’s servers with this info, and then you can switch back to your newer Sidekick which will pull it back down. This is at your own risk and should only be tried after you’ve backed up all the contents on that old device. If you do try this, let us know if it works for you.

We’re hearing a number of reports from people that T-Mobile is shouldering the brunt of this disaster (which is truly Microsoft’s fault) by letting customers upgrade to new phones. The reports in our comments are mixed, but it seems like some people are getting a new G1 or a new MyTouch 3G and are paying a discounted rate or even getting it free. What’s more important is that we’re hearing many of these customers are being given the data plan at the $19.99 rate many of us are grandfathered in for with the Sidekick. There are also reports that people are being let out of their contract if they have a Sidekick, but we’re also hearing that they may not be allowed to do that anymore. I guess it depends on who you talk to and when as I’m sure things at T-Mobile are changing rapidly.

I’m holding out hope, but if you’re calling T-Mobile to switch, let us know your experience!

UPDATE: “Free” seems to be more of a rumor than truth. It looks like the norm for most people is that they are getting to upgrade to a phone at the upgrade price you see on T-Mobile.com ($129.99 for a G1, $149.99 for a MyTouch, etc). You’ll probably be required to sign a contract, but should be able to keep the $19.99 data plan. This is even for people that had just renewed their contract with a purchase of the Sidekick LX 2009. Not sure if people out of contract are getting any special treatment on top of this to keep them as a T-Mobile customer.

By now the word is out on the street. Microsoft/Danger has most likely lost everyone’s personal info including contacts, notes, calendar entries, to-dos, etc. The question remains: How did this happen? Microsoft is a big software company, they’re well versed in the enterprise world and should have systems in place that allow them to weather any sort of issue like this. Of course everyone (T-Mobile, Microsoft/Danger) hasn’t come out with any details on the cause of the failure, but we’ve got some theories and rumors floating around.

Currently the rumor with the most weight is as follows:
Microsoft was upgrading their SAN (Storage Area Network aka the thing that stores all your data) and had hired Hitachi to come in and do it for them. Typically in an upgrade like this, you are expected to make backups of your SAN before the upgrade happens. Microsoft failed to make these backups for some reason. We’re not sure if it was because of the amount of data that would be required, if they didn’t have time to do it, or if they simply forgot. Regardless of why, Microsoft should know better. So Hitachi worked on upgrading the SAN and something went wrong, resulting in its destruction. Currently the plan is to try to get the devices that still have personal data on them to sync back to the servers and at least keep the data that users have on their device saved.

We’ve heard this from what appears to be several sources and it seems to hold weight. Needless to say it all boils down to one thing: Microsoft did not have a working backup.

How this happens in today’s day and age is beyond belief. Hundreds of thousands of customers that generate millions of dollars in revenue means you back their stuff up, in triplicate. You test these backups regularly, and you move a copy off site that doesn’t get touched except in case of an emergency (i.e. right now). The head of the mobile division (and person in charge of what’s left of Danger) is Roz Ho, who has been at Microsoft for 18 years. You would think she’d know something about how to run a business.

What does this mean for the future of the Sidekick? Unless Microsoft pulls a miracle out of thin air the Sidekick is dead. People are already jumping ship to other phones with this news, and the exposure of how inept Microsoft is when it comes to the mobile world is huge. If Microsoft can’t continue to run Danger, a company that was ground-breaking and solidly built, how can we expect anything from the Windows Mobile department?

There’s an extensive (and well written) article over on RoughlyDrafted about Microsoft, Danger, Windows Mobile, Apple, et all. If you have the time, I highly recommend you read it to get a much fuller picture on the state of things. It’s one of the most comprehensive and up to date “State of the Union” when it comes to Danger/Microsoft/Project Pink/etc. That said, I know not everyone has the time to dive into such an article, so here are some highlighted snippets from a RoughlyDrafted’s Microsoft/Danger source that I thought would be most interesting to Hiptop3.com readers:

# The source says that the details about the Project Pink phones are true and that they wouldn’t be surprised if the photos were an intentional leak, however the rumors about Microsoft working with Verizon and the unhappy employees are definitely from an unhappy (ex) employee.

# What about handset makers?

“Let’s start with the handset makers. Motorola has announced their intention to abandon WinMo for Android, and Palm has already abandoned WinMo in favor of their own WebOS. A few others, such as HTC, have been hedging their bets, announcing their support for Android, but not yet willing to publicly disown WinMo. Now the story has been leaked that Microsoft has been secretly working with Sharp on this (doomed) [Pink] project for over a year, cutting everyone else out of the picture completely.

”You figured out the likely consequences of this over a month ago so I won’t belabor the point other than to say that Sharp, who has been a terrific partner to Danger throughout the Hiptop/Sidekick era, was no doubt hoping that Pink would allow them to expand their phone offerings in the Americas, a dream which has now been dashed due to typical Microsoft blundering. I doubt that either Sharp or any of the other handset OEM’s will ever take Microsoft at their word again.

# Trying to build 4 versions of the Pink phones?

“the Pink planners decided to try to build both UMTS and CDMA phones in both form factors, for a grand total of four different SKUs. None of the other smartphone platform builders were foolish enough to try to build more than one phone at a time, and whoever made this decision didn’t take into account the added complexity involved in coding to support two different radio modules (from two different vendors, no less!) for two completely different radio technologies, not to mention validating and testing the RF performance of four different antennas (since each form factor and radio technology requires its own antenna design). At some point the UMTS project was placed on the back burner, and for all I know it may have already been cancelled, but the damage to the project has been done.”

# Pitting Verizon against T-Mobile

”The final operator who is going to be pissed is T-Mobile, who has been just as loyal of a partner to Danger as Sharp has been. I don’t know exactly what Microsoft has been telling them, but they have no doubt realized that they’ve been cut out of this deal in favor of their largest competitor. What’s worse is that apparently Microsoft has been lying to them this whole time about the amount of resources that they’ve been putting behind Sidekick development and support (in reality, it was cut down to a handful of people in Palo Alto managing some contractors in Romania, Ukraine, etc.).

“The reason for the deceit wasn’t purely to cover up the development of Pink but also because Microsoft could get more money from T-Mobile for their support contract if T-Mobile thought that there were still hundreds of engineers working on the Sidekick platform. As we saw from their recent embarrassment with Sidekick data outages, that has clearly not been the case for some time.

# So what happens now?

“The most likely possibility is that anything worth salvaging from Pink will get folded into the larger WiMo 7 project, where it should have been located in the first place. Another possibility would be another massive layoff and shutting down the Danger offices, but with the economic crunch, I doubt that Microsoft would choose to take the hit of handing out severance packages to all the affected engineers, and breaking the lease on the buildings.”

# And finally, what all of us wish would happen, and the reasons why it probably won’t:

”The smartest course of action (and therefore the one that Microsoft is least likely to take!) would be for them to recognize the value of the original Danger platform that they acquired, and to rebuild the Hiptop/Sidekick into a force to compete with Android (which is, after all, “Sidekick 2.0” in many respects). This won’t happen for three reasons: 1) Microsoft’s irrational hatred of Java, 2) Microsoft’s irrational love of Windows in all of its horrible flavors, and 3) all the Danger folks who loved the Sidekick platform have left or likely will leave soon, and Microsoft has no in-house expertise in Java or the Danger platform.

October has been a bad month for the Sidekick and the future of what’s left of the Danger team. Not only was there the Great Sidekick Data Outage of October ‘09, now we’re hearing that Project Pink, a Microsoft project that the remaining Danger team was put on, might not even be seeing the light of day.

MobileCrunch heard some interesting news from an anonymous source that seems to be legit. Here’s the bullet points:

# As I reported previously, most of the Danger team has been fired or left.
# If any of the Pink devices ship, supposedly they will not have third party application support, i.e. no App Store/Download Catalog.
# The Project Pink employees are not happy. Most of the team still uses Sidekicks or even iPhones and “hate the product”. And here’s the best quote from MC’s source: many feel that the division exists only to “challenge [the Windows Mobile 7 team] and upset them into competing.”
# The project is about 2 years behind schedule and some basic applications are getting cut in an attempt to someday launch this thing.
# The touchscreen on the “Turtle” is not a viable touchscreen as the UI contains too many things on the screen at once and your finger ends up covering half the screen.
# The design of the user interface was done by a 3rd party and now Microsoft is trying to actually build the thing with some difficulty.
# And finally, the worst news for the 2 remaining Project Pink fans: MobileCrunch’s source says that the project “is near death and probably will be canceled.”

Ouch, this is of course all unconfirmed rumors, but if any of it is true… our sympathies to the remaining Danger employees.

Trash can photo from http://www.flickr.com/photos/katmere/. Device images via Gizmodo