‘CrackBerry’ outage forced millions to go cold turkey

An emptiness shook John Kleinschmidt’s world at 8:15 p.m. EDT Tuesday. His BlackBerry, usually buzzing with dozens of e-mail messages an hour, was silent.

“It felt like a tremor,” said Kleinschmidt, an engineer at a software-development company in Troy, Mich., who gets an average of 250 emails on his BlackBerry each day. He is so devoted to responding immediately that he recently tapped away on the gadget’s keyboard during his wife’s stepfather’s funeral; during showers, he keeps it within view but dry.

Panicked, he pulled out the battery several times, trying to reset the device. No dice. He stayed up the entire night, calling the BlackBerry help line every hour. No luck. Then, he logged on to his PC and began reaching out to other users to see what was going on.

He found a world of bereft people suffering the same kind of separation anxiety because of a North American outage of Research In Motion Ltd.’s BlackBerry.

The blackout hit millions of users of the popular wireless e-mail device for at least nine hours Tuesday night and Wednesday morning. Even White House spokesman Tony Fratto expressed frustration, joking with reporters that the White House had started a “twelve-step group” to cope with the withdrawal.

While dealing with the unprecedented failure of its service, RIM said Wednesday morning that service for most customers was restored overnight, adding the “root cause is currently under review.” The source of the problem remains largely unknown and RIM hasn’t provided a detailed explanation, even to clients. A spokeswoman for the company said it’s unclear when one will come.

RIM, based in Waterloo, Ontario, has experienced outages before, typically lasting a few hours or less. But this massive failure hit the company as its user base added one million new accounts in its most recent quarter. The number of BlackBerry users is still relatively small, with eight million customers world-wide, compared with more than one billion cellphone users.

But those who use the device — dubbed the CrackBerry — are often devotees who use it around the clock and include senators, investment bankers, Hollywood and media types, technology workers and lately soccer moms and other consumers — many of whom were just as upset as Kleinschmidt to see that e-mails weren’t trickling in.

An online poll of 70 large companies by expense-management service ProfitLine found that 81 percent of respondents had some disruption to operations, with 44.5 percent reporting “moderate or substantial” impact on productivity.

“It’s symptomatic of the increasing vulnerability of our economy,” adds Edwin L. McClendon, executive vice president of the investment-banking division at Terra Nova Financial LLC in Chicago. He received more than 30 emails hours after they were sent and had to reschedule two conference calls. “We now have network-centric work habits.”

Analysts say that epicenter of the problem was almost certainly related to the network operations center, the “post office” that receives emails from email servers and pushes them out to subscribers’ BlackBerrys. Unlike competitors, all emails sent through the BlackBerry service go through the center, which switches the data over to the carriers’ network and has multiple geographic locations. Such closed architecture, long touted as the secret behind the speed and security of BlackBerry’s email service, make it vulnerable to systemwide outages, analysts and competitors say.

Tuesday night, the West Coast was wrapping up its workday when the outage began. Lori Sale, a senior agent at the Los Angeles-based talent agency International Creative Management, was at her 14-year-old son’s baseball game when her BlackBerry stopped working. She first realized something was wrong at about 5:15 p.m. PDT, when she noticed she had received no emails on her BlackBerry since 5:03 p.m. Ms. Sale, who estimates she receives more than 500 emails a day, became alarmed when her boss then called and asked why she hadn’t responded to his email sent four minutes earlier about a sudden problem.

She had to leave the game and make the 20-minute drive to the office Sale ended up working on her office computer until 9 p.m. Wednesday morning when she checked her BlackBerry, she was hit by a deluge of 14 hours of emails.

Wireless carriers that offer BlackBerry service started getting complaints from subscribers soon after problems occurred. They quickly realized the problem was RIM’s and not theirs — cellphone and text-messaging services on BlackBerrys were still working, after all — but customers often blame carriers for any problems with their wireless handsets, so they swung into action.

The Sprint Nextel Corp. network team worked throughout the night to monitor the situation, exchanging a flurry of calls with RIM, a spokeswoman said. At Verizon Wireless, a joint venture between Verizon Communications Inc. and Vodafone Group PLC, the network operations team blasted an alert at 8:49 p.m. EDT via text message, email and voice-mail to top executives.

E-mail service began working for some earlier than for others. After Lucas Evans, an associate with Apollo Real Estate Advisors in Los Angeles, noticed he hadn’t received responses to several urgent emails about an impending deal, he started some self-troubleshooting, sending emails to himself from a personal account, turning his AT&T Inc. BlackBerry on and off and, the time-honored solution to all technical problems, trying to shake it back to health. But the 26-six-year-old Evans, who had gone to bed with business pending, was rudely awakened at 1 a.m. when service was restored and his BlackBerry, which he had forgotten to turn off, started vibrating loudly on his night table. “I was up for hours answering all those e-mails,” he said.