Google Maps performed flawlessly for Apple CEO Steve Jobs in today’s Macworld keynote. Yahoo’s Flickr? Utter fail. In a demo of Flickr photos appearing on Apple TV, Flickr was a technical no-show. To those inside the company, this may not have come as a surprise. “

Yahoo has Flickr’s servers on 24-hour watch,” says a former Yahoo employee. Something going wrong is practically the default state for the site, which has grown beyond its technical capabilities. Flickr was down for four hours over the weekend. With founder Stewart Butterfield on paternity leave, and not expected to return to Flickr when his leave ends — word is he might seek another role within Yahoo instead — Flickr’s staff strikes some as dispirited. And other Yahoo employees would rather gripe about Flickr than fix its code.

Millions of text messages were sent simultaneously at midnight on New Year’s. Many were delayed, or didn’t arrive at all. A delay in holiday greetings is not a big deal, but what about during a real emergency? Emergency personnel and government officials are automatically given priority on landline and cellular networks, leaving the average consumer in the lurch. After 9/11, cell-phone traffic in New York was at a standstill for days. Cellular networks, like highways, aren’t designed to have everyone use them at once. When everyone tries to make a call at once, for a holiday or emergency, communication breaks down. The communications infrastructure, as it is currently designed, will never be able to handle calling patterns thousands of times heavier than normal. Your best bet? Send an IM. (Photo by PhotoOptik)

Apple is in talks with Japan’s top cell-phone company, NTT DoCoMo and with Softbank, the No. 3 carrier in the market. Reuters spoke to a source inside DoCoMo who said “the negotiations are not going smoothly, as Apple’s conditions are extremely hard to meet.” No surprise there. Apple likely wants similar terms to ones that AT&T and Deutsche Telekom agreed to: a cut of the subscription fees and a hefty markup on the iPhone itself. Apple had similar negotiations in other countries, in which Jobs infamously referred to prospective partners as “girlfriends.” One of these companies will end up married to the iPhone, likely for the price that Apple quotes. When it comes to for-richer-or-for-poorer, Apple usually picks “for richer.” (Photo by AP/Paul Sakuma)

Source:Valleywag

Right before Penthouse acquired Andrew Conru’s Adult FriendFinder for $500 million, a rumor spread that FriendFinder was making offers itself to buy smaller porn sites. Sources told Adult Entertainment Today AFF made offers on two sites that might not have large profits but “have attracted media attention and large numbers of regular visitors.” Will the rollup strategy continue under Penthouse, or bring FriendFinder’s nascent buying spree to an end? Let us know if Conru has made you an offer you can’t refuse.

Source:Valleywag

A relatively harmless worm has rampaged through Google’s social network, Orkut. You probably haven’t noticed. That’s because Orkut, while popular in Brazil and India, is an also-ran in the U.S. Hundreds of thousands of Orkut users saw their accounts overwhelmed by spam on their “scrapbooks,” Orkut’s equivalent of Facebook’s Wall.

The worm reportedly uses JavaScript and Flash code to create a new scrapbook entry, containing a New Year’s message in Portuguese accompanied by a Flash ad. Once infected, the code propagates to your friends on the network, simply by their visiting your profile page. The worm may be related to a similar vulnerability discovered last year and supposedly fixed by Google, but details are scarce. It appears Google has responded quickly. Too bad. If Google had let the worm rampage, maybe some American users might actually hear about Orkut for the first time.

Ten years ago, on December 17, 1997, Jorn Barger coined the term “web log” for a webpage where an author “logs” the other webpages they find interesting. Since then, blogging has become serious business for some and a place for nonsensical blabber for others. Though Dave Winer frequently claims that he was the first to launch a weblog at Scripting News, it was really just an online archive of columns rejected by HotWired. Jorn Barger had the first true weblog worthy of the name at Robot Wisdom.

Source:Valleywag

“It is possible that the application does not exist.” Well, granted, if you took enough math and philosophy classes, that statement is 100 percent correct.

John Battelle dug up this gem that we missed from Om Malik’s GigaOm show. Skip ahead to nine minutes in for Om’s interview with Google’s Marissa Mayer. Om’s sidekick Joyce Kim starts it off by asking Mayer if she’s worried about startup search engines challenging Google. Short answer: No. But impress us enough and we’ll buy you. That’s not evil, is it?

Source:Valleywag

An accident near Rackspace’s Dallas datacenter sparked a late-night Web crisis, downing Internet service providers from Texas to California and bringing down 37signals’ Web-based software suite, on which many startups depend for coordinating their work. But Rackspace worked fast to fix the cause of the outage — balky chillers which failed to start when switched to backup power, causing the datacenter to overheat — and by midnight, most of Rackspace’s hosted websites were back online. Here’s the apology note from Rackspace CEO Lanham Napier, forwarded to us by a customer.

Dear [redacted],

It has been a long day and hard day here at Rackspace. We know it’s been a long day for you. We are deeply sorry for the events that have taken place at our Dallas/Fort Worth data center. Your satisfaction is what every Racker works towards every day, especially today.

To those affected by the outage, I apologize. We all apologize. We understand the frustration and uncertainty you have gone through. We take full responsibility for what happened and we will work with you to reach a remedy that satisfies you.

We have made the latest status update to the myrackspace.com customer portal. Please continue to visit it regularly for our most recent updates.

As always, your Account Manager and Support Team are available to help in any way. Likewise, please feel free to contact me if you have any questions or needs.

Sincerely,

Lanham Napier
CEO
Rackspace

The most recent email to customers from Rackspace, the Texas Web host whose datacenter outage in Dallas downed some of the Web’s most useful websites:

Nov. 12th 11:30PM CST — As of 10:50 p.m. CST, all DFW servers that were proactively powered down earlier this evening, to avoid overheating, have now been powered back up. The Data Center Engineering team has been working to resolve the power issues caused by tonight’s traffic incident. The team is preparing to transfer machines affected by tonight’s power outage from generator power back to utility power. The servers and devices that were affected by the unrelated event over this past weekend will remain on generator power. We anticipate transferring the machines affected this evening back to utility power within the hour and expect the transfer to be non-disruptive to customer environments. We apologize again for the inconvenience these events have caused and have all hands on deck working fanatically to minimize the impact on your business.

Previously: Rackspace outage was third in two days

Source:Valleywag

There are a lot of posts popping up from various Rackspace customers apologizing to their customers and readers for the Web host’s downtime caused by an errant truck. Here is a selection:

37signals

Will be using this situation as both a wake-up call and a learning experience. While our systems are engineered to chug through major failure, this “perfect storm” chain of events beat both our set-up and our data center’s sophisticated back-up systems. We will work hard to further diversify our systems in order to make an future downtime event like this even more rare.

Threadless

As you noticed Threadless took a little nap this evening. Apparently a truck jumped a railing and attacked our data center. I think their may have even been an explosion. I hope so.

Laughing Squid

This is the first time we have had power outage like this in the 9 years that we have been in hosting (8 of which at Rackspace).

And here’s Rackspace’s public response:Rackspace

*crickets*

The most recent email to customers from Rackspace, the Texas Web host whose datacenter outage in Dallas downed some of the Web’s most useful websites:

Nov. 12th 11:30PM CST — As of 10:50 p.m. CST, all DFW servers that were proactively powered down earlier this evening, to avoid overheating, have now been powered back up. The Data Center Engineering team has been working to resolve the power issues caused by tonight’s traffic incident. The team is preparing to transfer machines affected by tonight’s power outage from generator power back to utility power. The servers and devices that were affected by the unrelated event over this past weekend will remain on generator power. We anticipate transferring the machines affected this evening back to utility power within the hour and expect the transfer to be non-disruptive to customer environments. We apologize again for the inconvenience these events have caused and have all hands on deck working fanatically to minimize the impact on your business.

Previously: Rackspace outage was third in two days

If you’re wondering what an exploding transformer is like, look no further:

Source:Valleywag

This was sent to us by a Rackspace customer. It is from my.rackspace.com — Rackspace’s customer support website. It reveals this was the third power issue in two days: one Sunday morning, one Monday morning and another this evening. Not all customers were affected by all three outages, but three outages at one hosting facility is not good.

Monday Nov. 12th 8:00PM CST — Thank you for your patience today as we work through a root cause analysis of the power issues in the DFW data center. The Data Center Engineering team is continuing to work on the plan to move back to utility power without any further interruption in service. Rackspace will notify you in advance before we move to switch back to utility power.
In the meantime, here is a brief timeline of events:

Sunday, Nov. 11, 2007
4:19 a.m. CST — A problem in the internal utility power distribution grid caused an outage to cabinets in one section of the DFW data center.
6:49 a.m. CST — Power was fully transferred to generator power. Based on building monitoring systems, outage times varied for every customer. DC engineering worked to isolate the internal utility problem and restore the integrity of the internal distribution system.
6:32 p.m. CST — A separate incident occurred when a breaker in the generator power grid tripped, causing one of the Power Distribution Units (PDUs) in the same section of the DFW Datacenter to fail, affecting a much smaller group of the customers in this section. All customer devices with dual power supplies in this section of the datacenter remained online and were not affected. Customer devices with single power supplies in this area were affected. Data Center technicians immediately acted to minimize the impact on these customers by moving these devices manually to alternate power supplies - resulting in just a few minutes of downtime.
7:40 p.m. CST — The breaker problem was diagnosed and resolved, bringing the down PDU back online.

Monday, Nov. 12, 2007
4:00 a.m. CST — The Data Center engineering team had the initial utility distribution grid realigned and resynchronized. All systems reported ready for operation.
4:30 a.m. CST — Transfer of power was initiated and affected devices were slowly moved off of generator power and back to internal utility distribution power.
5:10 a.m. CST — Transfer of power was completed.
5:25 a.m. CST — Unfortunately, the internal distribution grid failed again. Data Center engineering was able to transfer all affected devices back to generator power in under 15 minutes.
5:40 a.m. CST — All affected devices were back on generator power. The Data Center environment is stable and is designed to be able to run indefinitely on generator power. Data Center engineering is continuing to diagnose the problem and engaging all vendors onsite.

We will continue to provide updates via the MyRackspace portal. In addition, you will receive notification of any maintenance windows. We are committed to supporting your business and minimizing any impact on your hosted environment.

Here is a another report that was sent to us regarding this evening’s problems.

DFW Datacenter Power Update

Nov. 12th 8:30PM CST

In a completely unrelated incident to this weekend’s power problems in DFW, a traffic accident caused damage to a power transformer which provides utility power to our DFW data center. Here is the current sequence of events:

* At approximately 6:00 p.m. CST utility power was lost to the DFW data center
* Power automatically switched over to backup generators without disrupting service for any customers
* When generator power was established two chillers within the data center failed to start back up
* Utility power was re-established through a secondary utility source
* As a result of temporary data center temperature increases, we proactively shutdown a number of customer servers to protect them from overheating

At this point, the chillers are back up and running and we are operating on generator power throughout the data center. We have contractors on site to repair the damage and will be in contact with all customers who have been affected by this outage. We apologize for any disruption to your business operations and will work diligently to restore your service.

Previously:
Rackspace was “most reliable” webhost in September 2007 [Internal]
Truck driver in Texas kills all the websites you really use [Internal]
Rackspace outage affects Texas ISP [Internal]

Source:Valleywag

A tipster writes that his ISP blamed a service disruption on Rackspace issues.

Hey, that Rackspace outage? It also caused internet service disruptions as far away as College Station, Texas… where I am … my intertubes went down last night for a while, and then tonight they went out again. I just thought my ISP sucked, but when I finally got through to them they blamed Rackspace for the outages … Apparently a bunch of services (DNS, RADIUS, etc) for SuddenLink are hosted out of there.

Previously:
Rackspace was “most reliable” webhost in September 2007 [Internal]
Truck driver in Texas kills all the websites you really use [Internal]

If you’re wondering what an exploding transformer is like, look no further:

Source:Valleywag

This was sent to us by a Rackspace customer. It is from my.rackspace.com — Rackspace’s customer support website. It reveals this was the third power issue in two days: one Sunday morning, one Monday morning and another this evening. Not all customers were affected by all three outages, but three outages at one hosting facility is not good.

Monday Nov. 12th 8:00PM CST — Thank you for your patience today as we work through a root cause analysis of the power issues in the DFW data center. The Data Center Engineering team is continuing to work on the plan to move back to utility power without any further interruption in service. Rackspace will notify you in advance before we move to switch back to utility power.
In the meantime, here is a brief timeline of events:

Sunday, Nov. 11, 2007
4:19 a.m. CST — A problem in the internal utility power distribution grid caused an outage to cabinets in one section of the DFW data center.
6:49 a.m. CST — Power was fully transferred to generator power. Based on building monitoring systems, outage times varied for every customer. DC engineering worked to isolate the internal utility problem and restore the integrity of the internal distribution system.
6:32 p.m. CST — A separate incident occurred when a breaker in the generator power grid tripped, causing one of the Power Distribution Units (PDUs) in the same section of the DFW Datacenter to fail, affecting a much smaller group of the customers in this section. All customer devices with dual power supplies in this section of the datacenter remained online and were not affected. Customer devices with single power supplies in this area were affected. Data Center technicians immediately acted to minimize the impact on these customers by moving these devices manually to alternate power supplies - resulting in just a few minutes of downtime.
7:40 p.m. CST — The breaker problem was diagnosed and resolved, bringing the down PDU back online.

Monday, Nov. 12, 2007
4:00 a.m. CST — The Data Center engineering team had the initial utility distribution grid realigned and resynchronized. All systems reported ready for operation.
4:30 a.m. CST — Transfer of power was initiated and affected devices were slowly moved off of generator power and back to internal utility distribution power.
5:10 a.m. CST — Transfer of power was completed.
5:25 a.m. CST — Unfortunately, the internal distribution grid failed again. Data Center engineering was able to transfer all affected devices back to generator power in under 15 minutes.
5:40 a.m. CST — All affected devices were back on generator power. The Data Center environment is stable and is designed to be able to run indefinitely on generator power. Data Center engineering is continuing to diagnose the problem and engaging all vendors onsite.

We will continue to provide updates via the MyRackspace portal. In addition, you will receive notification of any maintenance windows. We are committed to supporting your business and minimizing any impact on your hosted environment.

Here is a another report that was sent to us regarding this evening’s problems.

DFW Datacenter Power Update

Nov. 12th 8:30PM CST

In a completely unrelated incident to this weekend’s power problems in DFW, a traffic accident caused damage to a power transformer which provides utility power to our DFW data center. Here is the current sequence of events:

* At approximately 6:00 p.m. CST utility power was lost to the DFW data center
* Power automatically switched over to backup generators without disrupting service for any customers
* When generator power was established two chillers within the data center failed to start back up
* Utility power was re-established through a secondary utility source
* As a result of temporary data center temperature increases, we proactively shutdown a number of customer servers to protect them from overheating

At this point, the chillers are back up and running and we are operating on generator power throughout the data center. We have contractors on site to repair the damage and will be in contact with all customers who have been affected by this outage. We apologize for any disruption to your business operations and will work diligently to restore your service.

Previously:
Rackspace was “most reliable” webhost in September 2007 [Internal]
Truck driver in Texas kills all the websites you really use [Internal]
Rackspace outage affects Texas ISP [Internal]