jenyum's Blog

May. 31, 2008 at 8:02pm

Houston, We Have a Problem

OH NOZ I need a new hobby

Due to a generator fire near the facility that houses our servers in Houston, Tacomamama is experiencing a temporary outage.  Hopefully not for more than one evening.


comments [7]  |  posted under servers, Tacoma, teh internets

Comments

by jenyum on 5/31/2008 @ 8:53pm
Update: Sounds Like a Real Fun Day

Mebbe not until tomorrow afternoon.

From the Planet Forums:

This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.

We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.

This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.

We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.

There is no impact in any of our other five data centers.

I am sorry that this accident has occurred and I apologize for the impact.

by KevinFreitas on 5/31/2008 @ 9:15pm
Eesh. Sorry to hear about that, jen. Sounds like a royal mess down there. Hope you have everything up and humming soon. In the meantime, any good Tacoma- and/or kid-related stuff going on with you?

by jenyum on 5/31/2008 @ 10:19pm
We went to the new Animal Avenue exhibit at the zoo. No Meerkats yet, due to a minor glitch in quarantine, but lemurs, naked mole rats, and sand pit as promised.

Lemurs getting used to the new exhibit and mostly interested in grooming, but it looks like there are about 10 and they are all of the ring-tailed, zaboomafoo variety. Very cute.

Then we went to Brain Candy at Grant. Tacoma Astronomical Society stole the show with stomp rocket demonstration. Way more exciting than bubbles, apparently. They do free rocket parties for local schools, and I highly recommend them.

by jenyum on 6/1/2008 @ 11:30am
Latest word is that testing will begin at 5:00 central time, which will last for about four hours. After that, servers will slowly be brought back online. Your guess is as good as mine as to how long this will take.

I'm just glad it's not a big holiday weekend.

by jenyum on 6/1/2008 @ 7:28pm
No good news.

The Planet is being very quiet about how the testing went.

Think happy server thoughts for me.

Edited to add: I like one of their latest press releases,

"The Planet Announces Server Blowout Sale"

yeah.

by intacoma on 6/1/2008 @ 7:50pm
you have a backup?

by jenyum on 6/1/2008 @ 8:00pm
not as current as I would like.