Thursday, September 8, 2011

San Diego Regional Utility Power Outage

Currently there is a very wide-spread power outage in Southern California affecting about 1.7 million customers. Both M5 Hosting server hosting facilities are located within the outage area. One Wilshire Blvd (downtown Los Angeles) where we peer with over 100 other networks is not in the affected area.

All systems have transitioned to backup generator power. Standard procedure for both data center locations is to have 72 hours of fuel stored on site. Both facilities have 4-hour fuel delivery contracts with multiple suppliers. All network systems are normal. All servers are on-line. Environmental conditions are normal.

Keep an eye here for any updates. If the situation changes this page will be updated.

UPDATE - 19:45, 9/8/11: The SDTC data center facility has received a fuel delivery to top off the generator tanks.

UPDATE - 01:45, 9/9/11: Utility power restored at SDTC at 01:35 local time. The plan is to wait another hour for it to remain stable before taking the facility off generator power.

UPDATE - 02:45, 9/9/11: The SDTC data center has been transitioned back to normal operating status and running on SDG&E Utility power.

UPDATE - 02:55, 9/9/11: Utility power restored at LWDC also, and will be moved off generator in the next 30 to 45 minutes.

UPDATE - 03:41, 9/9/11: All systems have returned to normal. All utility power restored and transitioned off from generator power.