San Diego Regional Utility Power Outage

Currently there is a very wide-spread power outage in Southern California affecting about 1.7 million customers. Both M5 Hosting server hosting facilities are located within the outage area. One Wilshire Blvd (downtown Los Angeles) where we peer with over 100 other networks is not in the affected area.

All systems have transitioned to backup generator power. Standard procedure for both data center locations is to have 72 hours of fuel stored on site. Both facilities have 4-hour fuel delivery contracts with multiple suppliers. All network systems are normal. All servers are on-line. Environmental conditions are normal.

Keep an eye here for any updates. If the situation changes this page will be updated.

UPDATE - 19:45, 9/8/11: The SDTC data center facility has received a fuel delivery to top off the generator tanks.

UPDATE - 01:45, 9/9/11: Utility power restored at SDTC at 01:35 local time. The plan is to wait another hour for it to remain stable before taking the facility off generator power.

UPDATE - 02:45, 9/9/11: The SDTC data center has been transitioned back to normal operating status and running on SDG&E Utility power.

UPDATE - 02:55, 9/9/11: Utility power restored at LWDC also, and will be moved off generator in the next 30 to 45 minutes.

UPDATE - 03:41, 9/9/11: All systems have returned to normal. All utility power restored and transitioned off from generator power.

More 10GigE at LWDC tonight...

Tonight's maintenance is to add more 10GigE interfaces to the core network at LWDC and move traffic over to 10GigE links from single- or bonded-1GigE links.

Level 3 in Dallas having issues

Level 3 is having some issues in Dallas which is affecting a large amount of traffic between the East Coast and West Coast of the US, on their network. The issue is causing high CPU on their router there, which has caused packet loss and eventually for it to lose it's BGP sessions and drop routes. This issue happened once yesterday too. When this happens, it takes a few minutes for the affected routes to normalize.

We are keeping a close eye on it. If it begins to look unstable again, we will de-peer from Level 3 entirely until they have regained control of their router in Dallas.

Network Maintenance Announcement - 01:00 to 04:00 Sunday (US/Pacific Time)

This weekend, we will be upgrading our network at our LWDC location.

We are upgrading the network core at LWDC to two new Cisco 6504-Es with SUP720-3BXL supervisors engines, 10GigE, and 1GigE SFP cards. This will replace two smaller Cisco 1GigE switches. We are also turning up a new 10GigE connection between our SDTC and LWDC locations and one of the two planned additional 1GigE carrier connections at LWDC. It's a lot of new stuff!

The migration to this new infrastructure will require a few hours of carefully planned steps followed by testing. This will cause a few brief interruptions in connectivity as routing protocols do their thing and we perform testing after the cut-over to the new systems. Each interruption will be less than 1 minute.

Window Start: Sunday 7/31/11 01:00 PDT
Window End: Sunday 7/31/11 04:00 PDT
Impact: Customers at LWDC may notice a few short interruptions under 1 minute each. SDTC customers will be unaffected.

This message is being sent to customers who have one or more servers at our LWDC location. Customers at SDTC will not be affected.

As always, your feedback, comments and suggestions are greatly appreciated. If you're thinking it, we want to hear it!

Sincerely,
Mike

Update: All work completed and tested successfully, and with time to spare.

Upgrading 6509s at SDTC

During our standard maintenance window this week (Sunday 02:00 to 04:00 PDT), we will be upgrading the redundant power supplies in the 6509s at SDTC to 6000W and adding a 4-port 10GigE card (Cisco WS-X6704-10GE) at SDTC.

Reminder: The Sunday maintenance window is when we do any work that has any risk, but is not expected to impact service. If there is an anticipated impact to service, we will send broadcast email.

Update: This maintenance went smoothly and was successful.

Power Systems Testing on 7/23/11

Our Data Center facility, American Internet Services, will perform a simulated "loss of primary (utility) power" transfer to generator power. This test will transfer the critical load to the emergency systems in order to verify the operation of the current emergency power systems solution. The data center will run for 30 minutes on engine generator during which proper function of each component will be validated. Electrical distribution will then be returned to the normal operational state.

SDTC Data Center - 7/23/2011 04:30 - 07:00 US/Pacific Time
LWDC Data Center - 7/23/2011 12:30 - 19:00 US/Pacific Time

This maintenance is not expected to be service impacting.

Update:The maintenance at SDTC has concluded successfully.

Update:The maintenance at LWDC has concluded successfully.

Power outage at SDTC

A few minutes ago we lost power at our San Diego Tech Center coloration provider's facility. Please keep an eye here for updates as we have them.

UPDATE: Power has been restored. If you are still having any connectivity issues, please send an email to support@m5hosting.com and our engineers will help as quickly as possible. Thank you for your patience!

UPDATE 18:09 PDT: All but 21 servers are back up. All hands are on deck, sorting these out as quickly as possible.

Pacific Tsunami Warning Status for March 11, 2011

San Diego is under a Tsunami Warning for a wave of 1ft - 3ft. The possible initial wave is expected between 08:00 and 09:00 US/Pacific time today. Our SDTC data center is over 200ft above sea-level and our LWDC and OWB locations are over 400ft above sea-level. All facilities have emergency power backup systems and multiple diverse network paths. No local impact to our operations is expected.