M5Hosting System Status

San Diego Regional Utility Power Outage

2011-09-08T17:54:00.000-07:00

Currently there is a very wide-spread power outage in Southern California affecting about 1.7 million customers. Both M5 Hosting server hosting facilities are located within the outage area. One Wilshire Blvd (downtown Los Angeles) where we peer with over 100 other networks is not in the affected area.

All systems have transitioned to backup generator power. Standard procedure for both data center locations is to have 72 hours of fuel stored on site. Both facilities have 4-hour fuel delivery contracts with multiple suppliers. All network systems are normal. All servers are on-line. Environmental conditions are normal.

Keep an eye here for any updates. If the situation changes this page will be updated.

UPDATE - 19:45, 9/8/11: The SDTC data center facility has received a fuel delivery to top off the generator tanks.

UPDATE - 01:45, 9/9/11: Utility power restored at SDTC at 01:35 local time. The plan is to wait another hour for it to remain stable before taking the facility off generator power.

UPDATE - 02:45, 9/9/11: The SDTC data center has been transitioned back to normal operating status and running on SDG&E Utility power.

UPDATE - 02:55, 9/9/11: Utility power restored at LWDC also, and will be moved off generator in the next 30 to 45 minutes.

UPDATE - 03:41, 9/9/11: All systems have returned to normal. All utility power restored and transitioned off from generator power.

More 10GigE at LWDC tonight...

2011-08-21T00:34:00.000-07:00

Tonight's maintenance is to add more 10GigE interfaces to the core network at LWDC and move traffic over to 10GigE links from single- or bonded-1GigE links.

Level 3 in Dallas having issues

2011-08-01T11:31:00.000-07:00

Level 3 is having some issues in Dallas which is affecting a large amount of traffic between the East Coast and West Coast of the US, on their network. The issue is causing high CPU on their router there, which has caused packet loss and eventually for it to lose it's BGP sessions and drop routes. This issue happened once yesterday too. When this happens, it takes a few minutes for the affected routes to normalize.

We are keeping a close eye on it. If it begins to look unstable again, we will de-peer from Level 3 entirely until they have regained control of their router in Dallas.

Network Maintenance Announcement - 01:00 to 04:00 Sunday (US/Pacific Time)

2011-07-29T03:20:00.000-07:00

This weekend, we will be upgrading our network at our LWDC location.

We are upgrading the network core at LWDC to two new Cisco 6504-Es with SUP720-3BXL supervisors engines, 10GigE, and 1GigE SFP cards. This will replace two smaller Cisco 1GigE switches. We are also turning up a new 10GigE connection between our SDTC and LWDC locations and one of the two planned additional 1GigE carrier connections at LWDC. It's a lot of new stuff!

The migration to this new infrastructure will require a few hours of carefully planned steps followed by testing. This will cause a few brief interruptions in connectivity as routing protocols do their thing and we perform testing after the cut-over to the new systems. Each interruption will be less than 1 minute.

Window Start: Sunday 7/31/11 01:00 PDT
Window End: Sunday 7/31/11 04:00 PDT
Impact: Customers at LWDC may notice a few short interruptions under 1 minute each. SDTC customers will be unaffected.

This message is being sent to customers who have one or more servers at our LWDC location. Customers at SDTC will not be affected.

As always, your feedback, comments and suggestions are greatly appreciated. If you're thinking it, we want to hear it!

Sincerely,
Mike

Update: All work completed and tested successfully, and with time to spare.

Upgrading 6509s at SDTC

2011-07-23T16:17:00.000-07:00

During our standard maintenance window this week (Sunday 02:00 to 04:00 PDT), we will be upgrading the redundant power supplies in the 6509s at SDTC to 6000W and adding a 4-port 10GigE card (Cisco WS-X6704-10GE) at SDTC.

Reminder: The Sunday maintenance window is when we do any work that has any risk, but is not expected to impact service. If there is an anticipated impact to service, we will send broadcast email.

Update: This maintenance went smoothly and was successful.

Power Systems Testing on 7/23/11

2011-07-23T00:16:00.000-07:00

Our Data Center facility, American Internet Services, will perform a simulated "loss of primary (utility) power" transfer to generator power. This test will transfer the critical load to the emergency systems in order to verify the operation of the current emergency power systems solution. The data center will run for 30 minutes on engine generator during which proper function of each component will be validated. Electrical distribution will then be returned to the normal operational state.

SDTC Data Center - 7/23/2011 04:30 - 07:00 US/Pacific Time
LWDC Data Center - 7/23/2011 12:30 - 19:00 US/Pacific Time

This maintenance is not expected to be service impacting.

Update:The maintenance at SDTC has concluded successfully.

Update:The maintenance at LWDC has concluded successfully.

Power outage at SDTC

2011-06-22T14:23:00.000-07:00

A few minutes ago we lost power at our San Diego Tech Center coloration provider's facility. Please keep an eye here for updates as we have them.

UPDATE: Power has been restored. If you are still having any connectivity issues, please send an email to support@m5hosting.com and our engineers will help as quickly as possible. Thank you for your patience!

UPDATE 18:09 PDT: All but 21 servers are back up. All hands are on deck, sorting these out as quickly as possible.

Pacific Tsunami Warning Status for March 11, 2011

2011-03-11T02:12:00.000-08:00

San Diego is under a Tsunami Warning for a wave of 1ft - 3ft. The possible initial wave is expected between 08:00 and 09:00 US/Pacific time today. Our SDTC data center is over 200ft above sea-level and our LWDC and OWB locations are over 400ft above sea-level. All facilities have emergency power backup systems and multiple diverse network paths. No local impact to our operations is expected.

PremiumNet issues

2010-01-10T21:42:00.000-08:00

We are experiencing packet loss between PremiumNet and some remote networks on the Internet. This involves traffic that ingresses or egresses AmericanIS.net's network via Level 3 Communications. However, our ValueNet connection to Level 3 does not appear to be affected currently.

We are in constant communication with AmericanIS currently, we will update you as soon as we have additional information.

Thank you for your patience.

Update - 1/14/10 00:52:
The following post-incident follow-up will explain the root cause, what was done and what is planned to prevent this from happening again.

On Sunday January 10th, between 7:05pm and 9:40pm some customers noticed some difficulty connecting to some applications on our PremiumNet network from some locations. Not all customers or applications were affected, and not from all locations. Our ValueNet network was not affected.

All devices were pingable in all directions from all locations we were able to test from outside of our network. Our ping-based monitoring systems did not show packet loss, latency or jitter. However, some customers had difficulty when both of two conditions were met: 1) The data path either ingressed or egressed AIS via their 10 GigE connection with Level 3. 2) The packets were "full" (ie: 1500 Bytes). Small packets seemed not to be affected (ie: VoIP, DNS, games, etc).

Pings are small ICMP packets, and pings with the typical default packet sizes did not reveal the problem. However, all customer trouble reports were from customers who had difficulty only through paths that included Level 3 at the last hop outside of our upstream provider American Internet Services (AIS). Small web pages, DNS lookups, VoIP, games, and other applications which use small packets did not experience trouble. However, video conferencing, file downloads, "heavy" web pages, email (especially with attachments), etc., which transited Level 3 experienced dropped connections, and other bad behavior.

We buy our "PremiumNet" from American Internet Services (AIS), who is also our colocation facility. We have two redundant GigE connections to them for PremiumNet. Their network is built to be completely redundant and fault tolerant, just as we would build it. The failure in this case was not any component on our network, not on the AIS network or in between. According to AIS, the failure was a relatively new 10 GigE connection between AIS and Level 3. This new 10 GigE connection is an addition to two existing 1GigE connections between AIS and Level 3 at this facility. Eventually, the two 1GigE connections are to be replaced with two 10 GigE connections. This network at this site also has multiple other 10 GigE connections to transport to two other AIS facilities and then on to multiple other tier-1 carriers. The transport links to other facilities are *not* provided by Level 3.

According to information from AIS, poor optical performance on the 10 GigE fiber optic connection to Level 3 was the root cause of the issued on Sunday night. I have spoken directly with AIS Engineers, I have asked my questions both as an engineer and on your behalf as principal of M5Hosting. Most of my questions have been answered. I believe that the post incident handling of this event has been correct and appropriate. However, I am not altogether happy with the duration of the event and that we were unable to work around it for PremiumNet customers.

One of our projects for the new year is to change the way we connect with AIS and still provide top quality, route-optimized bandwidth. While AIS has always provided a solid and reliable network, we feel we need to address two key issues concerning our connectivity through AIS: 1) The cost. The cost of our "pure" AIS bandwidth option (PremiumNet) is far too high to be competitive with other "Premium" solutions. Our growth over the last several years gives us the scale needed to buy better and engineer bigger, as we have done with our ValueNet option. We have fantastic announcements to make within 2 months for our PremiumNet customers. 2) The ability to route around an AIS problem, if needed. We can drop any other network provider on our "ValueNet" offering without any significant effect to service. As great as AIS usually is, we need to be able to drop our peering session with them as well in the event of trouble. As our sole route in and out for PremiumNet, we can not route around AIS at this time. This will change very soon. An announcement will be made when this improvement has been made.

Our ValueNet network is directly connected to Level 3. We were not able to reproduce the same poor behavior on our own Level 3 connection. But, we had our finger on "the button". We were ready to drop our direct Level 3 connection to our ValueNet network about 20 minutes after the issue began, in the event that our connection to Level 3 began to act up too. Ultimately, AIS did de-peer Level 3 via the 10GigE connection to resolve the issue. However, not until 2 hours and 40 minutes after the problem began. The additional 2+ hours of time for AIS to take this action is the part that I am not satisfied with. If this event had happened after our planned network changes over the next month or so, we would have dropped our connection with AIS within 20 minutes, and this event would have been 20 minutes total for our PremiumNet customers. Work has already been planned which will enable us to drop AIS if needed during an event on their network.

This work will begin later this month. The related hardware purchases were made on 12/31/09, but not been received. Additional carrier connectivity is on the way as well. We will make more detailed announcements as we have details to report.

The most recent and most technically detailed message from AIS regarding this event follows:

To Our Valued Clients,

In a continued effort to provide our clients with the best state-of-the-art data center infrastructure, this is to notify you that we will open a maintenance window for Level3 re-peering on Wednesday, 1/13/10 at 10:00pm.

Type of Maintenance: Level3 re-peering

Location: All

Purpose: AIS will be reintroducing a Level3 peering link, which was removed by AIS Network Engineers on 1/10/10 (event #50289)

Window Start: Wednesday, 1/13/10 - 10:00pm PST
Window End: Thursday, 1/14/10 - 12:00am PST

Current Status: This scheduled maintenance window is now active.

Service Impact: As with any network maintenance, while highly unlikely, there is a possibility that something unexpected may occur during the maintenance process. Please rest assured that we will immediately address such occurrences, and will send a notification upon the successful completion of this maintenance.

Schedule: The work is scheduled to begin at 10:00pm on Wednesday, January 13th and should be completed within 2 hours time. Should additional time be required, notice will be provided and the maintenance window will be expanded. A notice will be sent at the completion of the maintenance window.

Testing & Planning: All testing and planning being conducted during this window is part of a pre-defined checklist designed by the AIS Network Engineering team.

Regression Planning: The AIS Network Engineering team will be on-site managing this maintenance window. Should any issues arise, all equipment will be placed back to previous configuration and the maintenance will be postponed until the issue is resolved.

Updated Event #50289 Root Cause Analysis: Between 7:00pm - 9:40pm on Sunday, 1/10/10, some clients may have noticed intermittent high latency and packet loss to certain Level3-connected sites. The ultimate root cause of this event, as provided by Level3, was optical degradation of the fiber circuit as a result of fluctuating light levels at the SDTC end of the 10Gig optical hand-off.

Updated Event #50289 Timeline: At approximately 7:00pm on Sunday, 1/10/10, AIS Client Support began to notice a pattern of network routing anomalies, which introduced intermittent latency for some clients. After testing, this issue was escalated to the Engineering department at 7:43pm, who isolated the fault to Level3. AIS escalated this issue to the Level3 engineering team, who were able to reproduce intermittent packet loss across certain Level3 transit. After additional troubleshooting, the issue was isolated to one 10Gbps fiber hand-off that was deemed physically faulty. At 9:40pm, the unstable Level3 peering link was removed from the bandwidth blend, clearing all issues.

Resolution: Level3 felt it was necessary to put the fiber circuit through the same network testing process that was conducted in the initial provisioning phase of this circuit. On-site engineers cleaned and replaced the fiber connectors, then remeasured and saw that the light levels were now stable. Level3 then connected the optics to a test set and tested throughput on the connection, producing no errors. AIS and Level3 independently tested network connectivity to verify that all latency and mtu-related packet loss has disappeared.

Brief Interruptions on PremiumNet

2009-08-27T13:18:00.000-07:00

We have had reports of intermittent network connectivity to some hosts on PremiumNet (but not ValueNet). We have identified the cause and implemented a fix. We are performing an analysis to determine the events that triggered this event.

There are currently no known connectivity issues remaining. A root cause explanation will be sent to all customers who have opened a support ticket regarding this issue.

Network Maintenance Window Tonight

2009-03-19T22:21:00.000-07:00

    We will be returning our network systems to normal fully redundant 
operating status tonight. Once this is done we will test the fail-over 
and fault-tolerance. This may cause brief (1 to 15 second) pauses on 
the network. The purpose is to ensure that all systems perform as 
intended.

   After the power incident yesterday, the M5Hosting "PremiumNet" 
network systems performed in an unexpected manner. All systems have 
been previously tested and performed fail-over flawlessly. This 
testing to take place tonight is to make sure that this is still the 
case, given the unexpected behavior yesterday. The unexpected behavior 
yesterday took place between our border routers and AIS's routers 
which face us when power was restored. Performing a test of the 
fail-over systems tonight will help determine if the network issues 
were due to the effects of the abrupt power interruption on the 
network devices, or a configuration of the network devices. Both M5 
and AIS Network Engineers will be on hand for this work.

   The M5Hosting "ValueNet" was not affected once power was restored, 
is running in normal operational status and should not be affected by 
tonights testing.
   The vast majority of our customers are on "PremiumNet". If you do 
not know which you are on, you are most likely on PremiumNet.

Window Start: Friday March 20, 2009 02:00 US/Pacific (GMT -7)
Window End: Friday March 20, 2009 04:00  US/Pacific (GMT -7)
Duration: Customers on M5Hosting's "PremiumNet" may notice a few very 
brief interruptions of a few seconds in length.

   As always, I want to hear from you regarding this email, and how 
we are doing... or anything at all.

Post Incident Update - Facility Power

2009-03-19T22:19:00.000-07:00

    I have attached the post-incident report from the data center 
facility provider to this email below. As one of our most important 
vendors, we keep engaged with American Internet Services (AIS) plans, 
upgrades, and operational changes. Naturally, as such a critical 
supplier, we scrutinize their people's actions, technology, and 
facility, especially when it comes to service impacting events. It is 
our job to provide the best service possible to our customers, and 
that requires us to be a tough and determined customer of AIS, on your 
behalf.

   While power has been restored and the facility has resumed normal 
operating status, this event is not "over" for us. M5Hosting is 
evaluating our own response to the event, and how our processes, 
systems and technologies can be improved to mitigate the impact of 
another service affecting event. AIS will certainly be doing the 
same... and we will follow their actions closely and remain engaged 
and involved with them.

   With this said, and with due respect that we are talking about an 
unplanned loss of power in an Internet Data Center, I am pleased with 
AIS's response to the incident once it happened. All of M5Hosting's 
technical staff and I were on site for up to 18hrs after the event and 
observed their response to it first hand. It is clear that AIS's 
response was well directed and planned.

   Please find their Post Incident Report attached below. A diagram 
of the AIS power infrastructure for the affected facility can be found 
at:
http://www.m5hosting.com/AIS_SDTC_Power_Diagram_l.jpg

   As always, I'd like to hear from you about this email, and the 
events and actions described in it... or anything at all.

Sincerely,

Michael J. McCafferty
Principal Engineer
M5 Hosting
mike[at]5hosting.com
877-344-4678 x501

[quote]
Dear Valued Customer:

As a follow up to the power event which occurred on the morning of  
March 18th, 2009 at the 9725 Scranton Data Center (SDTC), American  
Internet Services has compiled the following post incident report for  
our customer base.  As always, our Account Relations and Management  
team members are available to discuss specific customer issues or  
concerns, while this report is intended to provide comprehensive  
overview of the event itself.

At approximately 08:15AM PDT, March 18th, the SDTC datacenter suffered  
a complete power failure for approximately 30 seconds while conducting  
routine maintenance to the critical datacenter systems. The work that  
was being performed is part of AIS? Standard Operating Procedure. This  
procedure is in alignment with industry guidelines, and our commitment  
to provide customers with the highest availability in data center  
solutions. As we have informed our customers in the past, all critical  
systems are tested bi-monthly by our team of mechanical engineers in  
conjunction with our outside contractors under service agreements.  
Standard maintenance is performed during normal business hours and is  
carefully planned to incorporate the strictest test procedures to  
ensure the success of the work performed. Our SOP incorporates  
escalation processes and back out procedures in the unlikely event of  
an alert or anomaly during the standard maintenance.

Regretfully, during our maintenance yesterday, we encountered a  
mechanical failure. The Powerware 9515 UPS plant failed during the  
transition of building load from street power to generator power.  
Approximately 30 seconds upon failure of the UPS plant, our CTO,  
Richard Sears, who was present for the maintenance, restored power to  
the data center by manually moving the building to generator, quickly  
isolated the failure to the UPS plant, reset all four UPS modules, and  
brought all four UPS modules back online. Following, he moved the UPS  
plant from bypass mode to normal operational mode.

At that time, senior management called to initiate the Emergency  
Response Plan (ERP) and made a decision not to move the data center  
back to street power until our mechanical engineers and external  
contractors had an opportunity to perform diagnostics of all  
datacenter systems to determine what caused the failure to the UPS  
plant, as well as, test the general state of health of all critical  
systems.

Within approximately 15 minutes of initiating ERP, we had mobilized 18  
Customer Service Engineers, 5 Networking Engineers and Facilities and  
HVAC teams to the datacenter, in an effort to assists our customers  
with recovery. We also had UPS, battery and power experts from Eaton  
Powerware, CPD and Emerson there to assist in the investigation of the  
issue. As part of our emergency communication plan, all customers were  
proactively contacted and informed of the situation and were provided  
multiple progress updates throughout the day.

Upon reviewing of the findings, it was determined that one of our  
battery strings failed, resulting in their not being able to hold  
system load once the UPS plant went fully to battery. This caused a  
critically low battery voltage condition to the entire UPS plant and  
the plant protected itself by bypassing its system load to the main  
bus. This was during the time the building was being transferred from  
street power to generator power, so the main busses were both dead. In  
order to prevent a dead-head of the generator and utility systems, the  
SEL electrical system has a failsafe that prevents the main breakers  
from closing after the emergency breakers have been commanded to  
close, and there is power on the emergency bus. This condition  
prevented us from closing the main breakers, while we were still able  
to close the emergency breakers.

As with all of our critical datacenter systems, we have external  
contractors under maintenance agreements to provide system  
maintenance.  JT Packard is responsible for system maintenance on our  
entire UPS and battery plant at SDTC datacenter. We rely on our vendor  
to test each battery at specific intervals to determine if and when  
our batteries are approaching the threshold that requires replacement.  
JT Packard has been performing this system maintenance on a regular  
basis for several years now, of which most recently, reported 100%  
System Health.

The result of the investigation; in the opinion of both Eaton and CPD  
who conducted their investigation under separate check is that the  
battery string in question failed due to bad batteries that were not  
identified during the latest battery tests by JT Packard.

Upon validation of the findings, we mobilized 160 replacement  
batteries from Orange County to our datacenter and proceeded to  
schedule a three hour Emergency Maintenance window to start at 7:15PM  
PDT in order to replace the batteries and perform the load transfer  
back to street power. The evening's emergency maintenance window was  
completed successfully at approximately 10:00PM PDT and all critical  
systems where again, checked and diagnosed to be operating at 100%.

We sincerely apologize for the inconvenience yesterday?s event caused  
you. We want to assure you that we spare no expense when it comes to  
designing, deploying and maintaining our datacenter systems in order  
to meet the industry?s highest levels of reliability which our  
customers have come to expect. If you would like to receive any more  
detailed information regarding this matter, or would like a detailed  
layout of our power infrastructure, please let us know. We are here to  
be of assistance.

We want to thank all of our customers for their continued support  
while we worked together to mitigate this critical event.

Sincerely,

Alessandra M. Carrasco
Chief Executive Officer
American Internet Services

[/quote]

Emergency Maintenance and Initial Network/Power Event Overview

2009-03-18T21:03:00.000-07:00

Dear M5Hosting Customer,
As you may be aware, your server was rebooted today due to a momentary power loss at the data center facility (American Internet Services) from which M5Hosting provides your service. The root cause was a failed power system component during a scheduled generator test.

After the power was restored, M5Hosting also had some network issues due to some redundant network components behaving in an unexpected manner.

The failed power system components have been replaced and power will be taken off generator and back on the "street" power tonight during an emergency maintenance window declared by the data center facility. This maintenance window will begin at 8:30pm US/Pacific Time. While the window has been declared to be 2hrs, the actual cutover will be quite short. The window will allow for testing. This work is not expected to impact service at all.

I have pasted the most recent announcement from the data center facility below. We will also forward any subsequent announcements from them regarding this incident.

While the network and facility are fully functional and stable at this time, there is some work to do return all systems back to full redundancy and normal operating status. The maintenance discussed below will be the first step. This will return the power system. Our network work will take place during a late night maintenance window to be announced later and scheduled to mitigate any risks of an impact to service.

If you are still experiencing any issues with your server or other services which we provide for you at this time, please contact us at 877-344-4678 or submit a support ticket, so we take care of your needs immediately.

Sincerely,
Mike

The most recent Data Center facility announcement from AIS:

To Our Valued Customers,

Following this morning's facility issue at the 9725 Scranton Data Center, AIS Facilities and Engineering teams are announcing a 2 hour emergency maintenance window which will begin at 17:15 and which will conclude at or before 20:15.

During this window, main power for the facility will be returned from generator to normal operating mode. Throughout the day, engineering and mechanical teams have taken every precaution and performed considerable diagnostics to ensure that all systems are once again operating at 100% effectiveness. Even following these precautions, all maintenance windows involving critical power systems still carry the risk of causing of a power event during the maintenance operation, thus, we are bringing this important issue to your attention at this time. While maintaining a cautious posture, we at AIS do not anticipate any issues during this window.

During the maintenance window, the facility will remain fully staffed by 18 customer service engineers, 3 different mechanical engineering contractors, and AIS senior management personnel.

We are in the process of producing a detailed post incident report which will be made available during business hours tomorrow, Thursday March 19th.

A follow up communication will be sent upon completion of the maintenance window. We apologize for any inconvenience this issue may have caused.

AIS Support Services

2009-03-18T10:54:00.001-07:00

There was a short (less than a minute) power interruption at the data facility which houses the majority of M5Hosting customer servers. This caused all of the servers to reboot.
It is too early to know the root cause. The majority of customer servers came back up right away, as expected. However, some did not come back up as expected.
We have our entire technical staff at the data center working on the issues as quickly as possible.
There will be a complete post-mortem available as soon as possible.

Thank you for your patience and trust in us.

Sincerely,
Mike

UPDATE 13:45 PDT: There has some packet loss, which may be related to the power event this morning.

UPDATE 15:30 PDT: All network issues have been stable for a few hours. Only a few servers which have hardware failures (ie: failed power supplies or hard disks) remain down. Tech staff are addressing them as quickly as possible.

UPDATE 17:34 PDT: The failed power system components have been replaced. The data center will be transitioning off backup power and back to SDG&E power at about 18:00 PDT. Tech staff remains on site to address any issues that may arise.

Packet Loss Event

2009-03-11T12:28:00.000-07:00

This morning at 7:35am PDT, some customers reported up to 33% packet loss. The issue was resolved within 18 minutes. The root cause was a Denial of Service attack against a customer. The offending system was removed from the network and the DoS stopped.

Our Network Engineers have identified the resources which were DoS'd and the technical issues which occurred. We will develop a plan to mitigate the risk of the same events happening again.

Urgent Network Maintenance Tonight

2008-11-10T16:39:00.000-08:00

One of our main network providers (AmericanIS) will be performing maintenance on the hardware and software of the equipment to which we are directly attached. This work is to resolve an issue that was discovered after a very brief event last Sunday night.

The event last Sunday night occurred when another customer of AmericanIS produced an extreme increase in load, possibly due to a Denial of Service attack or another unusual event). This negatively affected the SUP720 Route Engine performance in the Cisco 6509's which we are connect to AmericanIS through. This ultimately caused a brief period of packet loss and high latency to our customer's whose traffic flowed through AmericanIS at the time.

The solution is to upgrade the SUP720 Engines to most powerful Route Engines available for the Cisco 6509, the SUP720-3BXL. These engines were already on hand, and this upgrade has already been performed on other routers on the AmericanIS network that your traffic may pass through to different destinations on the Internet. However, we are directly connected to the last two routers to be upgraded.

This upgrade will be performed between 22:00 and 23:59 US/Pacific Time, tonight. (06:00 to 08:00 GMT Nov 11th, 2008)

During this time, the two 6509s will be upgrade one at a time. We are connected to these two routers and using HSRP fail-over protocol, so that there will be only a momentary pause in network traffic through AmericanIS as each router is taken down for the upgrade. Except for the brief pauses, the network is expected to remain up.

This maintenance is being performed by one of our upstream providers. This work is to upgrade their network devices to which we are onnected. The time and nature of the upgrade was chosen by them and not by M5Hosting. AmericanIS has historically provided M5Hosting with very highly reliable and high performance Internet transit service. This upgrade is expected to keep that level of service.

As always your feedback and comments relating to this announcement or the work is describes is encouraged.

UPDATE @ 01:30:
The upgrade being performed by AIS on the AIS network has exceeded their announced time window. There has also been a greater impact on the service than expected starting near the end of their announced window. Their very capable Network Engineers are working diligently to complete the upgrade.
While this is work being performed on a network not in our control, I have been on-site at the data center myself for the duration of the maintenance, and will continue to be for any contingency that may require our action and to get first hand updates and to render updates to you if needed.
The original plan was for very brief interruptions to cut over from one router to another. However, there have been a few periods of packet loss and latency on their network during this maintenance. This has affected our network and service to our customers, some routes are affected less than others.

You are welcome to contact us directly if you have specific questions not otherwise addressed.

UPDATE @ 04:30:
The network upgrade is complete and has been stable for a couple of hours now.

Urgent "Failover" to redundant core switch @ 2am

2008-08-21T03:13:00.000-07:00

A probable bug in the Cisco IOS running on the current active layer 3 core switch is preventing us from remotely managing the device. We have exhausted the options for restoring the functionality without interrupting the traffic flow through it.
Tonight, after 02:00 PDT we will force the redundant layer 3 core switch to take over while we address the issue with the current primary switch. The cut-over to the redundant switch should only take a few seconds. If we fail-back to the current primary within a short time, it too should only take a few seconds.

Date/Time: Thursday 8/21/08 @ 02:00 US/Pacific (GMT -7)
Duration: 5 to 30 seconds (1 minute "worst case")

As always, I invite your comments on this message, or the events it describes. Your feedback helps us deliver better services to you.

Thanks,
Mike

Hardware failure in shared hosting server "Witt"

2008-08-14T01:11:00.000-07:00

The shared hosting server called witt.m5hosting.com suffered a disk failure. To replace the disk required a few minutes of downtime. This was performed shortly after midnight tonight. The server will be a little slow at disk intensive tasks for a few hours while the array is rebuilt and the disks are synchronized.

Update 8/14/08 17:00 PDT: Disk rebuild is complete. System IO tasks running at normal speed.

Intermittent Issues with Level 3 Communications Link

2008-08-08T14:26:00.000-07:00

We have had a few reports, confirmed by the data center facility and our own monitoring systems, that routes through Level3 Communications were intermittently "flapping" from 13:31 to 13:58 PDT.

We have asked the data center to keep us updated with anything they may find, as they are the direct Level 3 customer and it is though the data center facility's network that we are connected to Level 3 Communications.

If you have a traceroute or MTR information which shows the issue, please send them to support at m5hosting.com

Thanks !

Network Maintenance - 02:00 to 06:00 Aug 10th, 2008

2008-08-08T14:20:00.001-07:00

We will be making some nifty upgrades to our network this weekend. The maintenance was previously announced here.

We will be replacing our border routers, and completing some VLAN segmentation work that remained uncompleted from a previous maintenance window.

Network Maintenance - July 6th, 2008

2008-07-05T11:21:00.000-07:00

We will be making some upgrades to the network tonight. This work will cause some brief interruptions between 12:01am PDT and 5:00am PDT on Sunday July 6th, 2008.

This maintenance was announced via broadcast email announcement. An archive of broadcast emails is maintained on the web.

If you did not receive this message, please check if it may have ended up in your spam folder, or subscribe yourself here

What goes here

2008-06-29T22:46:00.000-07:00

This blog is specifically to communicate timely system status information to customers of M5hosting.com.