Urgent Network Maintenance Tonight

One of our main network providers (AmericanIS) will be performing maintenance on the hardware and software of the equipment to which we are directly attached. This work is to resolve an issue that was discovered after a very brief event last Sunday night.

The event last Sunday night occurred when another customer of AmericanIS produced an extreme increase in load, possibly due to a Denial of Service attack or another unusual event). This negatively affected the SUP720 Route Engine performance in the Cisco 6509's which we are connect to AmericanIS through. This ultimately caused a brief period of packet loss and high latency to our customer's whose traffic flowed through AmericanIS at the time.

The solution is to upgrade the SUP720 Engines to most powerful Route Engines available for the Cisco 6509, the SUP720-3BXL. These engines were already on hand, and this upgrade has already been performed on other routers on the AmericanIS network that your traffic may pass through to different destinations on the Internet. However, we are directly connected to the last two routers to be upgraded.

This upgrade will be performed between 22:00 and 23:59 US/Pacific Time, tonight. (06:00 to 08:00 GMT Nov 11th, 2008)

During this time, the two 6509s will be upgrade one at a time. We are connected to these two routers and using HSRP fail-over protocol, so that there will be only a momentary pause in network traffic through AmericanIS as each router is taken down for the upgrade. Except for the brief pauses, the network is expected to remain up.

This maintenance is being performed by one of our upstream providers. This work is to upgrade their network devices to which we are onnected. The time and nature of the upgrade was chosen by them and not by M5Hosting. AmericanIS has historically provided M5Hosting with very highly reliable and high performance Internet transit service. This upgrade is expected to keep that level of service.

As always your feedback and comments relating to this announcement or the work is describes is encouraged.

UPDATE @ 01:30:
The upgrade being performed by AIS on the AIS network has exceeded their announced time window. There has also been a greater impact on the service than expected starting near the end of their announced window. Their very capable Network Engineers are working diligently to complete the upgrade.
While this is work being performed on a network not in our control, I have been on-site at the data center myself for the duration of the maintenance, and will continue to be for any contingency that may require our action and to get first hand updates and to render updates to you if needed.
The original plan was for very brief interruptions to cut over from one router to another. However, there have been a few periods of packet loss and latency on their network during this maintenance. This has affected our network and service to our customers, some routes are affected less than others.

You are welcome to contact us directly if you have specific questions not otherwise addressed.

UPDATE @ 04:30:
The network upgrade is complete and has been stable for a couple of hours now.