|
i just got this...
The upgrade this morning was a mass-software upgrade in preparation for some large additions to our network and to maintain some general housekeeping.
The upgrade path was pretty clear cut and, as usual, verified by a third party. The initial upgrade was successful, but around 3am EST certain routing loops causes unusually high processor utilization. This issue result in the datacenter's decision to back out of the upgrade. The process of downgrading the software on the core switches was impeded by a version mismatch between previously existing firware on external routing cards and the management software on the switch itself. This left the datacenter in a limbo between two versions and an inability to reimplement the original code. They are currently waiting on a software patch to be written by the manufacturer to enable to either go full forward or fall back on the old version.
Right now, the most obvious symptom of this condition is packet loss. The datacenter are working with the manufacturer, who is onsite, to resolve this.
We apologize for the on-going issues and will update you as soon as we have more information.
|