|
To All Customers: 09:28 AM PST 3/1/2004
This morning we had a system wide outage that effected all customers. The outage began at 08:11:21 PST. Recovery concluded at 09:08:32.
We have been working on this for just about an hour, so information and data in this e-mail is subject to change as we continue with our investigation and analyze the information at hand.
** Preliminary ** fault isolation shows us receiving a flood of packets directly at our both core routers, not targeting any particular customer.
We have made some changes here, and have made some changes with our upstream providers that are helping.
We will continue to monitor, collect data, and make changes.
We need to investigate further, so I believe it would be premature to draw any type of definitive conclusions at this point.
We are going to reserve the comment "it's resolved" until we have 1) had more time to review data 2) definitive evidence showing the root cause 3) a viable and long term fix in place.
~Ray
|