|
Sorry folks,
We did some routine maintenance today (due to sundays being off-peak time for us traffic-wise), which involved moving multiple fiber connections to our new demarc and router core.
This should not have had any impact, but unfortunately due to a tech error we lost both gige connections to one of our providers, which happened to be the preferred inbound route for approximately 70-80% of the internet. The connection was lost for no more than 60 seconds, but due to the route tables needing to reconverge for a substantial amount of the internet we were "down" to most locations for a substantial amount of time, approximately 4-10 minutes based on our internel traffic monitoring.
If you would like a more technical explanation, I'll get one of the techs to post as well.
We are almost complete with the maintenance, with just 1 more fiber to move which should not effect anything visibly to our customers. The moves were neccessary, as as of today we'll be 100% on our new routing core which is failover. After today, every one of our customers will be connected to multiple core routers, and the customer aggregation switches are connected via diverse fiber paths to our central switching core.
Customers will be contacted invidivually this coming week to move over to the new failover gateways, and also to move to IP space in our own assigned netblock. For virtual customers this will be transparent.
Again, we apologize for the problems. We will modify our maintenance policy to update customers beforehand on things such as these that MAY impact performance/availability but are not expected to. This was our mistake.
Thanks much,
-Phil
phil at reflected.net
|