August 25, 2003
Dear ISPrime Customer,
On August 24th 2003 ISPrime network came under DDOS attack which caused
our core Juniper routers to overload and fail.
Cause: Juniper's JUNOS 5.x PR/31166 explains a memory leak in its Arp
resolver, where every time a failed ARP query is sent, 16 bytes of ram
is leaked. (The router did not free memory consumed in certain cases
when it used Address Resolution Protocol (ARP) or multicast protocol to
resolve IP addresses for transit traffic. [PR/31166]) Under normal
levels of ARP traffic, the amount of time for the memory leak to reach a
critical point is measured in years.
The MSBlaster worm had a 2nd iteration which began spreading late last
week, and had a different goal than the first. Instead of just
attacking Microsoft, it tried to exploit a memory leak in the ARP
resolvers on some platforms, or route caches on others, by sending 92
byte ICMP Echo (ping) packets to random hosts, it increased the amount
of arp queries for hosts that do not answer arp significantly, and
lowered the time to go to a critical amount of memory leaked from years,
to hours.
There is redundancy built in to our network, but as both of our Juniper
Core Routers were affected by the same bug in its operating systems,
both failed.
This problem is not unique to our setup, nor is it unique to Juniper
routers. There have been reports of other networks running into similar
problems with Cisco, and Lucent networking equipment. (
http://www.merit.edu/mail.archives/nanog/msg12693.html )
Resolution process: Our network engineers preformed an emergency
maintenance to our core routers during which an upgrade to router
operating system was made from JunOS 5.4 to JunOS 6.0R1 platform which
fixes that bug, and adds graceful failover features which would allow
packets to be forwarded in the event of a Routing Engine failure, as the
backup Routing Engine reconverges.
Long term solution: Utilizing Junos 6's new features to allow for
immediate failover in the event of software failure in the future.
Our Network Operation Center was handling this procedure quickly and
efficiently, therefore we were able to minimize downtime to our clients
to absolute minimum. We would like to thank you for your understanding
and cooperation. We would also like to reassure you in our network's
stability and reliability. If you have any questions or comments
regarding this or any other issue, please feel free to contact us at
[email protected]
Best Regards,
ISPrime Team
http://www.isprime.com
The Ultimate Hosting Company
Dear ISPrime Customer,
On August 24th 2003 ISPrime network came under DDOS attack which caused
our core Juniper routers to overload and fail.
Cause: Juniper's JUNOS 5.x PR/31166 explains a memory leak in its Arp
resolver, where every time a failed ARP query is sent, 16 bytes of ram
is leaked. (The router did not free memory consumed in certain cases
when it used Address Resolution Protocol (ARP) or multicast protocol to
resolve IP addresses for transit traffic. [PR/31166]) Under normal
levels of ARP traffic, the amount of time for the memory leak to reach a
critical point is measured in years.
The MSBlaster worm had a 2nd iteration which began spreading late last
week, and had a different goal than the first. Instead of just
attacking Microsoft, it tried to exploit a memory leak in the ARP
resolvers on some platforms, or route caches on others, by sending 92
byte ICMP Echo (ping) packets to random hosts, it increased the amount
of arp queries for hosts that do not answer arp significantly, and
lowered the time to go to a critical amount of memory leaked from years,
to hours.
There is redundancy built in to our network, but as both of our Juniper
Core Routers were affected by the same bug in its operating systems,
both failed.
This problem is not unique to our setup, nor is it unique to Juniper
routers. There have been reports of other networks running into similar
problems with Cisco, and Lucent networking equipment. (
http://www.merit.edu/mail.archives/nanog/msg12693.html )
Resolution process: Our network engineers preformed an emergency
maintenance to our core routers during which an upgrade to router
operating system was made from JunOS 5.4 to JunOS 6.0R1 platform which
fixes that bug, and adds graceful failover features which would allow
packets to be forwarded in the event of a Routing Engine failure, as the
backup Routing Engine reconverges.
Long term solution: Utilizing Junos 6's new features to allow for
immediate failover in the event of software failure in the future.
Our Network Operation Center was handling this procedure quickly and
efficiently, therefore we were able to minimize downtime to our clients
to absolute minimum. We would like to thank you for your understanding
and cooperation. We would also like to reassure you in our network's
stability and reliability. If you have any questions or comments
regarding this or any other issue, please feel free to contact us at
[email protected]
Best Regards,
ISPrime Team
http://www.isprime.com
The Ultimate Hosting Company