One other thing to bear in mind, once a server starts overloading it's usually the start of an exponential spiral. Doubling the load might triple or quadruple the delay that the surfers experience. So making even minor changes to ease the load can produce a much more apparent benefit.
The site I mentioned above was once reviewed as "the slowest site on the net" before I did those tweaks.
