top in ssh would give an idea of the problem:
CPU , memory or the most likely culprit .
Config of apache ( how many allowed connections ) could also be the problem.
Personnally, I have quite a few clients running some WP sites, with substantial traffic... This requires at least a Core2 Duo CPU and lots of memory...
I know you don't want people to say " it is the server ", but giving the specs of the box would help.
