Originally Posted by Judas_Hansen

have you hear about robot.txt? just tell the spiders what to search, and what not to search.. The perfect solution would be to build a "SE archive" from the daily backup located on an other server, and then let the Spiders look at that one, and not the live site