Quote:
Originally Posted by Mr Pheer
Googlebot has taken down my servers before when I was running stuff I don't talk about. Pretty much same scenario as yours, a few dozen domains and it was crawling pages like 40 - 50 per second for 3 days straight.
Running tail -f on the logs and shit was just flying off the screen.
|
Yandex has done it to me too but for that shit I can 404 'em b4 they hit the database bc they send no traffic anyway...with google I gotta seriously think about how to balance letting them molest my servers and how much traffic I think they'll send if I let them