DotBot from moz.com is not obeying robots.txt directives
It was pounding like crazy 1000s of my pages daily, I had not choice but to ban on webserver level.
Besides what good it does? It's just spying on your keywords and links.
Well, even after ban it is still trying to access my sites. (I allowed only robots.txt for it. It reads it and does not give a fuck.) So their shit is obviously broken.
What I’m trying to say is that, perhaps you should review your access logs from time to time and see what is wasting your bandwidth and server resources.
For example, my other observation is that within last 3 years I have received from 500 000 to 1000 000 hacking and exploiting attempts per each live website. While all of them were not successful, it must have some impact on server performance.
Well configured webserver can reduce all this bad stuff by 90-99%.
|