Quote:
|
Originally Posted by duckduckgoose
Their spiders hammer sites to the point of abuse sometimes. Even Googlebot isn't as rampant and intensive as the MSN spiders. I ended up blocking them from some sites because of the incessant and abusive pounding they were doing.
MSN says they observe robots.txt directions, so if you find them being a problem, you could block them there. In my experience, I found out it was them only after it was thought to be abuse.
|
ah... I see. Thanks a lot!