Quote:
Originally Posted by wankawonk
But shouldn't I just be able to easily tell them, "don't hit my site more than 100k times per day" and they should respect that?
|
Absolutely agree. Even if Crawl-Rate is non standard, if that directive appears in robots.txt, it's a clear instruction that should be respected.
Google seems to be following the letter of the law, rather than the spirit, which doesn't always work so well on the internet.