|
And the point isn't so much what they're doing specifically to my site, more that they're arrogant enough to ignore (defacto?) robots.txt settings that every other major search engine bot respects. The webmaster dashboard robots.txt checker even helpfully points out that each of the crawl-delay lines in my robots.txt are ignored!
|