Quote:
Originally Posted by ColetteX
so google bots are ignoring crawl delay? i have delays as i am updating my sites, i dont want to see google every day on site that i am updating weekly. if crawl delay do not work, it is really sick cause 6 days from week google see my site as static not updated. oh snap
|
Crawl-delay is related to how fast a bot hits your site if it has multiple pages, not how long it will wait between refetching the same page.
http://www.google.com/support/webmas...n&answer=48620
My site has 200 million pages so technically googlebot isn't fetching fast enough... at the rate of 120k fetches per day it would take 4 1/2 years to index everything. At this point the benefit of indexing 100% of the site (or at least as much as it's trying to) isn't worth the load it's placing on the server.