February 22, 2010
New crawl rate: Custom rate
1.000 requests per second
1.000 seconds per request
This new crawl rate will stay in effect for 90 days.
Funny, 2 weeks later googlebot is still requesting 120k+ pages per day, which is about 150% the rate of the above setting.
Their webmaster tools system also sent me a notification encouraging me to increase the rate so they can fetch more pages. Looks like their bot is doing it anyway.
