google spidering takes more bandwidth now
 

Anyone noticed the google spiders are gobbling a lot more bandwidth now? Daretwo got hold of me and informed me that the CPU usage my boxes was getting pegged out, but we couldn?t figure out from where. Checked the server log to find out it was those ?damn? google spiders doing it.

I just hoped the grabbed and ranked a ton of my pages well. Anyone else noticed this?

yes
but quite some time now if im not mistaken

Ive yet to see any but I dont think google likes me. If they are taking that much I hope you get some nice SE traffic and a PR from it.

That does depend on how many pages google is spidering and you must have a lot. I know for a webhost defiantweb.com I see when the spiders check all the pages its about 3 times a week sometimes 5.

They're storing every bit of information on the web for the Information Awareness Office.

Google is suppose to be server friendly. Are you sure they're google? Did you check the ips against the known google ips?

is this true?

I believe google hates me personally.

I haven't noticed, anyway I think google made some changes recently about..

Samuel

There was a huge increase in google activity lately, could be related to the going public. I don't know what's going on, but it grabs pages like there is no tomorrow. I recognized that suddenly links google never liked before are now crawled, forums and shops without mod rewrite, links with affiliate codes and stuff like this.

Wow. :1orglaugh

hahaha :1orglaugh :moon

You can block Google with robots.txt if you think they are taking too much of your bandwidth:winkwink: