Googlebot jerks me 100,000 times a day but Google won't show me more love

PS: This phenomenon is also causing overloading issues because it's a database driven profile site... Googlebot is hitting me pretty much as fast as it can (around 85% of all fetches are from Googlebot), yet there seems to be no additional long term benefit from this. If the referral traffic scaled with the number of profile pages indexed then I'd just add more capacity, but it's not worth it.

I've noticed it too, there is definitely a cap system in place


off your topic a bit, but I've got a site that is pr5, up from pr4 last update, that googlebot visits all the time and lately google is sending the site zero traffic, very strange

google is a strange Beast

yeah i can see that and i hate it...its almost constant

been there done that

Yesterday's referral counts

google - 33500
bing - 209
yahoo - 23

I'm going nowhere

From what I have read on an interesting black hat site, there is a sandbox on adult oriented content. My guess is that goog knows you have adult content and has sandboxed you into adult, given goog also can find networks of sites -- they've capped you based on your footprint given the adult nature of your network.

I'm an seo noob though.

google is god!

no adult cap.

Referrals sampled on the first day of each month...

4622 01/Oct/2008
6343 01/Nov/2008
9534 01/Dec/2008
15682 01/Jan/2009
27923 01/Feb/2009
31315 01/Mar/2009
31235 01/Apr/2009
30495 01/May/2009
31308 01/Jun/2009
31259 01/Jul/2009
32537 01/Aug/2009

You can see it clearly tops out in Feb and is capped from there on.

that;s a lot of google traffic, you should not complain

You probably know this, but you can set a crawl delay in your robots.txt file to slow down googlebot.

From what I can tell Google now ignores crawl-delay, and requires webmasters to submit/verify/configure their site via webmaster tools.

You have tons more traffic than I do. About the only thing we have in common is that most traffic by far comes from Google. I've always relied heavily on using Blogger (Google's own software) for SEO. I have 15+ blogs and a few of them are 3+ years old. They've consistently delivered traffic based on page-1 results for several good keywords. About 2 weeks ago, my traffic began slowing, so I started digging to find out what was happening.

I use statcounter for tracking and I noticed that my traffic from Google was dropping based on keywords. One by one, this started happening with all my blogs. Within one day, 4 of my blogs lost 60% of their traffic and they haven't recovered since. So, I went to Google's webmaster tools and started tweaking and managing my sites and sitemaps.
I've been trying to conform as much as possible to what Google's reporting there with crawls, etc. It's not working. On Monday, I added 5 pages to one of my better blogs in hopes of getting back on track. Today, my submitted URLs via my sitemap reflect the addition of the 5 pages, but I've actually dropped by 3 the number of pages Google is now indexing. I don't know what's going on... but all of a sudden what I've been doing successfully for the past 3+ years doesn't work at all. A lot of hard and diligent work down the tubes.

In fact, on a couple of my blogs the only traffic I'm getting through Google is "image" traffic. I've ceased getting "keyword" traffic because the pages I had on Google's first results page have slipped back to the 5th, 6th or 7th and beyond results pages. My sites are slowly disappearing. And, at this point, I'm not sure what to do. I'm not doing any black hat stuff, or anything that Google's had a known problem with (webmaster guidelines). My conclusion is that Google newest algorythms no longer like its own blogging platform.

I just thought of a sticky tape and bandaids solution to the crawl-delay issue... since Googlebot appears to fetch sequentially, why not just add a delay yourself?

if (strpos($_SERVER["HTTP_USER_AGENT"], " Googlebot/") > 0) sleep(2);

Why not setup it in the webmasters area from Google directly...?? It's free and a valid solid solution...

I don't like Google knowing too much about my sites.

By removing support for robots.txt crawl-delay (they used to support it) they're forcing people to use their proprietary webmaster tools system.

5,8 million pages indexed and 33k hit's a day, can only be black hat lol

How can you be generating 100k profile pages per day? What kind of site are you running?

I believe it. Same with casino sites.

Nothing blackhat about it, the profile pages are related to domains, which is why there's hundreds of millions of them...

why do you think that just because you add a page... that page is important and should be showing up in the SERP's and your traffic should be growing?

something like aboutus.org?

Let google do what it wants to do - fucking around with it could invoke the Law of Unintended Consqeuences.

Google is like the micro manager... always checking up on you and giving you no love....

Google's own blogging platform IS a cesspool of spam and linkschemes and piracy and other assorted garbage, generally speaking

there is a simple reason

cant post it on a board tho