Googlebot... how arrogant can they be? - GoFuckYourself.com - Adult Webmaster Forum

X

rowan

Too lazy to set a custom title

Join Date: Mar 2002

Posts: 17393
Share

Post
#1

Googlebot... how arrogant can they be?

02-21-2010, 10:07 AM

Google ignores crawl-delay in robots.txt. Instead, they force you to register your site in webmaster tools, so you can set a custom crawl rate. This is then RESET to the default after 90 days, so at that time you have to login again and change it again! How fucking arrogant is that?

Other search engines respect crawl-delay, imagine if they all wanted us to create accounts and login every 90 days to stop their bots hammering our servers?
Tags: None
Agent 488

Registered User

Join Date: Feb 2006

Posts: 22511
Share

Post
#2

02-21-2010, 10:09 AM

yeah that damn google bot racking up bandwidth.
Comment
2012

So Fucking What

Join Date: Jul 2006

Posts: 17189
Share

Post
#3

02-21-2010, 10:15 AM

best host: Webair | best sponsor: Kink | best coder: 688218966 | Go Fuck Yourself
Comment
rowan

Too lazy to set a custom title

Join Date: Mar 2002

Posts: 17393
Share

Post
#4

02-21-2010, 10:16 AM

Originally posted by Agent 488

yeah that damn google bot racking up bandwidth.

Server load, actually. They're hitting me 120k times a day.
Comment
rowan

Too lazy to set a custom title

Join Date: Mar 2002

Posts: 17393
Share

Post
#5

02-21-2010, 10:20 AM

And the point isn't so much what they're doing specifically to my site, more that they're arrogant enough to ignore (defacto?) robots.txt settings that every other major search engine bot respects. The webmaster dashboard robots.txt checker even helpfully points out that each of the crawl-delay lines in my robots.txt are ignored!
Comment
ColetteX

So Fucking Banned

Join Date: Jun 2004

Posts: 539
Share

Post
#6

02-21-2010, 10:23 AM

so google bots are ignoring crawl delay? i have delays as i am updating my sites, i dont want to see google every day on site that i am updating weekly. if crawl delay do not work, it is really sick cause 6 days from week google see my site as static not updated. oh snap
Comment
rowan

Too lazy to set a custom title

Join Date: Mar 2002

Posts: 17393
Share

Post
#7

02-21-2010, 10:28 AM

Originally posted by ColetteX

so google bots are ignoring crawl delay? i have delays as i am updating my sites, i dont want to see google every day on site that i am updating weekly. if crawl delay do not work, it is really sick cause 6 days from week google see my site as static not updated. oh snap

Crawl-delay is related to how fast a bot hits your site if it has multiple pages, not how long it will wait between refetching the same page.

http://www.google.com/support/webmas...n&answer=48620

My site has 200 million pages so technically googlebot isn't fetching fast enough... at the rate of 120k fetches per day it would take 4 1/2 years to index everything. At this point the benefit of indexing 100% of the site (or at least as much as it's trying to) isn't worth the load it's placing on the server.
Comment
ColetteX

So Fucking Banned

Join Date: Jun 2004

Posts: 539
Share

Post
#8

02-21-2010, 10:31 AM

Originally posted by rowan

Crawl-delay is related to how fast a bot hits your site if it has multiple pages, not how long it will wait between refetching the same page.

http://www.google.com/support/webmas...n&answer=48620

My site has 200 million pages so technically googlebot isn't fetching fast enough... at the rate of 120k fetches per day it would take 4 1/2 years to index everything. At this point the benefit of indexing 100% of the site (or at least as much as it's trying to) isn't worth the load it's placing on the server.

thank you man, there is still much to learn. bump for your answer
Comment
woj

<&(©¿©)&>

Join Date: Jul 2002

Posts: 47880
Share

Post
#9

02-21-2010, 10:36 AM

User-agent: Googlebot
Disallow: /

problem solved

Custom Software Development, email: woj#at#wojfun#.#com to discuss details or skype: wojl2000 or gchat: wojfun or telegram: wojl2000
Affiliate program tools: Hosted Galleries Manager Banner Manager Video Manager
Wordpress Affiliate Plugin Pic/Movie of the Day Fansign Generator Zip Manager
Comment
rowan

Too lazy to set a custom title

Join Date: Mar 2002

Posts: 17393
Share

Post
#10

02-21-2010, 10:38 AM

Originally posted by woj

User-agent: Googlebot
Disallow: /

problem solved

I was waiting for it. I'm surprised this "solution" wasn't posted sooner.
Comment
CunningStunt

Confirmed User

Join Date: Aug 2006

Posts: 5594
Share

Post
#11

02-21-2010, 12:02 PM

Originally posted by woj

User-agent: Googlebot
Disallow: /

problem solved

To the OP - agreed. The trouble is, they can do what the hell they want with over 70% of the search market.
Comment
MrMaxwell

Too lazy to set a custom title

Join Date: Jul 2005

Posts: 10057
Share

Post
#12

02-21-2010, 02:32 PM

What kind of site has 200,000,000 pages?
That's alot of pages
Comment
Adam_M

Confirmed User

Join Date: Mar 2006

Posts: 3800
Share

Post
#13

02-21-2010, 02:34 PM

Originally posted by woj

User-agent: Googlebot
Disallow: /

problem solved

I wish everyone would just do this!

DiscountedPorn.Com
ReviewedPorn.com
Comment
Cyber Fucker

Hmm

Join Date: Sep 2005

Posts: 12642
Share

Post
#14

02-21-2010, 02:55 PM

Originally posted by Adam_WildCash

I wish everyone would just do this!

Lol
Comment
fatfoo

ICQ:649699063

Join Date: Mar 2003

Posts: 27763
Share

Post
#15

02-21-2010, 02:57 PM

They can be very arrogant.

Send me an email: [email protected]
Comment
Waddymelon

So Fucking Banned

Join Date: Oct 2009

Posts: 17349
Share

Post
#16

02-21-2010, 04:19 PM

Annoying isnt it? I've had googlebot hit my servers up to 15 times per second, for hours at a time. Dynamic pages really make googlebot go nuts.
Comment
u-Bob

there's no $$$ in porn

Join Date: Jul 2005

Posts: 33063
Share

Post
#17

02-21-2010, 04:30 PM

They're evil, plain and simple....
Comment
StrokeKing

Confirmed User

Join Date: May 2008

Posts: 232
Share

Post
#18

02-21-2010, 05:30 PM

Originally posted by Waddymelon

Annoying isnt it? I've had googlebot hit my servers up to 15 times per second, for hours at a time. Dynamic pages really make googlebot go nuts.

15times in a second?! really annoying! well its a smart idea to disallow the googlebot.. thanks for the advice guys.

Adult Reviews by the Stroke King.
Comment
czarina

Webmaster Extraordinaire

Join Date: Jul 2002

Posts: 10751
Share

Post
#19

02-21-2010, 05:38 PM

woj's solution is good if you want to keep googlebot off your site completely. But are you sure you want to do that?
Comment
epitome

So Fucking Lame

Join Date: Jun 2009

Posts: 12156
Share

Post
#20

02-21-2010, 05:48 PM

Originally posted by MrMaxwell

What kind of site has 200,000,000 pages?
That's alot of pages

Google.

Rowan actually founded Google and he's frustrated because his baby is stuck in a loop.

Even as the founder, he cannot get support at Google and has to do the same thing as the rest of us.
Comment
Dirty Dane

Sick Fuck

Join Date: Feb 2004

Posts: 9491
Share

Post
#21

02-21-2010, 06:33 PM

Sue them.
Comment
rowan

Too lazy to set a custom title

Join Date: Mar 2002

Posts: 17393
Share

Post
#22

02-21-2010, 07:44 PM

Originally posted by epitome

Google.

Rowan actually founded Google and he's frustrated because his baby is stuck in a loop.

Even as the founder, he cannot get support at Google and has to do the same thing as the rest of us.

Yeah, I got pushed out by The Man! Fuck them!
Comment
papill0n

Unregistered Abuser

Join Date: Oct 2007

Posts: 15547
Share

Post
#23

02-21-2010, 09:40 PM

Originally posted by StrokeKing

15times in a second?! really annoying! well its a smart idea to disallow the googlebot.. thanks for the advice guys.
Comment
MrMaxwell

Too lazy to set a custom title

Join Date: Jul 2005

Posts: 10057
Share

Post
#24

02-22-2010, 03:58 AM

What kind of site has 200,000,000 pages?
Comment
rowan

Too lazy to set a custom title

Join Date: Mar 2002

Posts: 17393
Share

Post
#25

02-22-2010, 04:35 AM

Originally posted by MrMaxwell

What kind of site has 200,000,000 pages?

Think of something on the net that there might be 200,000,000 of to profile.
Comment
raymor

Confirmed User

Join Date: Oct 2002

Posts: 3745
Share

Post
#26

02-22-2010, 09:40 AM

Originally posted by rowan

My site has 200 million pages so technically googlebot isn't fetching fast enough... at the rate of 120k fetches per day it would take 4 1/2 years to index everything. At this point the benefit of indexing 100% of the site (or at least as much as it's trying to) isn't worth the load it's placing on the server.

200 MILLION pages? I'm curious, is that 200 million legitimate pages, or 200 million
pieces of fake SE spam crap? If you tried to spam the crap out of Google by creating
200 million bogus pages, I'd say you got what you deserved, and really what you
asked for. If you pretended to have 200 million pages so that Google would spider
you 200 million times, that was your decision. You can't blame Google if you chose to
create fake stuff for them to spider.

Note the repeated use of "IF" - I'm asking IF that's what you did.

For historical display only. This information is not current:
support@bettercgi.com ICQ 7208627
Strongbox - The next generation in site security
Throttlebox - The next generation in bandwidth control
Clonebox - Backup and disaster recovery on steroids
Comment
tiger

Confirmed User

Join Date: Apr 2002

Posts: 6986
Share

Post
#27

02-22-2010, 10:26 AM

Originally posted by woj

User-agent: Googlebot
Disallow: /

problem solved

Yeah I was wondering when someone would post this.

Bottom line is, if its causing you more problems then its worth just block it. If they are hitting you that hard you should be getting some good traffic because of it, more traffic = more money, Just upgrade the servers.
Comment
DamnGoodRatio

Confirmed User

Join Date: Feb 2003

Posts: 855
Share

Post
#28

02-22-2010, 10:38 AM

I return a 503 page and they seem to respect that. Then I set it so they can crawl during my off peak loads which they seem to do.
Should work and I know this is documented somewhere on Google's FAQ just can't seem to find the link right now.

Obama Said: "We can absorb a terrorist attack."
Comment
rowan

Too lazy to set a custom title

Join Date: Mar 2002

Posts: 17393
Share

Post
#29

02-22-2010, 01:07 PM

Originally posted by DamnGoodRatio

I return a 503 page and they seem to respect that. Then I set it so they can crawl during my off peak loads which they seem to do.
Should work and I know this is documented somewhere on Google's FAQ just can't seem to find the link right now.

I've noticed that a connection or server error will make G-bot back right off, but I'm concerned that doing this routinely might affect my rank. I've seen recent articles saying that site response time may become a factor in the future.

raymor: not useless spam, it's all genuine profiling of... domains.
Comment
rowan

Too lazy to set a custom title

Join Date: Mar 2002

Posts: 17393
Share

Post
#30

03-03-2010, 01:59 AM

February 22, 2010

New crawl rate: Custom rate

1.000 requests per second

1.000 seconds per request

This new crawl rate will stay in effect for 90 days.

Funny, 2 weeks later googlebot is still requesting 120k+ pages per day, which is about 150% the rate of the above setting.

Their webmaster tools system also sent me a notification encouraging me to increase the rate so they can fetch more pages. Looks like their bot is doing it anyway.
Comment
seeandsee

Check SIG!

Join Date: Mar 2006

Posts: 50945
Share

Post
#31

03-03-2010, 02:36 AM

bad bad google

BUY MY SIG - 50$/Year

Contact here
Comment

Previous template Next

Working...