Huh? Google is indexing GFY?

**rowan** · 08-23-2004, 07:41 AM

http://www.google.com/search?q=%22Ne...e=off&filter=0

**chupachups** · 08-23-2004, 07:44 AM

Google doesnt give a shit about robot rules...

**julioxp** · 08-23-2004, 08:03 AM

Yeap, I have found some pages of known posts at GFY when do a search at google..

julio

**beergood** · 08-23-2004, 08:40 AM

I think that the robots.txt file for gfy only causes the robot not to index the index page.

http://www.google.com/search?hl=en&l...ckyourself.com

User-agent: *
Disallow: *

for the whole site

**rowan** · 08-23-2004, 08:42 AM

Originally posted by beergood
I think that the robots.txt file for gfy only causes the robot not to index the index page.

http://www.google.com/search?hl=en&l...ckyourself.com

User-agent: *
Disallow: *

for the whole site

No, the robots.txt entry for GFY is correct. "Disallow: /" will exclude the entire site from crawling.

**rowan** · 08-23-2004, 08:44 AM

BTW Google has indexed the root, just not "gofuckyourself.com"

http://www.google.com/search?hl=en&l...om&btnG=Search

**beergood** · 08-23-2004, 08:45 AM

Originally posted by rowan
No, the robots.txt entry for GFY is correct. "Disallow: /" will exclude the entire site from crawling.

See I wasn't entirely sure because I never blocked their bot. You are absolutely right. I looked it up.

**modF** · 08-23-2004, 08:47 AM

I don't think it matters now that you need an account to read the board, those threads should start dropping off soon.

**beergood** · 08-23-2004, 08:48 AM

Its fairly neat that they still don't have a bot that can obey even the most simple robots.txt file.

**rowan** · 08-23-2004, 08:51 AM

The date on the GFY robots.txt is July 8th 2004, so it's possible that it didn't exist (or didn't have the exclude) the last time that Googlebot passed over it.

**beergood** · 08-23-2004, 08:55 AM

Originally posted by rowan
The date on the GFY robots.txt is July 8th 2004, so it's possible that it didn't exist (or didn't have the exclude) the last time that Googlebot passed over it.

Yup. could be. Wonder what it was before.

**chemicaleyes** · 08-23-2004, 08:59 AM

Originally posted by rowan
The date on the GFY robots.txt is July 8th 2004, so it's possible that it didn't exist (or didn't have the exclude) the last time that Googlebot passed over it.

The robots file used to be up and google didn't index GFY, then the file disappeared months ago, google started indexing and now the file is back up. Google will stop indexing.. why is this a big deal for you anyway?

**beergood** · 08-23-2004, 09:03 AM

Originally posted by chemicaleyes
The robots file used to be up and google didn't index GFY, then the file disappeared months ago, google started indexing and now the file is back up. Google will stop indexing.. why is this a big deal for you anyway?

Just something to talk about.

**Basic_man** · 08-23-2004, 09:14 AM

Originally posted by Chupachups
Google doesnt give a shit about robot rules...

Looks like that..

Huh? Google is indexing GFY?

Huh? Google is indexing GFY?

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment