Huh? Google is indexing GFY? - GoFuckYourself.com

rowan · 08-23-2004, 08:40 AM

# robots.txt file for http://www.gfyboard.com/

User-agent: *
Disallow: /

This says nothing is allowed to crawl this site, but there are plenty of pages indexed in Google...

Here's an example

http://www.google.com/search?hl=en&l...&btnG=Sear ch

rowan · 08-23-2004, 08:41 AM

http://www.google.com/search?q=%22Ne...e=off&filter=0

chupachups · 08-23-2004, 08:44 AM

Google doesnt give a shit about robot rules...

julioxp · 08-23-2004, 09:03 AM

Yeap, I have found some pages of known posts at GFY when do a search at google..

julio

beergood · 08-23-2004, 09:40 AM

I think that the robots.txt file for gfy only causes the robot not to index the index page.

http://www.google.com/search?hl=en&l...ckyourself.com

User-agent: *
Disallow: *

for the whole site

rowan · 08-23-2004, 09:42 AM

Quote:

Originally posted by beergood
I think that the robots.txt file for gfy only causes the robot not to index the index page.

http://www.google.com/search?hl=en&l...ckyourself.com

User-agent: *
Disallow: *

for the whole site

No, the robots.txt entry for GFY is correct. "Disallow: /" will exclude the entire site from crawling.

rowan · 08-23-2004, 09:44 AM

BTW Google has indexed the root, just not "gofuckyourself.com"

http://www.google.com/search?hl=en&l...&btnG= Search

beergood · 08-23-2004, 09:45 AM

Quote:

Originally posted by rowan
No, the robots.txt entry for GFY is correct. "Disallow: /" will exclude the entire site from crawling.

See I wasn't entirely sure because I never blocked their bot. You are absolutely right. I looked it up.

modF · 08-23-2004, 09:47 AM

I don't think it matters now that you need an account to read the board, those threads should start dropping off soon.

beergood · 08-23-2004, 09:48 AM

Its fairly neat that they still don't have a bot that can obey even the most simple robots.txt file.

rowan · 08-23-2004, 09:51 AM

The date on the GFY robots.txt is July 8th 2004, so it's possible that it didn't exist (or didn't have the exclude) the last time that Googlebot passed over it.

beergood · 08-23-2004, 09:55 AM

Quote:

Originally posted by rowan
The date on the GFY robots.txt is July 8th 2004, so it's possible that it didn't exist (or didn't have the exclude) the last time that Googlebot passed over it.

Yup. could be. Wonder what it was before.

chemicaleyes · 08-23-2004, 09:59 AM

Quote:

Originally posted by rowan
The date on the GFY robots.txt is July 8th 2004, so it's possible that it didn't exist (or didn't have the exclude) the last time that Googlebot passed over it.

The robots file used to be up and google didn't index GFY, then the file disappeared months ago, google started indexing and now the file is back up. Google will stop indexing.. why is this a big deal for you anyway?

beergood · 08-23-2004, 10:03 AM

Quote:

Originally posted by chemicaleyes
The robots file used to be up and google didn't index GFY, then the file disappeared months ago, google started indexing and now the file is back up. Google will stop indexing.. why is this a big deal for you anyway?

Just something to talk about.

Basic_man · 08-23-2004, 10:14 AM

Quote:

Originally posted by Chupachups
Google doesnt give a shit about robot rules...

Looks like that..

08-23-2004, 08:40 AM	#1
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Huh? Google is indexing GFY? # robots.txt file for http://www.gfyboard.com/ User-agent: * Disallow: / This says nothing is allowed to crawl this site, but there are plenty of pages indexed in Google... Here's an example http://www.google.com/search?hl=en&l...&btnG=Sear ch

08-23-2004, 09:03 AM	#4
julioxp Confirmed User Join Date: Aug 2004 Location: wake me up, before you go go.. Posts: 160	Yeap, I have found some pages of known posts at GFY when do a search at google.. julio __________________ SIG TOO BIG! Maximum 120x60 button and no more than 3 text lines of DEFAULT SIZE and COLOR. Unless your sig is for a GFY top banner sponsor, then you may use a 624x80 instead of a 120x60.

08-23-2004, 09:47 AM	#9
modF Confirmed User Join Date: Aug 2002 Posts: 1,888	I don't think it matters now that you need an account to read the board, those threads should start dropping off soon. __________________ I do things skype:themodF

08-23-2004, 08:41 AM	#2
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	http://www.google.com/search?q=%22Ne...e=off&filter=0

08-23-2004, 08:44 AM	#3
chupachups Confirmed User Join Date: Dec 2002 Location: Sweden/Spain you sum bitch! Posts: 6,576	Google doesnt give a shit about robot rules...

08-23-2004, 09:40 AM	#5
beergood Confirmed User Join Date: Jun 2003 Location: United States Posts: 2,918	I think that the robots.txt file for gfy only causes the robot not to index the index page. http://www.google.com/search?hl=en&l...ckyourself.com User-agent: * Disallow: * for the whole site

08-23-2004, 09:44 AM	#7
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	BTW Google has indexed the root, just not "gofuckyourself.com" http://www.google.com/search?hl=en&l...&btnG= Search

08-23-2004, 09:48 AM	#10
beergood Confirmed User Join Date: Jun 2003 Location: United States Posts: 2,918	Its fairly neat that they still don't have a bot that can obey even the most simple robots.txt file.

08-23-2004, 09:51 AM	#11
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	The date on the GFY robots.txt is July 8th 2004, so it's possible that it didn't exist (or didn't have the exclude) the last time that Googlebot passed over it.