Huh? Google is indexing GFY?
 

# robots.txt file for http://www.gfyboard.com/

User-agent: *
Disallow: /



This says nothing is allowed to crawl this site, but there are plenty of pages indexed in Google...

Here's an example

http://www.google.com/search?hl=en&l...&btnG=Sear ch

Google doesnt give a shit about robot rules...

Yeap, I have found some pages of known posts at GFY when do a search at google..

julio

I think that the robots.txt file for gfy only causes the robot not to index the index page.

http://www.google.com/search?hl=en&l...ckyourself.com


User-agent: *
Disallow: *

for the whole site

No, the robots.txt entry for GFY is correct. "Disallow: /" will exclude the entire site from crawling.

BTW Google has indexed the root, just not "gofuckyourself.com" :)

http://www.google.com/search?hl=en&l...&btnG= Search

See I wasn't entirely sure because I never blocked their bot. You are absolutely right. I looked it up.

I don't think it matters now that you need an account to read the board, those threads should start dropping off soon.

Its fairly neat that they still don't have a bot that can obey even the most simple robots.txt file.

The date on the GFY robots.txt is July 8th 2004, so it's possible that it didn't exist (or didn't have the exclude) the last time that Googlebot passed over it.

Yup. could be. Wonder what it was before.

The robots file used to be up and google didn't index GFY, then the file disappeared months ago, google started indexing and now the file is back up. Google will stop indexing.. why is this a big deal for you anyway?

Just something to talk about.

Looks like that..