I am an idiot. I accidentally downloaded GFY's robots.txt into my OWN site directory. - GoFuckYourself.com

rowan · 09-03-2004, 09:48 AM

I usually have at least one SSH session open to one of my servers, and if I want to view the text of a web file I'll usually just type "fetch http://url.com/" into the unix prompt.

I just discovered that when I downloaded GFY's robots.txt to check it my working directory was the web root of one of my major sites.

# robots.txt file for http://www.gfyboard.com/

User-agent: *
Disallow: /

So I effectively blocked all robots from indexing my site... plus overwrote the old robots.txt file. Damn!!

I'm very lucky that I just checked that robots.txt file because I was too lazy to do a web search on how to configure it for another site. Who knows when I would have noticed otherwise.

TheLegacy · 09-03-2004, 09:50 AM

you still have the whole weekend to find someway else to screw up

brand0n · 09-03-2004, 09:50 AM

does google even use robots.txt any more?

rowan · 09-03-2004, 09:55 AM

Quote:

Originally posted by brand0n
does google even use robots.txt any more?

Actually the reason I was checking GFY's robots.txt was because I noticed recently that it was indexed in Google.

Looks like GFY's robots.txt disappeared for a little while, which is how the board got into Google again.

09-03-2004, 09:48 AM	#1
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	I am an idiot. I accidentally downloaded GFY's robots.txt into my OWN site directory. I usually have at least one SSH session open to one of my servers, and if I want to view the text of a web file I'll usually just type "fetch http://url.com/" into the unix prompt. I just discovered that when I downloaded GFY's robots.txt to check it my working directory was the web root of one of my major sites. # robots.txt file for http://www.gfyboard.com/ User-agent: * Disallow: / So I effectively blocked all robots from indexing my site... plus overwrote the old robots.txt file. Damn!! I'm very lucky that I just checked that robots.txt file because I was too lazy to do a web search on how to configure it for another site. Who knows when I would have noticed otherwise.

09-03-2004, 09:50 AM	#2
TheLegacy SEO Connoisseur Industry Role: Join Date: Apr 2003 Location: Brantford, Ontario Posts: 17,099	you still have the whole weekend to find someway else to screw up __________________ SEO Connoisseur Microsoft Teams: Robert Warren SEO Telegram: @TheLegacy54 RobertWarrenSEO.com

09-03-2004, 09:50 AM	#3
brand0n been very busy Join Date: Nov 2002 Location: the queen city Posts: 26,983	does google even use robots.txt any more? __________________ want to buy this spot for cheap? it is of course for sale. long term deals are always the best bet. brand0n/ at/ a o l dot commies.