GoFuckYourself.com - Adult Webmaster Forum - View Single Post - I am an idiot. I accidentally downloaded GFY's robots.txt into my OWN site directory.

rowan · 09-03-2004, 09:48 AM

I usually have at least one SSH session open to one of my servers, and if I want to view the text of a web file I'll usually just type "fetch http://url.com/" into the unix prompt.

I just discovered that when I downloaded GFY's robots.txt to check it my working directory was the web root of one of my major sites.

# robots.txt file for http://www.gfyboard.com/

User-agent: *
Disallow: /

So I effectively blocked all robots from indexing my site... plus overwrote the old robots.txt file. Damn!!

I'm very lucky that I just checked that robots.txt file because I was too lazy to do a web search on how to configure it for another site. Who knows when I would have noticed otherwise.

09-03-2004, 09:48 AM
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	I am an idiot. I accidentally downloaded GFY's robots.txt into my OWN site directory. I usually have at least one SSH session open to one of my servers, and if I want to view the text of a web file I'll usually just type "fetch http://url.com/" into the unix prompt. I just discovered that when I downloaded GFY's robots.txt to check it my working directory was the web root of one of my major sites. # robots.txt file for http://www.gfyboard.com/ User-agent: * Disallow: / So I effectively blocked all robots from indexing my site... plus overwrote the old robots.txt file. Damn!! I'm very lucky that I just checked that robots.txt file because I was too lazy to do a web search on how to configure it for another site. Who knows when I would have noticed otherwise.