Can someone explain this robots.txt for me - GoFuckYourself.com

mrgica · 09-02-2004, 04:15 AM

What does this robots codes mean for search engines?

http://candicecam.com/robots.txt

ClamSmacker · 09-02-2004, 04:20 AM

http://www.robotstxt.org/wc/norobots.html

woj · 09-02-2004, 04:29 AM

blocks ia_archiver and scooter crawlers
and
blocks all crawlers from /s and /c folders

doesn't really "block" them, just asks them nicely not to do it...

Project-Shadow · 09-02-2004, 05:03 AM

ia_archiver = www.archive.org
Basically means it'll delete/block any archiving of the site.

erehwon · 09-02-2004, 06:02 AM

There's always the FBI's robot file.

http://www.fbi.gov/robots.txt

woj · 09-02-2004, 06:19 AM

Quote:

Originally posted by erehwon
There's always the FBI's robot file.

http://www.fbi.gov/robots.txt

That one is boring, try the one at whitehouse.gov

johnbosh · 09-02-2004, 06:44 AM

Quote:

Originally posted by woj
That one is boring, try the one at whitehouse.gov

wow:P
http://www.whitehouse.gov/robots.txt

erehwon · 09-02-2004, 01:58 PM

Quote:

Originally posted by johnbosh
wow:P
http://www.whitehouse.gov/robots.txt

DAMN!

09-02-2004, 04:15 AM	#1
mrgica Confirmed User Industry Role: Join Date: Jan 2004 Posts: 2,169	Can someone explain this robots.txt for me What does this robots codes mean for search engines? http://candicecam.com/robots.txt __________________ Fuck it dude, lets go bowling

09-02-2004, 04:29 AM	#3
woj <&(©¿©)&> Industry Role: Join Date: Jul 2002 Location: Chicago Posts: 47,882	blocks ia_archiver and scooter crawlers and blocks all crawlers from /s and /c folders doesn't really "block" them, just asks them nicely not to do it... __________________ Custom Software Development, email: woj#at#wojfun#.#com to discuss details or skype: wojl2000 or gchat: wojfun or telegram: wojl2000 Affiliate program tools: Hosted Galleries Manager Banner Manager Video Manager Wordpress Affiliate Plugin Pic/Movie of the Day Fansign Generator Zip Manager

09-02-2004, 06:02 AM	#5
erehwon Confirmed User Industry Role: Join Date: Nov 2003 Location: A secure undisclosed location... Posts: 3,759	There's always the FBI's robot file. http://www.fbi.gov/robots.txt __________________ Money NEVER $leep$...

09-02-2004, 04:20 AM	#2
ClamSmacker Confirmed User Join Date: May 2004 Location: Oztralia Posts: 677	http://www.robotstxt.org/wc/norobots.html

09-02-2004, 05:03 AM	#4
Project-Shadow Confirmed User Industry Role: Join Date: Feb 2003 Posts: 7,340	ia_archiver = www.archive.org Basically means it'll delete/block any archiving of the site.