Can someone explain this robots.txt for me

**ClamSmacker** · 09-02-2004, 03:20 AM

http://www.robotstxt.org/wc/norobots.html

**woj** · 09-02-2004, 03:29 AM

blocks ia_archiver and scooter crawlers
and
blocks all crawlers from /s and /c folders

doesn't really "block" them, just asks them nicely not to do it...

**Project-Shadow** · 09-02-2004, 04:03 AM

ia_archiver = www.archive.org
Basically means it'll delete/block any archiving of the site.

**erehwon** · 09-02-2004, 05:02 AM

There's always the FBI's robot file.

http://www.fbi.gov/robots.txt

**woj** · 09-02-2004, 05:19 AM

Originally posted by erehwon
There's always the FBI's robot file.

http://www.fbi.gov/robots.txt

That one is boring, try the one at whitehouse.gov

**johnbosh** · 09-02-2004, 05:44 AM

Originally posted by woj
That one is boring, try the one at whitehouse.gov

wow:P
http://www.whitehouse.gov/robots.txt

**erehwon** · 09-02-2004, 12:58 PM

Originally posted by johnbosh
wow:P
http://www.whitehouse.gov/robots.txt

DAMN!

Can someone explain this robots.txt for me

Can someone explain this robots.txt for me

Comment

Comment

Comment

Comment

Comment

Comment

Comment