GoFuckYourself.com - Adult Webmaster Forum - Can someone explain this robots.txt for me

GoFuckYourself.com - Adult Webmaster Forum (https://gfy.com/index.php)

- Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)

- - Can someone explain this robots.txt for me (https://gfy.com/showthread.php?t=350589)

mrgica

09-02-2004 04:15 AM

Can someone explain this robots.txt for me

What does this robots codes mean for search engines?

http://candicecam.com/robots.txt

ClamSmacker

09-02-2004 04:20 AM

http://www.robotstxt.org/wc/norobots.html

woj	09-02-2004 04:29 AM

blocks ia_archiver and scooter crawlers
and
blocks all crawlers from /s and /c folders

doesn't really "block" them, just asks them nicely not to do it...

Project-Shadow

09-02-2004 05:03 AM

ia_archiver = www.archive.org
Basically means it'll delete/block any archiving of the site.

erehwon

09-02-2004 06:02 AM

There's always the FBI's robot file. :1orglaugh

http://www.fbi.gov/robots.txt

woj	09-02-2004 06:19 AM

Quote:

Originally posted by erehwon
There's always the FBI's robot file. :1orglaugh

http://www.fbi.gov/robots.txt

That one is boring, try the one at whitehouse.gov

johnbosh

09-02-2004 06:44 AM

Quote:

Originally posted by woj
That one is boring, try the one at whitehouse.gov

wow:P
http://www.whitehouse.gov/robots.txt

erehwon

09-02-2004 01:58 PM

Quote:

Originally posted by johnbosh
wow:P
http://www.whitehouse.gov/robots.txt

DAMN!

All times are GMT -7. The time now is 11:13 AM.