Thread: Can someone explain this robots.txt for me
View Single Post
Old 09-02-2004, 06:02 AM  
erehwon
Confirmed User
 
erehwon's Avatar
 
Industry Role:
Join Date: Nov 2003
Location: A secure undisclosed location...
Posts: 3,759
There's always the FBI's robot file.

http://www.fbi.gov/robots.txt
__________________
Money NEVER $leep$...
erehwon is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote