GoFuckYourself.com - Adult Webmaster Forum - View Single Post - Anyone Familiar with error logs, know what this is?

TFCash · 04-22-2001, 12:52 PM

Web_Mistress =

Most nice searchengine spiders will look for the robot.txt file to see if you have any instructions for them, to as what they can spider, and what you want left alone

No need to be worried.

Tim

04-22-2001, 12:52 PM
TFCash Confirmed User Industry Role: Join Date: Apr 2001 Posts: 1,738	Web_Mistress = Most nice searchengine spiders will look for the robot.txt file to see if you have any instructions for them, to as what they can spider, and what you want left alone No need to be worried. Tim