GoFuckYourself.com - Adult Webmaster Forum - View Single Post

chodadog · 10-06-2002, 07:46 AM

I think you only need to use robots.txt when you don't want certain sites, or parts of your sites to be spidered. But i could be wrong.

10-06-2002, 07:46 AM
chodadog Confirmed User Join Date: Apr 2002 Posts: 9,736	I think you only need to use robots.txt when you don't want certain sites, or parts of your sites to be spidered. But i could be wrong.