GoFuckYourself.com - Adult Webmaster Forum - View Single Post

mkx · 06-01-2005, 08:03 PM

is it good to have robots.txt for VB forum? This way reply.php, newthread.php, etc, don't get accidently crawled several times?

06-01-2005, 08:03 PM
mkx Confirmed User Industry Role: Join Date: Nov 2003 Location: Toronto Posts: 4,001	robots.txt question is it good to have robots.txt for VB forum? This way reply.php, newthread.php, etc, don't get accidently crawled several times?