GoFuckYourself.com - Adult Webmaster Forum

- Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)

- - robots.txt question (https://gfy.com/showthread.php?t=475436)

mkx	06-01-2005 08:03 PM

robots.txt question

is it good to have robots.txt for VB forum? This way reply.php, newthread.php, etc, don't get accidently crawled several times?

Sams_gfy

06-02-2005 04:56 AM

Yes. It IS good.

Muff	06-02-2005 05:33 AM

I think you can limit certain urls from being crawled... Google it to see for sure.

Muff	06-02-2005 05:37 AM

http://www.searchengineworld.com/rob...s_tutorial.htm

User-agent: *
Disallow: newreply.php

Then just add other lines for files.

All times are GMT -7. The time now is 03:29 AM.