is it good to have robots.txt for VB forum? This way reply.php, newthread.php, etc, don't get accidently crawled several times?
-
-
Yes. It IS good.
Work with PPC SE ( Umax, 3FN Marketing) and GET PAID FOR EVERY CLICK !!!
Next step after PPC - Pay Per Lead/Action
Good, filtered traffic: Chocker, Traffic Holder || My favourite hosting: HQ Host -
I think you can limit certain urls from being crawled... Google it to see for sure.Comment
-
http://www.searchengineworld.com/rob...s_tutorial.htm
User-agent: *
Disallow: newreply.php
Then just add other lines for files.Comment
Comment