|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
06-01-2005, 08:03 PM
|
#1 |
|
Confirmed User
Industry Role:
Join Date: Nov 2003
Location: Toronto
Posts: 4,001
|
robots.txt question
is it good to have robots.txt for VB forum? This way reply.php, newthread.php, etc, don't get accidently crawled several times?
|
|
|
|
06-02-2005, 04:56 AM
|
#2 |
|
Confirmed User
Join Date: Aug 2004
Location: in your imagination...
Posts: 491
|
Yes. It IS good.
__________________
 Work with PPC SE ( Umax, 3FN Marketing) and GET PAID FOR EVERY CLICK !!!  Next step after PPC - Pay Per Lead/Action Good, filtered traffic: Chocker, Traffic Holder || My favourite hosting: HQ Host |
|
|
|
|
06-02-2005, 05:33 AM
|
#3 |
|
Confirmed User
Join Date: Mar 2001
Location: Toronto
Posts: 1,782
|
I think you can limit certain urls from being crawled... Google it to see for sure.
|
|
|
|
|
06-02-2005, 05:37 AM
|
#4 |
|
Confirmed User
Join Date: Mar 2001
Location: Toronto
Posts: 1,782
|
http://www.searchengineworld.com/rob...s_tutorial.htm
User-agent: * Disallow: newreply.php Then just add other lines for files. |
|
|
|
