|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
11-15-2004, 12:23 AM
|
#1 |
|
Confirmed User
Join Date: Apr 2002
Location: Orlando, FL
Posts: 3,711
|
robots.txt question
my index page is just a redirect to my forum area, now most of my traffic just goes right to the forums. Does google bot check the root no matter what for the robots.txt file? Or I guess what am asking is if google checks each directory for a robots file or just the root.
so if I wanted a whole directory off limts to google can I just put a robot file in it or do I have to specify that dir from the root robot file? |
|
|
|
11-15-2004, 12:24 AM
|
#2 |
|
Confirmed User
Join Date: Feb 2004
Location: Vancouver, Canada
Posts: 7,662
|
It should check /robots.txt and not /forum/robots.txt
I don't see why it would do the /forum/robots.txt WiredGuy?
__________________
ICQ: 2262.73945 |
|
|
|
|
11-15-2004, 12:26 AM
|
#3 |
|
Confirmed User
Industry Role:
Join Date: Apr 2002
Location: Toronto
Posts: 2,646
|
with a robots.txt file you're telling google which directories to crawl and which not to crawl..
and it goes in your main directory .. public_html or whatever it is |
|
|
|
|
11-15-2004, 12:30 AM
|
#4 |
|
Confirmed User
Join Date: Apr 2002
Location: Orlando, FL
Posts: 3,711
|
ok thanks.
|
|
|
|
