GoFuckYourself.com - Adult Webmaster Forum - View Single Post

xanx · 11-15-2004, 12:23 AM

my index page is just a redirect to my forum area, now most of my traffic just goes right to the forums. Does google bot check the root no matter what for the robots.txt file? Or I guess what am asking is if google checks each directory for a robots file or just the root.

so if I wanted a whole directory off limts to google can I just put a robot file in it or do I have to specify that dir from the root robot file?

11-15-2004, 12:23 AM
xanx Confirmed User Join Date: Apr 2002 Location: Orlando, FL Posts: 3,711	robots.txt question my index page is just a redirect to my forum area, now most of my traffic just goes right to the forums. Does google bot check the root no matter what for the robots.txt file? Or I guess what am asking is if google checks each directory for a robots file or just the root. so if I wanted a whole directory off limts to google can I just put a robot file in it or do I have to specify that dir from the root robot file?