GoFuckYourself.com - Adult Webmaster Forum

GoFuckYourself.com - Adult Webmaster Forum (https://gfy.com/index.php)
-   Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)
-   -   robots.txt question (https://gfy.com/showthread.php?t=388730)

xanx 11-15-2004 12:23 AM

robots.txt question
 
my index page is just a redirect to my forum area, now most of my traffic just goes right to the forums. Does google bot check the root no matter what for the robots.txt file? Or I guess what am asking is if google checks each directory for a robots file or just the root.

so if I wanted a whole directory off limts to google can I just put a robot file in it or do I have to specify that dir from the root robot file?

Steen2 11-15-2004 12:24 AM

It should check /robots.txt and not /forum/robots.txt

I don't see why it would do the /forum/robots.txt

WiredGuy?

sake 11-15-2004 12:26 AM

with a robots.txt file you're telling google which directories to crawl and which not to crawl..

and it goes in your main directory .. public_html or whatever it is

xanx 11-15-2004 12:30 AM

ok thanks.


All times are GMT -7. The time now is 08:40 PM.

Powered by vBulletin® Version 3.8.8
Copyright ©2000 - 2025, vBulletin Solutions, Inc.
©2000-, AI Media Network Inc123