Quote:
|
Originally Posted by JupZChris
well google is hitting my site already but my stats show robots.txt keeps getting requested
|
I think the spiders default behavior is to request that file, so if it's there the spider will know what to index or not.