View Single Post
Old 01-16-2005, 06:17 PM  
Yug
Confirmed User
 
Join Date: Dec 2002
Location: Brisbane, Australia
Posts: 218
put in a robots.txt file in your root directory.

In it, specify the directories or files you dont want the spiders to access:

for example, if u dont want spiders accessing any files in your site named 'email.htm' u would put:

Disallow: email.htm

if you dont want spiders accessing any specific folders, like the cgi-bin folder for example, u would put:

Disallow: /cgi-bin/


And last of all, if you have a pr of 6, what is the website URL? I think you should put a link to my site on it for helping you out
__________________
YugSTAR.com
The online cartoon about the adult internet industry
Yug is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote