What Do I Put In My Robot.txt File?
 

I'm trying to optimize my SE traffic. I was told to put this in my main directory
# /robots.txt for http://www.yoursite.com/
# comments to [email protected]/

User-agent: *
Disallow:
Allow: *
Im cofused as to what I place in for user-agent and disallow/allow parts.And does this help with robots spidering your other pages?Can anybody help with this?.Much appreciated.

this help?

http://www.searchengineworld.com/rob...s_tutorial.htm

robots.txt files aren't particularly important, unless you get annoyed with the 404 entries in your logs, or you're trying to keep a particular search engine out of part of your site.

For example, if you built a set of pages specifically for Inktomi on http://www.yoursite.com/inktomi/ and another set for Altavista on http://www.yoursite.com/altavista/ you could do the following:

User-agent: *
Disallow:

User-agent: Slurp
Disallow: /altavista/

User-agent: Scooter
Disallow: /inktomi/

This will keep Scooter out of the inktomi files and Slurp out of the Altavista files.

For more information check out the following website:

http://www.robotstxt.org/wc/robots.html

Cool..Thanks for the information :)