I'm trying to optimize my SE traffic. I was told to put this in my main directory
# /robots.txt for
http://www.yoursite.com/
# comments to
[email protected]/
User-agent: *
Disallow:
Allow: *
Im cofused as to what I place in for user-agent and disallow/allow parts.And does this help with robots spidering your other pages?Can anybody help with this?.Much appreciated.