|
Not every site needs a robots.txt file - not every site would benefit by having one. Most sites however do benefit by having a robots.txt file
The largest benefit is the ability to tell google, yahoo, and ask which sections/pages (like affiliate code redirect pages) to not list in google. Doing this keeps rank values focused and not on needless pages or pages that could get you into trouble (like redirects, popup codes, iframes)
I block ai archiver too. You can use it to block mass download bots for TGP's, or include the url to your sitemap.
|