Thread: 5 files every website should have
View Single Post
Old 10-24-2009, 07:10 PM  
CYF
Coupon Guru
 
CYF's Avatar
 
Industry Role:
Join Date: Mar 2009
Location: Minneapolis
Posts: 10,973
Quote:
Originally Posted by ArsewithClass View Post
I have robots in my html inc. but what does the robots.txt file actually do please?
robots.txt blocks certain web crawlers from accessing your site. It also can specify certain off-limits areas, like your /members area.
__________________
Webmaster Coupons Coupons and discounts for hosting, domains, SSL Certs, and more!
AmeriNOC Coupons | Certified Hosting Coupons | Hosting Coupons | Domain Name Coupons

CYF is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote