Thread: Robots.txt
View Single Post
Old 10-27-2008, 10:18 AM  
baddog
So Fucking Banned
 
Industry Role:
Join Date: Apr 2001
Location: the beach, SoCal
Posts: 107,089
Use it to keep spiders out of certain parts of your site.
baddog is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote