Thread: robots.txt question
View Single Post
Old 06-02-2005, 05:33 AM  
Muff
Confirmed User
 
Join Date: Mar 2001
Location: Toronto
Posts: 1,782
I think you can limit certain urls from being crawled... Google it to see for sure.
Muff is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote