Thread: robots.txt....WTF?
View Single Post
Old 12-06-2003, 11:19 AM  
gleb
Confirmed User
 
Join Date: Nov 2002
Location: NY
Posts: 311
Quote:
Originally posted by spunky1
SE use them to spider your pages?
yupz, rules for search engines http://www.searchengineworld.com/rob...s_tutorial.htm
gleb is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote