Thread: robots.txt
View Single Post
Old 12-15-2004, 01:54 PM  
European Lee
Confirmed User
 
Join Date: Dec 2002
Location: Daytona Beach
Posts: 7,133
Why woud you want to stop them looking for one?

Just dont upload one, if there isnt one on the server, it will return a 404 error and youll at least know your site is being spidered.

Just a thought.

Regards,

Lee
European Lee is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote