Thread: Internet Shinchakubin -Spider crawling my site?
View Single Post
Old 01-12-2004, 03:26 AM  
Chris
Too lazy to set a custom title
 
Chris's Avatar
 
Industry Role:
Join Date: May 2003
Location: icq: 71462500 Skype: Jupzchris
Posts: 27,880
Quote:
Originally posted by Rorschach
First result on Google:

http://www.robotstxt.org/wc/active/html/myweb.html
Um that is the first thing I did .. was just posting here to see if anyone else ever gets that and to see if it will even benifit me or if I should just block it
__________________
[email protected]
Chris is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote