Bad Robots, site rippers

[Captcha]

I am sick with bad robots, site rippers etc
Lots and lots of "noref" hits in ATX!......... I just launch a new site and get fucked with this shit again.... has anyone a list up to date like this one to block this shit?

http://www.javascriptkit.com/howto/htaccess13.shtml

[JamesM]

bots are smart these days, they use fake user agent.
anyways why are worried, is it because of bandwidth ? just curious.

popular sites seems to get this.

[Barry-xlovecam]

her are a few links I found that looked interesting
http://www.wizcrafts.net/exploited-s...blocklist.html

http://antiscraper.com/

From my experience it is ''wack-a-mole'' you need the time or manpower to grep the webserver logs looking for page requests ( or head request ) with no requests for images -- that is a dead give away. Then you have to be careful not to disallow legitimate search engine bots.

you can look for unusual activity ...

using SSH with permissions (in user or root depends on the log's location);

tail -5000 access.log | awk '{print $1}' | sort | uniq -c | sort -n | tail >tofilename
tail -5000 access.log | awk '{print $1}' | sort | uniq -c | sort -r | tail >tofilename
grep 'GET' access.log | cut -d' ' -f1 | sort | uniq -c | sort -r >tofilename

Welcome to the dark side ...