Bad Robots, site rippers

**JamesM** · 01-05-2013, 02:55 AM

bots are smart these days, they use fake user agent.
anyways why are worried, is it because of bandwidth ? just curious.

popular sites seems to get this.

**Barry-xlovecam** · 01-05-2013, 05:07 AM

Longstanding problem

her are a few links I found that looked interesting
http://www.wizcrafts.net/exploited-s...blocklist.html

http://antiscraper.com/

From my experience it is ''wack-a-mole'' you need the time or manpower to grep the webserver logs looking for page requests ( or head request ) with no requests for images -- that is a dead give away. Then you have to be careful not to disallow legitimate search engine bots.

you can look for unusual activity ...

using SSH with permissions (in user or root depends on the log's location);

tail -5000 access.log | awk '{print $1}' | sort | uniq -c | sort -n | tail >tofilename
tail -5000 access.log | awk '{print $1}' | sort | uniq -c | sort -r | tail >tofilename
grep 'GET' access.log | cut -d' ' -f1 | sort | uniq -c | sort -r >tofilename

Welcome to the dark side ...

Bad Robots, site rippers

Bad Robots, site rippers

Comment

Comment