In the past few hours I've been checking out the various hitbots being offered online (want to do as much as I can to be protected from them), and it's really disturbing how many of them are around and what they are able to do.
Most accept cookies, are able to go up to 5 pages deep, support multiple proxies/referrers/user agents, simulate surfer's behaviour etc. There is really no way to detect some of them, and I am starting to think at least 10% of all tgp traffic is in fact hitbot traffic.
Only funny thing I found out is that there are cracks available for most of the shareware hitbots, so at least the makers aren't making any money by selling them
