Hitbot Detecting Software

[notjoe]

I am writing some scripts and one of them is to detect a hitbot on a TGP or any site for that matter. I am just wondering what people think is the best way to nail a hitbot/detect it?

The couple things i can think of are:
1) Roughly the same amount of time will laspe between requests.
2) Large volume of unique ips doing the exact same amount of hits.

Lets hear some more suggestions.

Joe

[Va2k]

What about checking for the scripts out there that download all your images etc...

[Zyber]

Originally posted by notjoe
I am writing some scripts and one of them is to detect a hitbot on a TGP or any site for that matter. I am just wondering what people think is the best way to nail a hitbot/detect it?

The couple things i can think of are:
1) Roughly the same amount of time will laspe between requests.
2) Large volume of unique ips doing the exact same amount of hits.

Lets hear some more suggestions.

Joe

All the stuff you want to check for can be adjusted by some creative hitbots... the best way to catch a hitbot is not to tell what you are going to look for...
Never assume anything... I think you should just make a series of tests on the traffic and hope that one of your tests is one the hitbotter had not been prepared for.

[notjoe]

Originally posted by va2k
What about checking for the scripts out there that download all your images etc...

Images can be protected with mod_rewrite, unless the program is spoofing the referring url, which some programs do do ;)

[Theo]

webmasters like notjoe will put us out of business

[Amputate Your Head]

Gary's hitbot is undetectable....

[SleazyDream]

Originally posted by Amputate Your Head
Gary's hitbot is undetectable....

it even fakes signups that don't charge back..........

[Mr.Fiction]

If you make the script publicly available, it won't work.

[Mutt]

i'm not much of a techie but if you can detect proxies like ProxyPass can wouldn't this be a big help in a hitbot detection script because hitbotters use proxies? So figure out a threshold number for proxies, if the number of proxies detected is above normal, good sign it's a hitbotter??

[Mutt]

speaking of URL spoofers, anybody got any good ideas how to protect a leased plugin from these spoofer programs that spoof the referring URL of a valid customer? Not sure cookies will work any better because i think some of these programs also can accept a cookie. Has to be a good way to control access to a plugin other than requiring every user type in a username password combination.