Hitbots

[Libertine]

In the past few hours I've been checking out the various hitbots being offered online (want to do as much as I can to be protected from them), and it's really disturbing how many of them are around and what they are able to do.
Most accept cookies, are able to go up to 5 pages deep, support multiple proxies/referrers/user agents, simulate surfer's behaviour etc. There is really no way to detect some of them, and I am starting to think at least 10% of all tgp traffic is in fact hitbot traffic.

Only funny thing I found out is that there are cracks available for most of the shareware hitbots, so at least the makers aren't making any money by selling them

[Fletch XXX]

Originally posted by punkworld


Only funny thing I found out is that there are cracks available for most of the shareware hitbots, so at least the makers aren't making any money by selling them

The people who get those written are big tgp sites.

To keep up the entire facade, they ofer it to people to also look like theyhave 400k a day.

More tgp trafic = more traffic to trade = more traffic to sell.

[Fro]

Perl & LWP!

[Libertine]

Originally posted by Fro
Perl & LWP!

If you believe perl would be a good language to write a hitbot in, you most likely should stay away from computers for your own safety.

[Fro]

Hehe. What?

[Lane]

Originally posted by punkworld
There is really no way to detect some of them

Wrong

[jimmyf]

Originally posted by Lane



Wrong

Lane know's ALL...

[Fletch XXX]

Just gonna link these 2 threads for future readers.

http://gofuckyourself.com/showthread...threadid=74829

[Libertine]

Originally posted by Lane
Wrong

Oh, really?
If a hitbot spoofs all user variables and uses a good randomizer, you lose many methods of detection.
If it spoofs user behaviour, you lose many more (times between clicks, invisible links, prod, etc).
If it uses (fully) anonymous proxies that you don't know, you lose the last ones.

Only way to detect really good hitbots (that spoof everything, accept cookies, parse javascript and execute java), is to know the proxies they are using.
The info available about a visitor is pretty limited, and if it is all faked and there are no suspicious patterns in behaviour, you have no way to detect the hitbot.

Ofcourse, you can just create a huge list of all proxies in the world, and then cross-reference that to your traffic's IPs. That would work, if you managed to get the proxies those hitbots are using... however, it would be a helluva lot of work to gather all those proxies.

[Chris R]

Originally posted by punkworld


Oh, really?
If a hitbot spoofs all user variables and uses a good randomizer, you lose many methods of detection.
If it spoofs user behaviour, you lose many more (times between clicks, invisible links, prod, etc).
If it uses (fully) anonymous proxies that you don't know, you lose the last ones.

Only way to detect really good hitbots (that spoof everything, accept cookies, parse javascript and execute java), is to know the proxies they are using.
The info available about a visitor is pretty limited, and if it is all faked and there are no suspicious patterns in behaviour, you have no way to detect the hitbot.

Ofcourse, you can just create a huge list of all proxies in the world, and then cross-reference that to your traffic's IPs. That would work, if you managed to get the proxies those hitbots are using... however, it would be a helluva lot of work to gather all those proxies.

people have already done that work for you.

There are other things you can do.
You can't simulate 10,000 real visitors if the person is looking at real logs.
You might not be able to PROVE it is a hitbotter, but you can prove that the traffic from XYZ sucks.

[Alky]

no big deal just port scan for open proxy servers every ip that accesses your pages.

[Libertine]

Originally posted by Chris R


people have already done that work for you.

There are other things you can do.
You can't simulate 10,000 real visitors if the person is looking at real logs.
You might not be able to PROVE it is a hitbotter, but you can prove that the traffic from XYZ sucks.

Actually, people haven't done that. They have just gathered a lot of them. Still enough to be found that aren't on most lists.

Also, you can simulate 10k visitors quite easily... so long as there is nothing that shows them to be hitbot visitors, your logs won't tell you anything. (e.g. user agents, behavioral patterns, etc.)

[X37375787]

Fuck hitbots, use google's pigeons.

[Chris R]

Originally posted by punkworld


Actually, people haven't done that. They have just gathered a lot of them. Still enough to be found that aren't on most lists.

Also, you can simulate 10k visitors quite easily... so long as there is nothing that shows them to be hitbot visitors, your logs won't tell you anything. (e.g. user agents, behavioral patterns, etc.)

Two totally different source of legitimate traffic - one is forums ONLY - one is SEARCH ENGINE ONLY


aol.com 21.43%
IP addr 10.82%
attbi.com 7.07%
comcast.net 10.14%
verizon.net 4.18%
level3.net 2.81%
uu.net 2.45%
rcn.net 2.38%
rr.com 2.36%
cox.net 2.03%
mindspring.com 1.86%
att.net 1.67%
ameritech.net 1.11%
rcn.com 1.07%
adelphia.net 1.06%


IP Address Only 19.57%
aol.com 16.71%
t-dialin.net 4.69%
telenet-ops.be 1.82%
skynet.be 1.64%
wanadoo.fr 1.46%
ttd.es 1.37%
net.tr 1.37%
libero.it 1.18%
tpnet.pl 1.12%
mcbone.net 1.08%

Can you actually simulate all the different ISPs with proxy traffic?

I don't use hitbots, but it seems to me that ths would not be likely (I could be wrong)

If so - by looking at the sources of traffic - you could gauge something about them.

The first group is from a forum dealing with issues local to MD. The second is SE traffic from a keyword known to most in the world.

You can learn something just by looking at the numbers above 1% for the traffic. If you are looking for US traffic - there are ISPs you should be seeing.

To me - in the second set - you have a great deal of foreign ip - 7 of the top 11 are foreign. Even if they aren't hitbotters (which they aren't in this case) - I still know I wouldn't want this for an english language site.

I want US visitors (each person may be different)> I would expect to see ALL THE MAJOR Isps from the US represented.

Can this be done with a hitbot? Like I said - I don't know for sure, but when I go and evalute IPs further - I can see the breakup of different areas over cable providers - furthermore - as we progress down - I can see major universities and companies represented as well.

I would expect to see these from any traffic source - ALL the major ISPs, plus universities, and major companies.

If they can't get anyone of these - you should be able to be suspicious.

Also - even if I don't have all the anonymous proxies - which I agree no one could - if I have one that they use - that would help.

Like I said - I am no expert on hitbotters - maybe they can do all this - if not - I would suggest at looking for they type of traffic you WANT - if you aren't getting it from a source - who cares if their 20,000 chinese visitors aren't hitbotters - are you going to be able to make money off them?

[Libertine]

That would most certainly work, except for one thing: someone using a hitbot would be rather stupid if he used only hitbot traffic.
About 20% hitbot traffic seems more realistic, and would be enough to ensure fast growth. Ofcourse, that would mean all those ISPs would still be represented in the remaining 80% of traffic.

[Chris R]

Originally posted by punkworld
That would most certainly work, except for one thing: someone using a hitbot would be rather stupid if he used only hitbot traffic.
About 20% hitbot traffic seems more realistic, and would be enough to ensure fast growth. Ofcourse, that would mean all those ISPs would still be represented in the remaining 80% of traffic.

That is true. I wonder now many people using hitbots really have real traffic to start with.

I get your point though - they could certainly up their trades and stuff and get more traffic by using the bots.

I think you could still do somme weeding out of dishonest people.

[ServerGenius]

Yes you can create a really good hitbot but they will always have
1 flaw......they can't see the difference between real links and
fake links.

So put a number of links on your site that are not visible to users
and track who clicks them.

Search engine spiders and hitbots will click these....now it's fairly
simple to make a small script that matches the ip's with an array
of SE spider IP's if it doesn't match then write this IP to .htaccess
or whatever action you want to be taken.

DynaMite

[Libertine]

Personally, what I would do as a hitbotter is pretty simple.
Start a site with good design and such, start a number of trades, of which most with relatively small sites using free scripts, buy like 10k traffic from choker, and start adding about 20% hitbot traffic to the traffic going to those small sites with free tradescripts.
Since those sites will most likely be run by newbies, they won't find out, and it will provide you with the extra traffic you need to grow the trades with the large sites larger.
Ofcourse, the traffic going to those large sites would get about 5% hitbot traffic along with the normal traffic, which would be enough to help growth even more, but not enough to get found out.

[jimmyf]

Originally posted by punkworld
Personally, what I would do as a hitbotter is pretty simple.
Start a site with good design and such, start a number of trades, of which most with relatively small sites using free scripts, buy like 10k traffic from choker, and start adding about 20% hitbot traffic to the traffic going to those small sites with free tradescripts.
Since those sites will most likely be run by newbies, they won't find out, and it will provide you with the extra traffic you need to grow the trades with the large sites larger.
Ofcourse, the traffic going to those large sites would get about 5% hitbot traffic along with the normal traffic, which would be enough to help growth even more, but not enough to get found out.

You will get caught... and the people that can & will catch you...will not post in a public forum on how they can catch you...

[Libertine]

Originally posted by DynaSpain
Yes you can create a really good hitbot but they will always have
1 flaw......they can't see the difference between real links and
fake links.

So put a number of links on your site that are not visible to users
and track who clicks them.

Search engine spiders and hitbots will click these....now it's fairly
simple to make a small script that matches the ip's with an array
of SE spider IP's if it doesn't match then write this IP to .htaccess
or whatever action you want to be taken.

DynaMite

No such luck. Two of the hitbots I found yesterday have the option to select which links will be clicked and which will be not, as well as select percentages for groups of links.

[Libertine]

Originally posted by jimmyf
You will get caught... and the people that can & will catch you...will not post in a public forum on how they can catch you...

They sense it with their spidersense? That seems like the only option left, because when the hitbot spoofs all user data and behaviour, and uses proxies you don't have on your list, there really are very little other ways to find out.

[ServerGenius]

Originally posted by punkworld


No such luck. Two of the hitbots I found yesterday have the option to select which links will be clicked and which will be not, as well as select percentages for groups of links.

So have the invisible links rotating....on each load the invisible
link changes.....

I'm sure there is always a way to cheat but if you can make it
that hard that 95% of the fuckers get caught then that's at
least a start

DynaMite

[HQ]

Originally posted by DynaSpain
So have the invisible links rotating....on each load the invisible
link changes.....

How can you make links invisible? How can you make a link invisible to a surfer but make it impossible for a (smart) hitbot to detect that the link is invisible?

[SleazyDream]

an easy way to spot is hitbotter is shear stupidity.
they forget to turn down their hitbots on weekends, holidays, etc because they get scared that their traffic is going down, often they even turn it up on these times, so their traffic stays VERY stable over these dates. Sites with real traffic bounce up and down on weekends and holidays as much as 30-40% but always at least 15-20%.

[ServerGenius]

Originally posted by HQ


How can you make links invisible? How can you make a link invisible to a surfer but make it impossible for a (smart) hitbot to detect that the link is invisible?

many different way: style sheets, no link text, whitespaces,
linkcolor, bgcolor, etc, etc be creative...

DynaMite

[Libertine]

Originally posted by DynaSpain


many different way: style sheets, no link text, whitespaces,
linkcolor, bgcolor, etc, etc be creative...

DynaMite

All those would still make it possible for a smart hitbot to see the link is visible. Something that might work is using a color very close to the background (e.g. #111111 text on #000000 background). That's a different color for machines, but humans hardly see the difference.

[XP]

Post more ideas you idiots! And give your ideas to hitbot coders!
They're really looking new ideas, protection systems from idiots, who don't think while writing!

Think before write

[Lane]

ah cmon, none of your ideas are new and there are several other ways to detect hitbots.. just stop giving away your ideas to public (and to possible hitbot coders)..

if you have anything unique, contact me.. i will be working on a nice script soon..

[HQ]

I'll continue this conversation over ICQ to anyone familiar with hitbots/hitbot-detection/etc. I have written my own traffic scripts and I would like to discuss ideas about hitbot detection with some webmasters who are experienced in it.

94968434

[ServerGenius]

Originally posted by punkworld


All those would still make it possible for a smart hitbot to see the link is visible. Something that might work is using a color very close to the background (e.g. #111111 text on #000000 background). That's a different color for machines, but humans hardly see the difference.

hence the be creative statement.....I was only summing up
some obvious examples....but yes I agree not to make others
wiser than they already are....you guys are right about that.

DynaMite