GoFuckYourself.com - Adult Webmaster Forum - View Single Post - Hittbotters

TheFLY · 02-25-2002, 02:27 PM

Please don't read all this if you run a paysite -- this concerns mostly webmasters that make a living through link trades...

Let me paint a scary picture of how a hitbot could elude detection of even SleazyDream... if there are any genius level internet gods out there -- please prove me wrong...

First you need one very good hitbot -- one that simulates the browser perfectly -- ie: clicks through forms, sends and accepts cookies, pulls down images, etc... I know these bots are probably out there -- because if you read the programming forums there's a lot of programmers out there doing this type of thing already... either these programmers are coding search engines, collecting internet statistics, coding dead-link finders, information gathering, etc. etc... but you've got to be a moron if you don't think these guys are also aware they can cheat toplists, click programs, etc... PERL even comes included with robot objects...

Anyhow now assume you have a bot. Now you need lots of proxy servers... now this is where it gets tricky -- and I'm not 100% sure about my facts here -- but I'm pretty sure the following is possible. Most of us know that you need proxy servers to allow your hitbot to simulate many many individual "unique" IP's... There are a certain number of anonymous proxy servers on the net that you can "find" -- I won't go into specifics here... BUT many of these proxy servers are not 100% anonymous -- they leak out information about themselves -- go find yourself a proxy server analysis program like "aatools" -- and you can run tests on proxy servers and see how "anonymous" they really are... I found myself spending many many hours learning about proxy servers... So for the last few days I've been finding ways to detect proxy servers... but then I realized that all of this effort in detecting proxy servers is pretty much useless... and I think this for a few reasons...

1. Even if a proxy server is not anonymous (and seemingly harmless) -- how do we know that this proxy server is not actually being "commanded" by another totally anonymous proxy server...? You see, you can surf the web through a *chain* of proxy servers! I've even seen evidence of chained proxy servers that encrypt their communications... so hypothetically I'm thinking an advanced hitbot could be hiding behind protected chain of anonymous proxy servers and sending commands to a huge army of non-anonymous proxy servers... I wouldn't put this past an evil and greedy person -- even though 99% of webmasters are too dumb to even know what a proxy server is anyhow... I'm sure there are other ways to use proxy servers without being detected -- but there's not much point in investigation further... also to do an in-depth analysis on each and every IP that comes through your site requires a lot of your server's CPU power...

2. Even if I run a website and I detect unusually high proxy server activity from Referrer A -- that doesn't necessarily mean that Site A is using a bot! Odds are that a smart hitbotter is smart enough never to send mass amounts of fake hits from his own site... Much smarter would be to artificially inflate the traffic of your trading partners -- and you grow by association...

3. There are hundreds of bots moving through our sites on a daily basis -- and many of these bots aren't harmfull -- how do you think the search engines index you! Let's pretend I have a site that has 200 link trades -- why would I send thousands of fake hits to my 200 link trades? I could send only 20 fake hits to each of those 200 trades -- now my output increases by 4000 hits! Do you think any webmaster will be able to detect or even notice 20 hitbot hits? No! And yet now my site has increased output by 4000 -- and take this a step further -- if my hitbot sends 20 fake hits to each site -- this bot could hypothetically click 2 links on each site -- so how I have generated 8000 clicks productivity on my trades... You say so what... that's only 40 clicks for each site... but every day this will make your site grow faster than the competition... and your site will have a competitive advantage...

What can we do about this... call me paranoid -- but as far as I can tell there's no way to distinguish a real surfer from a fake -- if there aren't a significant # of fake hits to carefully observe BUT I believe that there's an obvious limit on how much cheating can go on (with large quantities of fake traffic). A hitbot can't simulate human behavior (unless the hitbotter spends a LOT of time making adjustments to the bots behavior) -- and humans are very complicated... I think the best solution is to carefully observe the behavior of your traffic (how you do this can and should be a creative enterprise)... if a particular site has behavior that doesn't make sense -- you should penalize that site somehow (how to do this is up to you...???)

It's still beyond the reach of most of us to do trading based on sales information due to lack of sales tracking information provided by sponsors -- referrer information is not sufficient. Regardless through -- a large trade can be very valuable indirectly even if it does not provide sales...

02-25-2002, 02:27 PM
TheFLY So Fucking Banned Join Date: Jan 2001 Location: http://www.thefly.net/ --- Quit your job and live off steady traffic. Posts: 11,856	Please don't read all this if you run a paysite -- this concerns mostly webmasters that make a living through link trades... Let me paint a scary picture of how a hitbot could elude detection of even SleazyDream... if there are any genius level internet gods out there -- please prove me wrong... First you need one very good hitbot -- one that simulates the browser perfectly -- ie: clicks through forms, sends and accepts cookies, pulls down images, etc... I know these bots are probably out there -- because if you read the programming forums there's a lot of programmers out there doing this type of thing already... either these programmers are coding search engines, collecting internet statistics, coding dead-link finders, information gathering, etc. etc... but you've got to be a moron if you don't think these guys are also aware they can cheat toplists, click programs, etc... PERL even comes included with robot objects... Anyhow now assume you have a bot. Now you need lots of proxy servers... now this is where it gets tricky -- and I'm not 100% sure about my facts here -- but I'm pretty sure the following is possible. Most of us know that you need proxy servers to allow your hitbot to simulate many many individual "unique" IP's... There are a certain number of anonymous proxy servers on the net that you can "find" -- I won't go into specifics here... BUT many of these proxy servers are not 100% anonymous -- they leak out information about themselves -- go find yourself a proxy server analysis program like "aatools" -- and you can run tests on proxy servers and see how "anonymous" they really are... I found myself spending many many hours learning about proxy servers... So for the last few days I've been finding ways to detect proxy servers... but then I realized that all of this effort in detecting proxy servers is pretty much useless... and I think this for a few reasons... 1. Even if a proxy server is not anonymous (and seemingly harmless) -- how do we know that this proxy server is not actually being "commanded" by another totally anonymous proxy server...? You see, you can surf the web through a chain of proxy servers! I've even seen evidence of chained proxy servers that encrypt their communications... so hypothetically I'm thinking an advanced hitbot could be hiding behind protected chain of anonymous proxy servers and sending commands to a huge army of non-anonymous proxy servers... I wouldn't put this past an evil and greedy person -- even though 99% of webmasters are too dumb to even know what a proxy server is anyhow... I'm sure there are other ways to use proxy servers without being detected -- but there's not much point in investigation further... also to do an in-depth analysis on each and every IP that comes through your site requires a lot of your server's CPU power... 2. Even if I run a website and I detect unusually high proxy server activity from Referrer A -- that doesn't necessarily mean that Site A is using a bot! Odds are that a smart hitbotter is smart enough never to send mass amounts of fake hits from his own site... Much smarter would be to artificially inflate the traffic of your trading partners -- and you grow by association... 3. There are hundreds of bots moving through our sites on a daily basis -- and many of these bots aren't harmfull -- how do you think the search engines index you! Let's pretend I have a site that has 200 link trades -- why would I send thousands of fake hits to my 200 link trades? I could send only 20 fake hits to each of those 200 trades -- now my output increases by 4000 hits! Do you think any webmaster will be able to detect or even notice 20 hitbot hits? No! And yet now my site has increased output by 4000 -- and take this a step further -- if my hitbot sends 20 fake hits to each site -- this bot could hypothetically click 2 links on each site -- so how I have generated 8000 clicks productivity on my trades... You say so what... that's only 40 clicks for each site... but every day this will make your site grow faster than the competition... and your site will have a competitive advantage... What can we do about this... call me paranoid -- but as far as I can tell there's no way to distinguish a real surfer from a fake -- if there aren't a significant # of fake hits to carefully observe BUT I believe that there's an obvious limit on how much cheating can go on (with large quantities of fake traffic). A hitbot can't simulate human behavior (unless the hitbotter spends a LOT of time making adjustments to the bots behavior) -- and humans are very complicated... I think the best solution is to carefully observe the behavior of your traffic (how you do this can and should be a creative enterprise)... if a particular site has behavior that doesn't make sense -- you should penalize that site somehow (how to do this is up to you...???) It's still beyond the reach of most of us to do trading based on sales information due to lack of sales tracking information provided by sponsors -- referrer information is not sufficient. Regardless through -- a large trade can be very valuable indirectly even if it does not provide sales...