How to stop a hitbot ?
 

One of my sites gets 1k an hour of no ref hits. Its not a cheater, someone just wants to fuck up my site. These hits don't click anywhere.

How the fuck do i stop it ?

I think it's a sitesucker.. just block all of those fuckers in your htaccess and see if the norefs slow down. If you've already done that, disregard this. :)

is the ip the same?

no the IPs are different - i entered them to .htaccess but it doenst help much


deny from 205.188.208.103
deny from 64.12.96.11
deny from 198.81.26.137
deny from 66.131.66.251
deny from 24.159.40.28
deny from 211.28.96.68
deny from 195.93.34.9
deny from 211.22.33.9
deny from 195.93.50.9
deny from 200.52.160.2
deny from 68.84.9.150
deny from 208.5.238.167
deny from 68.54.8.118
deny from 195.92.168.168
deny from 68.15.50.22
deny from 219.97.35.155
deny from 62.155.204.24
deny from 62.252.128.10
deny from 24.116.65.69
deny from 64.201.192.220
deny from 24.197.0.6
deny from 203.220.107.190
deny from 64.230.97.92
deny from 217.231.17.118
deny from 195.92.67.65
deny from 4.62.115.56
deny from 156.34.213.233
deny from 66.169.3.21
deny from 68.104.26.218
deny from 152.163.188.69
deny from 24.202.241.145
deny from 61.94.203.158
deny from 61.211.130.19
deny from 217.231.17.118
deny from 198.81.26.137
deny from 24.52.212.36
deny from 24.197.0.6
deny from 61.170.224.53
deny from 64.152.169.50
deny from 67.33.176.17
deny from 24.208.79.113
deny from 68.102.90.139
deny from 65.30.198.37
deny from 67.36.177.120
deny from 194.117.133.180
deny from 66.57.188.185
deny from 65.113.238.33
deny from 24.80.82.57
deny from 202.188.67.9
deny from 145.53.35.115

TGF whats a sitesucker ?

**cue fanfare**

http://www.gothicsex.com/brick/brick.gif

Hey Dark did you ever fig it out mate? I need to get something like that as well for my tgp site everything i have tried make the page not load :(

TOM

A site sucker is a program that will leech your entire site.. Teleport Pro does this for instance

They're usually used to download content without having to view the ads on the HTML

A program that downloads your pics and shit. Sorta offline browser.
like Download Ninja 7.0 etc. :)

here is a list to get you started, if anyone has an updated list please share.


goes into your .htaccess file if you dont know what your doing have your server admin add it..

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*WebZIP.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Iria.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Stripper.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Offline.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Copier.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Crawler.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Snagger.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Teleport.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Reaper.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Wget.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Grabber.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Sucker.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Downloader.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Siphon.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Collector.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Mag-Net.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Widow.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Pockey.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*DA.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Snake.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*BackWeb.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*gotit.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Vacuum.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*SmartDownload.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Pump.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*HMView.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Ninja.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*HTTrack.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*JOC.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*likse.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Memo.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*pcBrowser.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*SuperBot.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*leech.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Mirror.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Recorder.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*GrabNet.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Likse.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Navroad.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*attach.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Magnet.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Surfbot.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Bandit.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Ants.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Buddy.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Whacker.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*DISCoPump.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Drip.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*EirGrabber.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*ExtractorPro.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*EyeNetIE.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*FlashGet.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*GetRight.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Gets.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Go!Zilla.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Go-Ahead-Got-It.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Grafula.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*IBrowse.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*InterGET.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*InternetNinja.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*JetCar.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*JustView.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*MIDownool.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*MisterPiX.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*NearSite.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*NetSpider.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*OfflineExplorer.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*PageGrabber.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*PapaFoto.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Pockey.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*ReGet.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Slurp.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*SpaceBison.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*SuperHTTP.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Teleport.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebAuto.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebCopier.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebFetch.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebReaper.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebSauger.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebStripper.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebWhacker.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebZIP.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebImageCollector.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*WebSucker.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Webster.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Wget.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*eCatch.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*ia_archiver.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*lftp.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*tAkeOut.*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*FileHound.*$
RewriteRule /*$ http://www.redirectURLhere [L,R]

click on the button stop or disable... it depends on the hitbot :glugglug

Just a Q, I have a cronjob on my site which uses wget. But when I paste the code above into my .htaccess, the wget function gets redirected too ...

How can I stop this ?

Thanks!

va2k - i haven't figured it out yet. I still get 20k of hitbots a day.

speedy26 - that thing will slow down the server.

htaccess slows down the server and or page? you sure?

put a hidden link somewhere on your page to a script that will save the IP accessing it and than deny it in your .htaccess,
have in mind that a lot of IPs in your .htaccess will slow down
your web server, so find a way to ban them in your traffic trade script.

digital

Digging up an old thread i know... but i have a similar prob to this... and only after two days of ip banning (had over 400 ips banned) i posted on another board and someone notified me of those websuckers programs. So now i put that whole htaccess thing in there.. but still it continues @ 6-10k hits per hour.

Does anyone have an updated list? or is there any other thing that could be causing it? As its been going on for three days.. and i cant stop it.

Steve

new versions of UCJ have IP banning features - so if an IP hits your page too much, it winds up getting banned (I think the default ban is 2 hours) - UCJ basically auto-updates the htaccess file.

Im using TM3.. so unfortunately i cant use that feature. My next site will be on UCJ.. so good to know about that feature

can tell you what I did on one of my sites, and they quit after they found out they were getting htm's instead of .jpg's.

Try this filter, It's completely PHP written

http://tgp.icefire.org/filter.php.txt

If you want to see my configuration... then hit this:
http://tgp.icefire.org/filter.php?dump

Hope this helps you something

Nice script you have there -
Most hit botters play and soon leave after doing a bit of screwing up - This thread is about more permanent bots - The bastards that never go away -
I currently have bots on about 12 sites - Constantly, even though I have removed all traffic scripts - The c-u-n-t behind it wants to make sure I do not use scripts again - However - He has now fucked himself as we have traced via proxies to his original ip -

I am only running one script now - I started using it last week and sure enough, as soon as I did the bots increased -
So I used htaccess to block it -
I have three guys working shifts sitting and watching the site's activity and as a new ip comes in it is watched over - If the ip becomes over-active, it is blocked right away -
Today saw a new turn up - As proxies were being made useless - We had regular ips hitting hard - We didn't see it right away - But as we are always cross checking trades, we did find it in reasonably good time and was able to get this fixed.
We had Eskoemen go in tonight and he found three ips which we had missed - But these had caused no damage -
If you have read any of my posts - I do not need to say which SUNNY fucking Polish brainless, gutless brotherfucking C_U_N_T is behind this -

That does indeed look like a good script, but how would i implement this? Im not an expert with php nor htaccess :(

[linux root prompt]# iptables -A INPUT -s [OFFENDING IP] -j DROP

I thought Slurp was the Inktomi spider?

Just remove this line: RewriteCond %{HTTP_USER_AGENT} ^.*Wget.*$ [OR]

I just checked the top 4 and all of them were AOL proxies.
Probably not a very good idea to block them :winkwink:

205.188.208.103
64.12.96.11
198.81.26.137
66.131.66.251

I'm guessing you add this to the top of the page:

The page you put this on would have to end with an extension like .shtml or .php
But I could be wrong, so can anyone confirm this?

´

thanks alot!

Do you know how could I change the script above it to redirect to diferent pages based on the language?

once again cheers :thumbsup

Could anyone Confirm what vox said? So i know how to stop this websucker. I am getting 10k hits per hour of it.. So its a big pain in the ass. Unfortunately im not that good with php/htaccess and all that.

So some clear explenation on how to implement this please :)

Steve

I am working on the geotargeted version that will add automatically hitbots to the filter based on activity patterns. As always, it will be free :)

It will be done for day 15.

If you really need this feature very urgent you can use the $_SERVER["HTTP_ACCEPT_LANGUAGE"] variable.
Unfortunately some browsers (like the mine) accept several languages or do not really have a standard (MSIE 6.0 and Netscape 7.x), so the only way to get the real location is geotargeting.

:thumbsup

any news regarding this? :)

if those ip are anonymous proxies maybe someone is trying dictionary attack using some kind of soft for example AccessDiver

I'm not sure if he's released the code seperately, but he did make it available as part of the new version of CJOverkill.

Guess I'll repeat this.

I thought Slurp was the Inktomi spider?

Slurp is the Inktomi spider.

Using htaccess to eat up outbound clicks with no cookie presented will defeat most of the site suckers. However, it will also defeat legitimate spiders.

*** WARNING: THIS IS UNTESTED ***

RewriteEngine on
RewriteCond %{HTTP_COOKIE} !^.*ttt_cookie.*$
RewriteRule .*ttt-out.php.*$ /nocookies.html

If a surfer loads ttt-out.php and does not present the cookie 'ttt_cookie' (which is normally set when they load the main page), they will be redirected to /nocookies.html

Thank you. :)

Great info guys.

:thumbsup
Oz

You're welcome. :)

How to blackhole IP's at the server level:


Generic (should work on any unix OS)
/sbin/route add IP 333.444.555.666

Generic Linux
/sbin/route add -host IP gw 333.444.555.666

Newer versions of Linux support the reject flag now. (I believe kernel 2.2+)
For linux users, try this one first...if it tells you reject is not a valid flag, use the generic one.
/sbin/route add -host IP reject

Generic BSD (BSDI, OpenBSD, NetBSD, FreeBSD)
/sbin/route add IP 333.444.555.666

Generic Sun
/usr/sbin/route add IP 333.444.555.666 1

NEXTSTEP
/usr/etc/route add IP 127.0.0.1 1

FreeBSD
route add -net IP -netmask 255.255.255.255 127.0.0.1 -blackhole

Digital UNIX 4.0D (OSF/1 / Compaq Tru64 UNIX)
/sbin/route add -host -blackhole IP 127.0.0.1


where "IP" is replaced with the IP you want to block.

(courtesy of the iprotect docs)