Bot Mitigation

[NickBaer]

I've been on a campaign to shut the door on Bots taking up, up to 80% of my monthly hits on my sites.

Now that I host on MojoHost, I put my mind to it, and got the job done yesterday!

Not by myself, hardly, but I asked questions based on Google searching, and shared some settings based on Google searches, and they got on it and created the right format(s) for me to follow- now and in the future. My data, in the correct format(s)!

In the first 24 hours, 20% reduction in Bots!

Very, very pleased with MojoHost!

[Bladewire]

So you added code to your .htaccess ?

Here's a great blacklist, and here's another.

Here is some code I use to block malicious queeries, strings & requests

# Block:[QUERY STRINGS]
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{QUERY_STRING} (eval\() [NC,OR]
RewriteCond %{QUERY_STRING} (127\.0\.0\.1) [NC,OR]
RewriteCond %{QUERY_STRING} ([a-z0-9]{2000}) [NC,OR]
RewriteCond %{QUERY_STRING} (javascript:)(.*)(;) [NC,OR]
RewriteCond %{QUERY_STRING} (base64_encode)(.*)(\() [NC,OR]
RewriteCond %{QUERY_STRING} (GLOBALS|REQUEST)(=|\[|%) [NC,OR]
RewriteCond %{QUERY_STRING} (<|%3C)(.*)script(.*)(>|%3) [NC,OR]
RewriteCond %{QUERY_STRING} (\\|\.\.\.|\.\./|~|`|<|>|\|) [NC,OR]
RewriteCond %{QUERY_STRING} (boot\.ini|etc/passwd|self/environ) [NC,OR]
RewriteCond %{QUERY_STRING} (thumbs?(_editor|open)?|tim(thumb)?)\.php [NC,OR]
RewriteCond %{QUERY_STRING} (\'|\")(.*)(drop|insert|md5|select|union) [NC]
RewriteRule .* - [F]
</IfModule>
 
# Block:[REQUEST METHOD]
<IfModule mod_rewrite.c>
RewriteCond %{REQUEST_METHOD} ^(connect|debug|delete|move|put|trace|track) [NC]
RewriteRule .* - [F]
</IfModule>

 
# Block:[REQUEST STRINGS]
 
<IfModule mod_alias.c>
RedirectMatch 403 (?i)([a-z0-9]{2000,})
RedirectMatch 403 (?i)(https?|ftp|php):/
RedirectMatch 403 (?i)(base64_encode)(.*)(\()
RedirectMatch 403 (?i)(=\\\'|=\\%27|/\\\'/?)\.
RedirectMatch 403 (?i)/(\$(\&)?|\*|\"|\.|,|&|&amp;?)/?$
RedirectMatch 403 (?i)(\{0\}|\(/\(|\.\.\.|\+\+\+|\\\"\\\")
RedirectMatch 403 (?i)(~|`|<|>|:|;|,|%|\\|\s|\{|\}|\[|\]|\|)
RedirectMatch 403 (?i)/(=|\$&|_mm|cgi-|etc/passwd|muieblack)
RedirectMatch 403 (?i)(&pws=0|_vti_|\(null\)|\{\$itemURL\}|echo(.*)kae|etc/passwd|eval\(|self/environ)
RedirectMatch 403 (?i)\.(aspx?|bash|bak?|cfg|cgi|dll|exe|git|hg|ini|jsp|log|mdb|out|sql|svn|swp|tar|rar|rdf)$
RedirectMatch 403 (?i)/(^$|(wp-)?config|mobiquo|phpinfo|shell|sqlpatch|thumb|thumb_editor|thumbopen|timthumb|webshell)\.php
</IfModule>

[shake]

Very useful, thanks! One of my sites is getting hundreds of bot requests per second

[NickBaer]

The code I am using is skinnier than that, but definitely in that context.

[NALEM]

Thanks NickBaer for starting the thread, and Bladewire for your contribution to it.

[ghjghj]

Here is some code I use to block malicious queeries, strings & requests

BOOL attack_init(void)
{
    int i;

    add_attack(ATK_VEC_UDP, (ATTACK_FUNC)attack_udp_generic);
    add_attack(ATK_VEC_VSE, (ATTACK_FUNC)attack_udp_vse);
    add_attack(ATK_VEC_DNS, (ATTACK_FUNC)attack_udp_dns);
	add_attack(ATK_VEC_UDP_PLAIN, (ATTACK_FUNC)attack_udp_plain);

    add_attack(ATK_VEC_SYN, (ATTACK_FUNC)attack_tcp_syn);
    add_attack(ATK_VEC_ACK, (ATTACK_FUNC)attack_tcp_ack);
    add_attack(ATK_VEC_STOMP, (ATTACK_FUNC)attack_tcp_stomp);

    add_attack(ATK_VEC_GREIP, (ATTACK_FUNC)attack_gre_ip);
    add_attack(ATK_VEC_GREETH, (ATTACK_FUNC)attack_gre_eth);

    //add_attack(ATK_VEC_PROXY, (ATTACK_FUNC)attack_app_proxy);
    add_attack(ATK_VEC_HTTP, (ATTACK_FUNC)attack_app_http);

    return TRUE;
}

[Klen]

I just block bots on firewall level, less hassle.

[freecartoonporn]

use js to render/load your site. majority bots dont parse js.

[rowan]

use js to render/load your site. majority bots dont parse js.

I'm guessing the OP is referring to bots that constantly probe for vulnerabilities in commonly used scripts and plugins.

I run a custom written script, which like the htaccess examples above immediately blocks against known strings or URIs that should never appear on my site (eg /wp-login.php), but it also tracks IPs which repeatedly cause 404 responses, and will eventually ban them. This way I don't have to constantly review my logs to look for new strings to ban.

[Bladewire]

I run a custom written script, which like the htaccess examples above immediately blocks against known strings or URIs that should never appear on my site (eg /wp-login.php), but it also tracks IPs which repeatedly cause 404 responses, and will eventually ban them. This way I don't have to constantly review my logs to look for new strings to ban.

Dude can you post that little bit of code that blocks after so many 404's?

[JuicyBunny]

I'm guessing the OP is referring to bots that constantly probe for vulnerabilities in commonly used scripts and plugins.

I run a custom written script, which like the htaccess examples above immediately blocks against known strings or URIs that should never appear on my site (eg /wp-login.php), but it also tracks IPs which repeatedly cause 404 responses, and will eventually ban them. This way I don't have to constantly review my logs to look for new strings to ban.

Genius! Care to share? For a small paypal donation even?

[rowan]

I'd like to help, but it's pretty tightly integrated into my website... it would need work even to use on another of mine, let alone a totally separate setup. It also relies on the OS being FreeBSD to firewall really obnoxious IPs.

Basically I have a PHP script handle 404s. If it's a permitted 404 (eg a browser fetching favicon.ico) then it's ignored and returns a standard Apache 404 response, exactly like the server normally would. Otherwise, the count of 404s that IP has incurred is incremented. If that count goes too high, a ban for that IP is triggered.

The actual implementation is a bit more complicated because it includes logging, DNS resolution, and drop firewalling, but that's generally how it works.

[JuicyBunny]

I'd like to help, but it's pretty tightly integrated into my website... it would need work even to use on another of mine, let alone a totally separate setup. It also relies on the OS being FreeBSD to firewall really obnoxious IPs.

Basically I have a PHP script handle 404s. If it's a permitted 404 (eg a browser fetching favicon.ico) then it's ignored and returns a standard Apache 404 response, exactly like the server normally would. Otherwise, the count of 404s that IP has incurred is incremented. If that count goes too high, a ban for that IP is triggered.

The actual implementation is a bit more complicated because it includes logging, DNS resolution, and drop firewalling, but that's generally how it works.

IN this day and age of weaponized bots you could probably make a fortune if you made a script/s for general use.

[ghjghj]

https://www.modsecurity.org/