SABAI

how to prevent people from using WebCopier and Download Ninja kind of programs on my sites?

__________________

pr0

Not if they morph their referrers, but you can block them pretty easy if it sends something like "Ninja" in the referrer.

LBBV

Something like the following in a .htaccess works well...

SetEnvIfNoCase User-Agent "^Wget" bad_bot
SetEnvIfNoCase User-Agent "^ia_archiver" bad_bot
SetEnvIfNoCase User-Agent "^WebReaper" bad_bot
SetEnvIfNoCase User-Agent "^BloodHound" bad_bot
SetEnvIfNoCase User-Agent "^WebVac" bad_bot
SetEnvIfNoCase User-Agent "^w3mir" bad_bot
SetEnvIfNoCase User-Agent "^Xget" bad_bot
SetEnvIfNoCase User-Agent "^Download Express" bad_bot
SetEnvIfNoCase User-Agent "^NetAnt" bad_bot
SetEnvIfNoCase User-Agent "^FlashGet" bad_bot
SetEnvIfNoCase User-Agent "^GetRight" bad_bot

Deny from env=bad_bot

Just add any bots you wish...

__________________
National Net Hosting. Handled.
See our new line of inexpensive Dedicated Servers | Cabinets starting at $795 | CDN
[email protected] | 888-4-NATNET | www.nationalnet.com

Alky

heres a good list...

RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} ^Mozilla
RewriteRule /* - [L]

RewriteCond %{HTTP_USER_AGENT} ^Teleport [OR]
RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^ia_archiver [OR]
RewriteCond %{HTTP_USER_AGENT} ^Aculinx [OR]
RewriteCond %{HTTP_USER_AGENT} ^Ants [OR]
RewriteCond %{HTTP_USER_AGENT} ^attach [OR]
RewriteCond %{HTTP_USER_AGENT} ^Backstreet [OR]
RewriteCond %{HTTP_USER_AGENT} ^BackWeb [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bandit [OR]
RewriteCond %{HTTP_USER_AGENT} ^Buddy [OR]
RewriteCond %{HTTP_USER_AGENT} ^Burner [OR]
RewriteCond %{HTTP_USER_AGENT} ^Caitoo [OR]
RewriteCond %{HTTP_USER_AGENT} ^Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Commander [OR]
RewriteCond %{HTTP_USER_AGENT} ^Copier [OR]
RewriteCond %{HTTP_USER_AGENT} ^Crawler [OR]
RewriteCond %{HTTP_USER_AGENT} ^Curl [OR]
RewriteCond %{HTTP_USER_AGENT} ^DA\ .\.0 [OR]
RewriteCond %{HTTP_USER_AGENT} ^Devil [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo\ Pump [OR]
RewriteCond %{HTTP_USER_AGENT} ^Down2Web [OR]
RewriteCond %{HTTP_USER_AGENT} ^Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^DownloadIt [OR]
RewriteCond %{HTTP_USER_AGENT} ^Drip [OR]
RewriteCond %{HTTP_USER_AGENT} ^Eater [OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
RewriteCond %{HTTP_USER_AGENT} ^Eclipt [OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Enterprise [OR]
RewriteCond %{HTTP_USER_AGENT} ^Express [OR]
RewriteCond %{HTTP_USER_AGENT} ^Extractor [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
RewriteCond %{HTTP_USER_AGENT} ^FairAd\ Client [OR]
RewriteCond %{HTTP_USER_AGENT} ^FileHound [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlipBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetSmart [OR]
RewriteCond %{HTTP_USER_AGENT} ^Gets [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetWeb [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
RewriteCond %{HTTP_USER_AGENT} ^gotit [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grab [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
RewriteCond %{HTTP_USER_AGENT} ^Greed [OR]
RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
RewriteCond %{HTTP_USER_AGENT} ^Hoover [OR]
RewriteCond %{HTTP_USER_AGENT} ^HtGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^HTTrack [OR]
RewriteCond %{HTTP_USER_AGENT} ^IBrowse [OR]
RewriteCond %{HTTP_USER_AGENT} ^iFox98 [OR]
RewriteCond %{HTTP_USER_AGENT} ^IGJpg$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
RewriteCond %{HTTP_USER_AGENT} ^InternetLinkAgent [OR]
RewriteCond %{HTTP_USER_AGENT} ^IPhoto [OR]
RewriteCond %{HTTP_USER_AGENT} ^iwantmy [OR]
RewriteCond %{HTTP_USER_AGENT} ^iwantmyxxx.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC [OR]
RewriteCond %{HTTP_USER_AGENT} ^JustView [OR]
RewriteCond %{HTTP_USER_AGENT} ^Keepoint [OR]
RewriteCond %{HTTP_USER_AGENT} ^leech [OR]
RewriteCond %{HTTP_USER_AGENT} ^lftp [OR]
RewriteCond %{HTTP_USER_AGENT} ^likse [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Magnet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mag-Net [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^Memo [OR]
RewriteCond %{HTTP_USER_AGENT} ^MemoWeb [OR]
RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mirror [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]
RewriteCond %{HTTP_USER_AGENT} ^MSProxy [OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetDrag [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline [OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} ^PerManSurfer [OR]
RewriteCond %{HTTP_USER_AGENT} ^PlantyNet_WebRo [OR]
RewriteCond %{HTTP_USER_AGENT} ^Pockey [OR]
RewriteCond %{HTTP_USER_AGENT} ^Pump [OR]
RewriteCond %{HTTP_USER_AGENT} ^Reaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Recorder [OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Retriever [OR]
RewriteCond %{HTTP_USER_AGENT} ^SilentSurf [OR]
RewriteCond %{HTTP_USER_AGENT} ^Siphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^Snagger [OR]
RewriteCond %{HTTP_USER_AGENT} ^Snake [OR]
RewriteCond %{HTTP_USER_AGENT} ^Snarf [OR]
RewriteCond %{HTTP_USER_AGENT} ^Snatcher [OR]
RewriteCond %{HTTP_USER_AGENT} ^SpaceBison [OR]
RewriteCond %{HTTP_USER_AGENT} ^Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
RewriteCond %{HTTP_USER_AGENT} ^Vacuum [OR]
RewriteCond %{HTTP_USER_AGENT} ^Vampire [OR]
RewriteCond %{HTTP_USER_AGENT} ^wantmy [OR]
RewriteCond %{HTTP_USER_AGENT} ^Weazel [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
RewriteCond %{HTTP_USER_AGENT} ^Webexe [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [OR]
RewriteCond %{HTTP_USER_AGENT} ^Webster [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebVCR [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^WFARC [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^Whacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
RewriteCond %{HTTP_USER_AGENT} ^WWWCopy [OR]
RewriteCond %{HTTP_USER_AGENT} ^WWWoffle
RewriteRule .\.([gG][iI][fF]|[jJ][pP][gG])$ - [F]

alex79

What is WebCopier and Download Ninja??

Jimbo

softwares that "leech" the content of your site

__________________
free sex videos

TFCash

Sorry to tell you this guys, but those programs all can be made to look like IE But the good news is that all you have to do is to switch your links over to forms and it kills all those site suckers


Tim

andi_germany

Simple solution for most guys using the offline browsers. Create a cgi that checks the referrer and then forward them to the link. Its some work to replace the links but it works because they can fake the referrer but they mostly cannot execute a cgi.
Note that this can fuck with members using norton and other referrer blocking tools. Those need to turn off for browsing.

SABAI

thanks a lot guys this will be very usefull

__________________

gornyhuy

I don't know, most of these applications now come with a list of client types that you can choose from right in the most basic config. Some are even sophisticated enough to follow javascript links and submit form data. Trying to block these is only going to stop stupidest users.

A better (but much more complicated) solution is to check the rate of request from every user and when they exceed your limit, redirect 'em, or even suspend the account. That should shut down everyone except a real surfer.

Good example is GFY search function, they have it limited so you can only do one search every 20 seconds, otherwise it makes you wait.

__________________

icq:159548293

TFCash

It's as simple as changing this

to this


The web suckers can't read, and won't follow the forms on your page!

Tim