Anti Mass Site downloader, Bots Scripts?

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Samantha_Luvcox
    Confirmed User
    • Sep 2002
    • 396

    #1

    Anti Mass Site downloader, Bots Scripts?

    What are the best (if there are any) scripts for Automaitically and manually stopping Site sucking and Bots?

    We currenlty run antihotlinking, Iprotect and proxypass for password protection, throttling and ip burst attacks but would like to complement them with another script that gives us a simple way to block domains by IP or url of certain sites, I know that I can block IP at the network level by adding the IP to the IPChains firewall rules, or by site in htaccess, but looking for a solution that monitors BOT activity and blocks them, or give us the option to while giving us a nice managment console I can train my people to look at and work with.

    In the past I have used botbuster for this, but have not been able to reach them my email. Anyone know of a nice solution to this?

    It would be great if someone could build a console that could consolidate all the information from these various scripts into one place... dream a little dream...


    If anyone could help with ideas that would be great.
    SIG TOO BIG! Maximum 120x60 button and no more than 3 text lines of DEFAULT SIZE and COLOR. Unless your sig is for a GFY top banner sponsor, then you may use a 624x80 instead of a 120x60.
  • johnbosh
    Confirmed User
    • Aug 2002
    • 8965

    #2
    there is a robot.txt for

    Comment

    • Steen2
      Confirmed User
      • Feb 2004
      • 7662

      #3
      I don't think you can block manual wGet.
      ICQ: 2262.73945

      Comment

      • kaktusan
        Confirmed User
        • Mar 2003
        • 334

        #4
        I heard from somewhere that the script below works, but i haven't tested it manually to be 100% sure. It is a .htaccess file
        Please test it and notify with the results...

        RewriteEngine On
        RewriteCond %{HTTP_USER_AGENT} ^attach [OR]
        RewriteCond %{HTTP_USER_AGENT} ^BackWeb [OR]
        RewriteCond %{HTTP_USER_AGENT} ^bandit [OR]
        RewriteCond %{HTTP_USER_AGENT} ^BatchFTP [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:[email protected] [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Buddy [OR]
        RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Collector [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Copier [OR]
        RewriteCond %{HTTP_USER_AGENT} ^DA [OR]
        RewriteCond %{HTTP_USER_AGENT} ^DISCo\ Pump [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Download\ Wonder [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Downloader [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Drip [OR]
        RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
        RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Extractorpro [OR]
        RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
        RewriteCond %{HTTP_USER_AGENT} ^FileHound [OR]
        RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
        RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
        RewriteCond %{HTTP_USER_AGENT} ^GetSmart [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
        RewriteCond %{HTTP_USER_AGENT} ^gotit [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Grabber [OR]
        RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
        RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
        RewriteCond %{HTTP_USER_AGENT} ^HTTrack [OR]
        RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Iria [OR]
        RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
        RewriteCond %{HTTP_USER_AGENT} ^JOC [OR]
        RewriteCond %{HTTP_USER_AGENT} ^JustView [OR]
        RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]
        RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
        RewriteCond %{HTTP_USER_AGENT} ^lftp [OR]
        RewriteCond %{HTTP_USER_AGENT} ^likse [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Magnet [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Mag-Net [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Memo [OR]
        RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Mirror [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
        RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
        RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
        RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]
        RewriteCond %{HTTP_USER_AGENT} ^NetZip [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Ninja [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]
        RewriteCond %{HTTP_USER_AGENT} ^offline\ explorer [OR]
        RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]
        RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Pockey [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Pump [OR]
        RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Reaper [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Recorder [OR]
        RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Siphon [OR]
        RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
        RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Snake [OR]
        RewriteCond %{HTTP_USER_AGENT} ^SpaceBison [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Stripper [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Sucker [OR]
        RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
        RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
        RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
        RewriteCond %{HTTP_USER_AGENT} ^teleport\ pro [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Vacuum [OR]
        RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]
        RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]
        RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
        RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]
        RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
        RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Website [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Webster [OR]
        RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
        RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]
        RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Whacker [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
        RewriteCond %{HTTP_USER_AGENT} ^Xaldon
        RewriteRule /*$ http://www.yourpaysitedomain.com/ [L,R]


        Superior Blogging Scripts | Porn Blog Space - not just another free blog hosting!

        Comment

        • Samantha_Luvcox
          Confirmed User
          • Sep 2002
          • 396

          #5
          bump for the late crowd, other ideas?
          SIG TOO BIG! Maximum 120x60 button and no more than 3 text lines of DEFAULT SIZE and COLOR. Unless your sig is for a GFY top banner sponsor, then you may use a 624x80 instead of a 120x60.

          Comment

          • Project-Shadow
            Confirmed User
            • Feb 2003
            • 7340

            #6
            .htaccess is the best way to go about it. But if someone manually starts up the Download manager program they spoof the refferer and you can't block it.

            Only other way I can think of is to generate a cookie that is required for all downloads.. or enter a code when entering the site to identify the user as a user.. as apposed to a bot.

            Comment

            • Apollo
              Confirmed User
              • Sep 2002
              • 433

              #7
              Strongbox alledgly has a way to stop mass downloading and site suckers (assuming it's behind a protected area).

              I've never looked too much into it but if you really need a solution...might be something for you.

              http://www.bettercgi.com/strongbox/

              -----------
              Free automated way to rotate sponsor gallery thumbnails on your sites
              TgpWizards - Free TGP Gallery submission system

              Comment

              • Samantha_Luvcox
                Confirmed User
                • Sep 2002
                • 396

                #8
                Ok one last bump, I'm looking into the ideas, but wanted to see if the morning crowd had any.
                SIG TOO BIG! Maximum 120x60 button and no more than 3 text lines of DEFAULT SIZE and COLOR. Unless your sig is for a GFY top banner sponsor, then you may use a 624x80 instead of a 120x60.

                Comment

                Working...