Website downloading tools

Collapse
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • garett
    Confirmed User
    • Mar 2004
    • 683
    #1

    Website downloading tools

    I just caught a couple bastards ripping off a ton of my galleries .. chewing up a shit-load of bandwidth .. and I'm a little upset about this. But I guess it's to be expected.

    I won't stand for this and sit back and pay the bandwidth bills.. if you're trying to rip off all the free porn on my server ... fuck off ... I won't let you do it.

    So I'm doing something about it .. and I want to share it with GFY incase it's useful to anyone here.

    The tool that these guys were using is called Website eXtractor. Lucikly in this program's case .. it sets the User-Agent: to "Website eXtractor (http://www.asona.org)"

    Ima block this program on my server ;) To do this you need mod_access and mod_setenvif.

    Crack open your apache configuration file and add the following line:

    BrowserMatch "^Website eXtractor" downloader

    Now wherever you configure your virtual hosts add the line "deny from env=downloader" so it looks something like this:

    Code:
    <Directory "/foo/myhost.com">
      Order allow, deny
      allow from all
      deny from env=downloader
</Directory>
    I'm using the following page http://www.psychedelix.com/agents.html to search for other downloading tools that set "User-Agent" so I can block them too. If anyone knows any good ones / popular ones post 'em here. And I hope this was useful to some people here.
    Tags: None
    • Manowar
      jellyfish  
      • Dec 2003
      • 71528
      #2
      Originally posted by garett


      I won't stand for this and sit back and pay the bandwidth bills.. if you're trying to rip off all the free porn on my server ... fuck off ... I won't let you do it.
      here.
      Tell them that.

      Another one of the big website grabbers is Wget, it eats a hell of a lot more bandwidth, but not as much as Fusker will if you dont have hotlink protection

        Comment

        • maRRtin
          Registered User
          • Nov 2003
          • 82
          #3
          Is that like BlackWidow or something?

          M.
          If your not sending your interracial traffic to
          DARKCAVERN.com YOU ARE LOOSING MONEY!

            Comment

            • garett
              Confirmed User
              • Mar 2004
              • 683
              #4
              Going through the list of User-Agents I posted above..

              I searched each page for "downloading tool" .. then I grep'd my access_log (which is rotated daily .. so it's only today's logs) for those User-Agents... I was fucking dumbfounded how many hits I was getting from downloading tools.

              I am proudly sharing my blacklist with the world:

              BrowserMatch "^Website eXtractor" downloader
              BrowserMatch "^DDD" downloader
              BrowserMatch "^GetBot" downloader
              BrowserMatch "^Irvine" downloader
              BrowserMatch "^JBH" downloader
              BrowserMatch "^JOC" downloader
              BrowserMatch "^dCSbot" downloader
              BrowserMatch "^Download.exe" downloader
              BrowserMatch "WebCapture" downloader
              BrowserMatch "^Pockey" downloader
              BrowserMatch "^puf/0.*" downloader
              BrowserMatch "^Teleport Pro/*" downloader
              BrowserMatch "^URLGetFile" downloader
              BrowserMatch "^Web Image Collector" downloader
              BrowserMatch "^WebAuto/*" downloader
              BrowserMatch "^WebFetch" downloader
              BrowserMatch "^WebPix" downloader
              BrowserMatch "^Wget/*" downloader

              Not all of those I was getting hits from .. but lots of them. Wget was by far the biggest offender. I don't even wanna know how much bandwidth these tools have actually used up this month.

              Anyway I hope I at least provided a useful little tip for some people here on how to prevent these tools from hitting your site. Feel free to use my blacklist

                Comment

                Previous template Next
                Working...