GoFuckYourself.com - Adult Webmaster Forum

- Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)

- - Website downloading tools (https://gfy.com/showthread.php?t=258421)

Website downloading tools

I just caught a couple bastards ripping off a ton of my galleries .. chewing up a shit-load of bandwidth .. and I'm a little upset about this. But I guess it's to be expected.

I won't stand for this and sit back and pay the bandwidth bills.. if you're trying to rip off all the free porn on my server ... fuck off ... I won't let you do it.

So I'm doing something about it .. and I want to share it with GFY incase it's useful to anyone here.

The tool that these guys were using is called Website eXtractor. Lucikly in this program's case .. it sets the User-Agent: to "Website eXtractor (http://www.asona.org)"

Ima block this program on my server ;) To do this you need mod_access and mod_setenvif.

Crack open your apache configuration file and add the following line:

BrowserMatch "^Website eXtractor" downloader

Now wherever you configure your virtual hosts add the line "deny from env=downloader" so it looks something like this:

Code:

&lt;Directory "/foo/myhost.com"&gt;

      Order allow, deny

      allow from all

      deny from env=downloader

&lt;/Directory&gt;

I'm using the following page http://www.psychedelix.com/agents.html to search for other downloading tools that set "User-Agent" so I can block them too. If anyone knows any good ones / popular ones post 'em here. And I hope this was useful to some people here.

Quote:

Originally posted by garett

I won't stand for this and sit back and pay the bandwidth bills.. if you're trying to rip off all the free porn on my server ... fuck off ... I won't let you do it.
here.

Tell them that.

Another one of the big website grabbers is Wget, it eats a hell of a lot more bandwidth, but not as much as Fusker will if you dont have hotlink protection

Is that like BlackWidow or something?

M.

Going through the list of User-Agents I posted above..

I searched each page for "downloading tool" .. then I grep'd my access_log (which is rotated daily .. so it's only today's logs) for those User-Agents... I was fucking dumbfounded how many hits I was getting from downloading tools.

I am proudly sharing my blacklist with the world:

BrowserMatch "^Website eXtractor" downloader
BrowserMatch "^DDD" downloader
BrowserMatch "^GetBot" downloader
BrowserMatch "^Irvine" downloader
BrowserMatch "^JBH" downloader
BrowserMatch "^JOC" downloader
BrowserMatch "^dCSbot" downloader
BrowserMatch "^Download.exe" downloader
BrowserMatch "WebCapture" downloader
BrowserMatch "^Pockey" downloader
BrowserMatch "^puf/0.*" downloader
BrowserMatch "^Teleport Pro/*" downloader
BrowserMatch "^URLGetFile" downloader
BrowserMatch "^Web Image Collector" downloader
BrowserMatch "^WebAuto/*" downloader
BrowserMatch "^WebFetch" downloader
BrowserMatch "^WebPix" downloader
BrowserMatch "^Wget/*" downloader

Not all of those I was getting hits from .. but lots of them. Wget was by far the biggest offender. I don't even wanna know how much bandwidth these tools have actually used up this month.

Anyway I hope I at least provided a useful little tip for some people here on how to prevent these tools from hitting your site. Feel free to use my blacklist :)