GoFuckYourself.com - Adult Webmaster Forum - View Single Post

raymor · 02-29-2004, 09:51 AM

Script like PennyWize claim to limit what a user downloads,
but that's not exactly what they actually do.
Instead, all of the ones that I've seen limit the sum of the file
sizes for which they issue a request. What's the difference?
The difference is that if a user takes a look at the first 15 seconds
of a video, decides he doesn't want that one, and takes a peek
at another one, pennywize et al count it as though the user got
the whole video. This is a great way to piss off users.
"Sorry, bud, you can only peek at 5 videos a day because they're each 100 MB."

Mod_throttle can actually limit what they download,
but it's designed more to control how much they download
per minute than per day. On a busy site using it to track everyone's
usage for the last 24 hours may or may not work too well.

Normally when people ask this question what they are really
wanting to do is block slurpers (rippers).
That's a whole different question. The way to block
slurpers is NOT by limiting bandwidth, for several reasons.
Without going iinto too much detail, you can see that with the
pennywize style limiting you will end up blocking the legitimate
customer who peeks at a few videos while the slurper
is able to download several thousand images before he gets blocked.
Looking at user_agent isn't the way to go either.
That just blocks people who TELL you that they are slurping.
Especialy, having a list of disallowed user agents is
the wrong way to go. For why that's true, read chapter one
of any decent book on security.

If you want to block automated site spidering programs
(known as rippers or slurpers), they way to do that is by blocking
based on the fact that they are 1) automated and 2) spiders.
Block them immediately when they start spidering your site,
not after they download 2,000 images.
That's what Strongbox does - block them if they are ripping
your site, without any other BS.
Well, that and also protect from brute force attacks,
password sites, and all kinds of other nasties.

PS - Someone mentioned that on a 3 day trial if you block them
after 2GB they won't convert. News flash - if they bought a
trial membership and immediately began using a ripper to download
the whole site, they weren't going to convert anyway.
They don't need too - your site is on their hard drive.

02-29-2004, 09:51 AM
raymor Confirmed User Join Date: Oct 2002 Posts: 3,745	Script like PennyWize claim to limit what a user downloads, but that's not exactly what they actually do. Instead, all of the ones that I've seen limit the sum of the file sizes for which they issue a request. What's the difference? The difference is that if a user takes a look at the first 15 seconds of a video, decides he doesn't want that one, and takes a peek at another one, pennywize et al count it as though the user got the whole video. This is a great way to piss off users. "Sorry, bud, you can only peek at 5 videos a day because they're each 100 MB." Mod_throttle can actually limit what they download, but it's designed more to control how much they download per minute than per day. On a busy site using it to track everyone's usage for the last 24 hours may or may not work too well. Normally when people ask this question what they are really wanting to do is block slurpers (rippers). That's a whole different question. The way to block slurpers is NOT by limiting bandwidth, for several reasons. Without going iinto too much detail, you can see that with the pennywize style limiting you will end up blocking the legitimate customer who peeks at a few videos while the slurper is able to download several thousand images before he gets blocked. Looking at user_agent isn't the way to go either. That just blocks people who TELL you that they are slurping. Especialy, having a list of disallowed user agents is the wrong way to go. For why that's true, read chapter one of any decent book on security. If you want to block automated site spidering programs (known as rippers or slurpers), they way to do that is by blocking based on the fact that they are 1) automated and 2) spiders. Block them immediately when they start spidering your site, not after they download 2,000 images. That's what Strongbox does - block them if they are ripping your site, without any other BS. Well, that and also protect from brute force attacks, password sites, and all kinds of other nasties. PS - Someone mentioned that on a 3 day trial if you block them after 2GB they won't convert. News flash - if they bought a trial membership and immediately began using a ripper to download the whole site, they weren't going to convert anyway. They don't need too - your site is on their hard drive. __________________ For historical display only. This information is not current: support@bettercgi.com ICQ 7208627 Strongbox - The next generation in site security Throttlebox - The next generation in bandwidth control Clonebox - Backup and disaster recovery on steroids Last edited by raymor; 02-29-2004 at 09:57 AM..