Fellow coders I need some advice - GoFuckYourself.com

ClickBuster · 12-29-2007, 08:18 AM

I want to make a script that restricts the member area bandwidth usage per member per hour. My question is how do I check the used bandwidth without having to pass all content through scripts. So far the only whatso usable solution to this problem is using iptables for bandwidth info per IP. If anybody can give me any advice, it'd be greatly appreciated.

Thanks in advance,
Andrew

P.S.
Hope you all have a Happy New Year

pip · 12-29-2007, 08:34 AM

Think ftvcash does this how much you can allow per day, stops people from downloading everything

ClickBuster · 12-29-2007, 09:31 AM

Many sites have that actually

sortie · 12-29-2007, 09:45 AM

Build a table with the content size of each piece of media.

Use a redirect script (like traffic trading: /cgi-bin/media.cgi?url=/themedia.mpg) to print "location: /themedia.mpg\n\n"

The redirect script reads the content size table based on the url and adds to a cookie and IP file the total amount of bandwidth and redirects to an error page if bandwidth exceeded.

There's several other ways but this is probably the easiest.

Using htaccess to hide the redirect script can help make it harder to defeat.

FightThisPatent · 12-29-2007, 09:52 AM

Quote:

Originally Posted by sortie

Build a table with the content size of each piece of media.

Use a redirect script (like traffic trading: /cgi-bin/media.cgi?url=/themedia.mpg) to print "location: /themedia.mpg\n\n"

The redirect script reads the content size table based on the url and adds to a cookie and IP file the total amount of bandwidth and redirects to an error page if bandwidth exceeded.

There's several other ways but this is probably the easiest.

Using htaccess to hide the redirect script can help make it harder to defeat.

good idea, but this assumes that the person viewed the entire file... so if the mpg file is 400meg, and they got their satisfaction around 100meg point, then they got "over charged" (assuming content was streaming vs. downlaoded as a whole clip).

you can have a background cron job that runs every minute that does b/w stat checks by IP (as you were thinking) to fill in a table, and your PHP script checks that table based on IP. (this method would help with streamed content)

Most ppl have the same IP during their session with you... AOL users can have IP change in the middle of their time on your site...

Fight the .02!

MMarko · 12-29-2007, 09:54 AM

afaik, there are some robust production grade scripts available for this... I think guy who made strongbox has script for this too (it's actualy an apache module)

FightThisPatent · 12-29-2007, 10:22 AM

Quote:

Originally Posted by MMarko

afaik, there are some robust production grade scripts available for this... I think guy who made strongbox has script for this too (it's actualy an apache module)

ah, yes, apache modules... found this:
http://digg.com/linux_unix/Manage_Ap...th_ mod_cband

Fight the 411!

ClickBuster · 12-29-2007, 10:35 AM

Quote:

Originally Posted by FightThisPatent

ah, yes, apache modules... found this:
http://digg.com/linux_unix/Manage_Ap...th_ mod_cband

Fight the 411!

Thats not it

To be honest, so far the iptables version seems to be the smartest solution, but I still have too hook with a cron job to count the used bandwidth...

sortie · 12-29-2007, 10:48 AM

Quote:

Originally Posted by ClickBuster

Thats not it

To be honest, so far the iptables version seems to be the smartest solution, but I still have too hook with a cron job to count the used bandwidth...

Cron? We don't need no stinking cron!!

$n="dontstop";
while ($n eq "dontstop") {
&manage_bandwidth;
sleep(60);
if (-e "killme.txt") {exit;}
}

ultimatebbwdotcom · 12-29-2007, 10:52 AM

Doesnt Throttlebox do this?

ClickBuster · 12-29-2007, 12:31 PM

Quote:

Originally Posted by sortie

Cron? We don't need no stinking cron!!

$n="dontstop";
while ($n eq "dontstop") {
&manage_bandwidth;
sleep(60);
if (-e "killme.txt") {exit;}
}

:D 1337 but I need a bit more explanation to your prop...

Why · 12-29-2007, 12:31 PM

there are numerous apache modules that do this.

Why · 12-29-2007, 12:36 PM

Quote:

Originally Posted by ClickBuster

:D 1337 but I need a bit more explanation to your prop...

its a script that runs an endless loop every 60 seconds because $n never changes it just keeps looping with a 60 second sleep(pause) in the middle. the second to last line checks for the existence of a file, which could be used to turn it on or off. make sense?

ClickBuster · 12-29-2007, 12:40 PM

Quote:

Originally Posted by Why

its a script that runs an endless loop every 60 seconds because $n never changes it just keeps looping with a 60 second sleep(pause) in the middle. the second to last line checks for the existence of a file, which could be used to turn it on or off. make sense?

What if Apache timesout? Also, you can perform only JS redirects after the output. Any specific Apache module suggestions?

sortie · 12-29-2007, 01:43 PM

Quote:

Originally Posted by Why

its a script that runs an endless loop every 60 seconds because $n never changes it just keeps looping with a 60 second sleep(pause) in the middle. the second to last line checks for the existence of a file, which could be used to turn it on or off. make sense?

Exactamundo!

The "&manage_bandwidth" is a subroutine to do whatever you want.

If you have dedicated server then this is easy way to do some things but maybe some host is paranoid on shared hosting and don't like it.

It's just a choice other than cron; not better in anyway.

rowan · 12-29-2007, 01:46 PM

Apache logs show both (a) the authenticated member name and (b) the size of the object it has delivered. It shouldn't be hard to write a script that continuously "tails" the logs and keeps stats on how much each member has downloaded. You could then set up something to modify/rewrite htaccess to redirect any further requests from that member to a page explaining that they've exceeded their quota.

Are you actually having a problem with excessive downloads or are you just being careful? Sometimes pissing off members may cost you more than savings in bw.

sortie · 12-29-2007, 01:55 PM

NOTE: to my last post.

The "killme.txt" file can only turn it off!
I has to be executed again to restart.

k0nr4d · 12-29-2007, 04:14 PM

cron is still better to use then because if you reboot the box it will still do what needs to be done.

as for the issue at hand, I can't think of any way you could limit this outside of an apache module.

ClickBuster · 12-29-2007, 07:34 PM

Quote:

Originally Posted by rowan

Apache logs show both (a) the authenticated member name and (b) the size of the object it has delivered. It shouldn't be hard to write a script that continuously "tails" the logs and keeps stats on how much each member has downloaded. You could then set up something to modify/rewrite htaccess to redirect any further requests from that member to a page explaining that they've exceeded their quota.

Are you actually having a problem with excessive downloads or are you just being careful? Sometimes pissing off members may cost you more than savings in bw.

I like this one the best

so you say apache logs... mhm I wonder if there's a separate log for authenticated users or if Apache can be modified in anyway to log this data separately? I need this coded for various monitoring reasons, including extensive account usage, account sharing, etc.

rowan · 12-29-2007, 07:40 PM

Quote:

Originally Posted by ClickBuster

I like this one the best

so you say apache logs... mhm I wonder if there's a separate log for authenticated users or if Apache can be modified in anyway to log this data separately? I need this coded for various monitoring reasons, including extensive account usage, account sharing, etc.

Yep, look up the LogFormat directive to define a custom log format, then the CustomLog directive (inside <VirtualHost...> )to write it out to a separate file. I'm not 100% sure you can write to more than one log per vhost but I'd guess it's likely.

raymor · 12-29-2007, 09:48 PM

Quote:

Originally Posted by MMarko

afaik, there are some robust production grade scripts available for this... I think guy who made strongbox has script for this too (it's actualy an apache module)

Quote:

Originally Posted by ultimatebbwdotcom

Doesnt Throttlebox do this?

Yep, Throttlebox, from the makers of Strongbox, does this, but does it in an
ITELLIGENT way. Just counting bandwidth without more intelligence than that
seems to cause more problems than it solves.

I wouldn't want to redirect every hit, or even count everything through iptables,
the overhead seems to high to me. Instead we track bandwidth only when it
actually matters, and we do it using a custom Apache module. This is something
we worked on for a few years, so you may find that it makes much more
sense to take advantage of all of the time that we spent on this, all of the
research, the discussions we had with university professors about declining
averages, etc. rather than either doing all of that work over again yourself
or skipping the hard part and building something that doesn't work so well.

ClickBuster · 12-30-2007, 04:31 AM

Quote:

Originally Posted by raymor

Yep, Throttlebox, from the makers of Strongbox, does this, but does it in an
ITELLIGENT way. Just counting bandwidth without more intelligence than that
seems to cause more problems than it solves.

I wouldn't want to redirect every hit, or even count everything through iptables,
the overhead seems to high to me. Instead we track bandwidth only when it
actually matters, and we do it using a custom Apache module. This is something
we worked on for a few years, so you may find that it makes much more
sense to take advantage of all of the time that we spent on this, all of the
research, the discussions we had with university professors about declining
averages, etc. rather than either doing all of that work over again yourself
or skipping the hard part and building something that doesn't work so well.

Not looking for a ready solution, but thanks for the input.

k0nr4d · 12-30-2007, 05:16 AM

Give mod_leech a shot,
http://toroid.org/ams/mod_leech

ClickBuster · 12-30-2007, 06:38 AM

Quote:

Originally Posted by k0nr4d

Give mod_leech a shot,
http://toroid.org/ams/mod_leech

I like this one and gonna try it for sure. The hahahahahaha story sucks too...

Edit: ROFL the shady hosting company name got changed to hahahahahaha

raymor · 12-30-2007, 12:04 PM

Quote:

Originally Posted by ClickBuster

Not looking for a ready solution, but thanks for the input.

Ok, good luck with it. Be forwarned to get something to actually work well and
not create problems is a bigger job than it first appears.

12-29-2007, 08:18 AM	#1
ClickBuster Confirmed User Join Date: May 2004 Posts: 210	Fellow coders I need some advice I want to make a script that restricts the member area bandwidth usage per member per hour. My question is how do I check the used bandwidth without having to pass all content through scripts. So far the only whatso usable solution to this problem is using iptables for bandwidth info per IP. If anybody can give me any advice, it'd be greatly appreciated. Thanks in advance, Andrew P.S. Hope you all have a Happy New Year __________________ -- ClickBuster -- ICQ# 263653704 -- Email: clickbuster1 [-at-] gmail [-dot-] com

12-29-2007, 09:31 AM	#3
ClickBuster Confirmed User Join Date: May 2004 Posts: 210	Many sites have that actually __________________ -- ClickBuster -- ICQ# 263653704 -- Email: clickbuster1 [-at-] gmail [-dot-] com

12-29-2007, 09:45 AM	#4
sortie Confirmed User Industry Role: Join Date: Mar 2007 Posts: 7,771	Build a table with the content size of each piece of media. Use a redirect script (like traffic trading: /cgi-bin/media.cgi?url=/themedia.mpg) to print "location: /themedia.mpg\n\n" The redirect script reads the content size table based on the url and adds to a cookie and IP file the total amount of bandwidth and redirects to an error page if bandwidth exceeded. There's several other ways but this is probably the easiest. Using htaccess to hide the redirect script can help make it harder to defeat. __________________

12-29-2007, 10:52 AM	#10
ultimatebbwdotcom Confirmed User Join Date: Mar 2006 Posts: 591	Doesnt Throttlebox do this? __________________ Ultimatebbw.com Dangerouscurvesdesign.com

12-29-2007, 01:55 PM	#17
sortie Confirmed User Industry Role: Join Date: Mar 2007 Posts: 7,771	NOTE: to my last post. The "killme.txt" file can only turn it off! I has to be executed again to restart. __________________

12-29-2007, 08:34 AM	#2
pip So Fucking Banned Join Date: Nov 2005 Location: Netherlands Posts: 468	Think ftvcash does this how much you can allow per day, stops people from downloading everything

12-29-2007, 09:54 AM	#6
MMarko Confirmed User Join Date: Jun 2007 Posts: 160	afaik, there are some robust production grade scripts available for this... I think guy who made strongbox has script for this too (it's actualy an apache module)

12-29-2007, 12:31 PM	#12
Why MFBA Industry Role: Join Date: Mar 2003 Location: PNW Posts: 7,230	there are numerous apache modules that do this.

12-29-2007, 01:46 PM	#16
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Apache logs show both (a) the authenticated member name and (b) the size of the object it has delivered. It shouldn't be hard to write a script that continuously "tails" the logs and keeps stats on how much each member has downloaded. You could then set up something to modify/rewrite htaccess to redirect any further requests from that member to a page explaining that they've exceeded their quota. Are you actually having a problem with excessive downloads or are you just being careful? Sometimes pissing off members may cost you more than savings in bw.

12-29-2007, 04:14 PM	#18
k0nr4d Confirmed User Industry Role: Join Date: Aug 2006 Location: Poland Posts: 9,231	cron is still better to use then because if you reboot the box it will still do what needs to be done. as for the issue at hand, I can't think of any way you could limit this outside of an apache module. __________________ Mechanical Bunny Media Mechbunny Tube Script \| Mechbunny Webcam Aggregator Script \| Custom Web Development

12-30-2007, 05:16 AM	#23
k0nr4d Confirmed User Industry Role: Join Date: Aug 2006 Location: Poland Posts: 9,231	Give mod_leech a shot, http://toroid.org/ams/mod_leech __________________ Mechanical Bunny Media Mechbunny Tube Script \| Mechbunny Webcam Aggregator Script \| Custom Web Development