stolen passes - server security

[chicktrainer]

Hi Guys,

I've been in the biz for over two years now running a rather popular amateur site. The site's on a dedicated UNIX machine at Hostcentric. Problem is that more and more passwords seem to get stolen.
Last night the amount of blocked accounts (we're using Pennywize 3.0) has DOUBLED. That's an all time record and it worries me - a lot. I'd say about 10% of all usernames had been compromised in the last 24 hours.

Also right after discovering the stolen password "all-time-record" I notice Penywize had shut down.
Don't get me wrong, I love Pennywize and it works great, but to me it looked like someone got tired of stealing new passes and decided to simply shut down the "protection". Am I being paranoid? I don't think so.

So the question is, what can I do server-wise to make it harder on whoever's messing up my site? I'm not a tech-head and I don't have the faintest idea where to start. Do I need a Firewall? Should I hire a security expert to check out the server (if yes, where do I find one?) or should I simply contact Hostcentric (do they even offer "security" counseling)?

BTW billing/usernames are done via CCBill/IBill (in case it matters).

I'd really appreciate your help,
Tom

[Jizar II]

Normally they "guess" the passes by bruteforcing, but as far as I know Penny V3. should stop the brute forcing hackers, so thats pretty weird!.. I dont know how large your site site is but 10% of all your passes cracked on a 2 year old site seems very extreme!..

[extreme]

Some basic securityadvice:

- Shut down all services you dont need, especially thoose that listen to ports.

You can list services "listening for clients" by typing "netsat -a | grep LISTEN".

If you run redhat linux you could for example stop portmap by "/etc/init.d/portmap stop". Works pretty much the same for the other services.

Unixboxes often come with a shitload of services running that you dont need. For example I only got http,sendmail and ssh running on my box (MySQL too, but that only accepts connections from localhost).

- Keep your machine up to date with securitypatches. If you got a redhat box you can simply upgrade your system by typing "up2date -u" as root.

If you have php stick with the latest version, php has a very bad securityhistory :\.

You could also remove most of the local suid (+s) root binaries . Theese are often used by hackers and scriptkiddies to hack root once they got local access to the box.

type "find / -perm -4000 -type f -user root 2> /dev/null | xargs ls -l" to list them. If you dont have any local users that needs to do things you can often remove alot of thoose +s flags.

For example you could do "chmod -s /bin/ping". Now only root can use the pingbinary but on the other hand it cant be used to hack root if a securityvuln is discovered in it. Let sendmail,ssh, passwd and crontab (if you use it) be...


Firewall could tighten things up, for example could you restrict sshaccess (I take it that your Not running telnet?) to certain ips (your and your users ips) ... then you'll need to learn the ipchains or iptables program though (I assume you got a linuxbox).

Hm.. security is a whole science, but that's some basic advice, feel free to ask specific questions ...

good luck ;)

[goBigtime]

Originally posted by chicktrainer
Hi Guys,

I've been in the biz for over two years now running a rather popular amateur site. The site's on a dedicated UNIX machine at Hostcentric. Problem is that more and more passwords seem to get stolen.
Last night the amount of blocked accounts (we're using Pennywize 3.0) has DOUBLED. That's an all time record and it worries me - a lot. I'd say about 10% of all usernames had been compromised in the last 24 hours.

Also right after discovering the stolen password "all-time-record" I notice Penywize had shut down.
Don't get me wrong, I love Pennywize and it works great, but to me it looked like someone got tired of stealing new passes and decided to simply shut down the "protection". Am I being paranoid? I don't think so.

So the question is, what can I do server-wise to make it harder on whoever's messing up my site? I'm not a tech-head and I don't have the faintest idea where to start. Do I need a Firewall? Should I hire a security expert to check out the server (if yes, where do I find one?) or should I simply contact Hostcentric (do they even offer "security" counseling)?

BTW billing/usernames are done via CCBill/IBill (in case it matters).

I'd really appreciate your help,
Tom

Hey Tom.. you've got mail.


For a cheap bruteforce solution try MemLogin from http://www.paysitepowertools.com



Is your box Linux or FreeBSD? Do you have shell access? If you do.. type uname -a and paste it here..

better yet.......

Yeah your on linux. Your server could stand some updating though. Is it a dedicated or colocated box from hostcentric? Are they supposed to update software for you as part of your deal?

[goBigtime]

Originally posted by extreme
Some basic securityadvice:


- Keep your machine up to date with securitypatches. If you got a redhat box you can simply upgrade your system by typing "up2date -u" as root.

Extreme -- up2date -u? Whats this? (I'm a FreeBSD guy personally) Does it like check to see which installed rpms are out of date or something? Will it update the kernel and rebuild the OS and all that? Like the equiv of a buildworld in freebsd if your familiar with that.

hmm off to read about this up2date. Sounds cool.

[extreme]

Originally posted by goBigtime


Extreme -- up2date -u? Whats this? (I'm a FreeBSD guy personally) Does it like check to see which installed rpms are out of date or something? Will it update the kernel and rebuild the OS and all that? Like the equiv of a buildworld in freebsd if your familiar with that.

hmm off to read about this up2date. Sounds cool.

ye, it leeches and installs patched/updated versions of installed rpms, including the kernel. I usually run it after new redhat security advisories on bugtraq.

Im not familiar with freebsds equiv.

[LBBV]

We've also seen some very strange inconsistent behaviour with the 3.0 version of Pennywise (works great one day, then the next day, blocks everything, then the next day blocks nothing...things like that). I would not recommend running that version until they have added some bug fixes...

[opflix]

all this pennywize shit... get trackman & forget about. ;) url? http://www.siteextreme.com/ (its not my shit but if you want a solution, there it is).

[OneHungLo]

Let me tell you this...If your using ccbill you are fucked. Their scripts are wide open. I got friendly with this password cracker and has key generators for literally 1000's of ccbill sites so basically he can write paswords to your password file at free will. I contacted CCbill with this info and they were clueless, they were asking me to ask this 16 yr old how to fix it even after i sent them the whole convo WITH the script the kid made to be able to do this...what a joke. Another sad thing is that i HIGHLY doubt they contacted everyone about this.

[OneHungLo]

One more thing...pennywize, password cop, ect. are useless against this.

[AcidMax]

Originally posted by opflix
all this pennywize shit... get trackman & forget about. ;) url? http://www.siteextreme.com/ (its not my shit but if you want a solution, there it is).

Thanks for the good words man..we don't have Trackman listed on the website at the present time, but we can always chat if you are interested.

Andy

[andi_germany]

check out iprotect . It is compiled into apache and therefore not slowing your system.

[ChrisB]

If you are serious about password protection and server security you could do worse than talk to us - we are about to launch our Flex-eSentry access and content protection system - it's beta testing at the moment and is very successful so far. We also offer security checks for webservers etc.

ChrisB
www.flex-esystems.co.uk
[email protected]

[JohnnyUtah]

We run Iprotect.
Very stable. Doesnt slow your box down.
So far, it allways worked!

[pennywize_v3]

Hi Everyone, Steve from Pennywize here

We have been having a few teething troubles lately - we have done a big server move and i had hoped it to go a little more seemlessly than it has, but there were a few minor hiccups.

Anyway it should be smooth sailing now as everything is rocking along. Version 3 is going great and much of the userbase has been upgraded. We have noticed a dramatic decrease in the number of brute force attacks to clients, as V3 has some very proactive checking and blocking in it.

I dont want to say too much about the CCBill issue (because I have alot of respect for them), but it is something we have come across too.

Anyway if anyone needs to contact me directly regarding anything, it's steve at pennywize dot com


Steve

[PxG]

If you're going to upgrade your password protection software, you might as well go for the best.

<a href="http://www.proxypass.com" target="_proxy">Proxypass</a> is currently the best and most stable password protection program on the net. We are currently in negotiations with one of the top three adult merchant service companies and hosting outfits to integrate the ProxyPass into their packages after an impressive demonstration.

Please check out <a href="http://www.proxypass.com/modules.php?op=modload&name=News&file=article&sid= 3&mode=thread&order=0&thold=0" target="_new">this beta-test</a> that CyberHeat did when we first launched, we blow away both PennyWize and Iprotect hands down.

The reason? We are the only currently available brute force defense and password protection software that instantly blocks and denies accesses from open proxies, the culprits to blame in the brute force process.

But don't take our word for it, check out these previous posts where independent users have touted the ProxyPass software as the new king of password protection:



http://www.gofuckyourself.com/showth...ight=proxypass


http://www.gofuckyourself.com/showth...ight=proxypass

these are just a couple, just search for proxypass here on gfy.

If you are having a serious brute force issue, Pennywize will eventually crap out b/c it is a CGI that takes up too much overhead. We use an extremely low level C module of Apache that severes connections to open proxies instantly (and no, it does not block AOL proxies, and no it does not only check for common port open proxies). Of course, it does everything else like password sharing blocks, username burst blocking, and it even has support for propagating blocks to load balanced server clusters. Try it out, your money back if you're not satisfied.

[pennywize_v3]

Mate

Pennywize will not 'crap out' because of load. Some of the biggest websites in the world run Pennywize and it runs perfectly. It is only one persistant process which run for the lifetime of apache, not some compiled in module which will take down apache entirely if it crashes!

It's funny how your product (and every other new product out there) are pretty much complete rips of pennywize. Pennywize has been around since 1997, and the only other guy around was password cop. Since then, guys like you have ripped pennywize in every way.. sure it's written in C, but it does the same thing and you even stolen and used our mod_rewrite stuff VERBATIM!

Still, I guess its the best form of flattery when someone copies your work

To me your product will always be a pale copy of Pennywize.

Steve
Pennywize.

[LUCL0NELY]

i've been surfing the net for more than 5 years

[PxG]

Steve,
We dont want to start a pissing match with you, because we dont feel that you are a direct competitor. We offer an Instant denial of open abusable proxies, which is essential for players who want a SOLID running paysite and is only accomplished through our client-server architecture: here's the link for all interested: http://www.proxypass.com/docs/proxypass_tech_faq.pdf

..basically picking up where PW and Iprotect leave off...

On one of our most recent installs (for a customer who was using your product at the time) we helped lower the server load by 85 points and stopped their bandwidth from spiking uncontrollably, AND we locked out the hackers for good. Our customers experiences tells us we have a great product, I hope to see more of you on board!

For those of you who have this problem, the $49.95/server/month price is well worth the unlimted coverage you get.

Rest easy you have http://www.proxypass.com





PxG

[PxG]

And another thing, you said in your post that we stole portions of the 1997 CGI code you devised, VERBATIM. I really hope you're not implying that we stole your code.

Your code is perl/CGI, ours is a C module of apache. Can you please explain to me how we copied your code verbatim when ours is written in a totally different language? You make blocks based on your set of rules, we make ours based on entirely different rules.

And regarding performance, it's silly to suggest that CGI will outperform a C module of apache. But to make things extremely clear, I suggest a friendly publicly posted demonstration where we put your Pennywize up against our ProxyPass. <a href="http://www.proxypass.com/modules.php?op=modload&name=News&file=article&sid= 3&mode=thread&order=0&thold=0" target="_proxypass">We've done this already with Iprotect</a> (another venerable password software) and the results speak for themselves. So how about this, why don't we invite an independent third party webmaster to come forward, someone with a lot of brute force problems. Then let's set up your software and record any and all vital stats of your choosing, including user load. Then, let's install ProxyPass on the same machine and compare the results publicly. In fact, let's do this test on three or four machines if you like, it's up to you.

This should settle the debate about which password protection software is the best since this theme seems to be rehashed on this and other boards often.

[Jade]

I'd love to see the results of the comparison!

[pennywize_v3]

Hi,

Let me quote an email you made to one of my staff members:

"I must admit the configuration of Apache is similar, because it's quite elegant and straightforward solution. "

So let's see exactly how 'similiar' it is:

PENNYWIZE

RewriteEngine on
RewriteCond $cgi/pennywize/$site/%{LA-U:REMOTE_USER} -f [OR]
RewriteCond $cgi/pennywize/$site/%{REMOTE_ADDR} -f
RewriteRule ^/*members.*\$ http://www.pennywize.com/blocked.html [L,R]

YOURS

RewriteEngine on
RewriteCond $p/filters/$s/%{LA-U:REMOTE_USER} -f [OR]
RewriteCond $p/filters/$s/%{REMOTE_ADDR} -f
RewriteRule ^/+members http://www.passprotector.com/blocked.html [L,R]


In your own words, in your email dated on the 7th of November, 2002 you said "I must admit it's similiar". Seems pretty damn similiar to me!

(Even your blocked page is a complete RIP!!!)


Your username or IP address has been blocked
due to excessive usage for one of this reasons:

Your username has been used by too many IP addresses today
Your IP address has been blocked due to excessive failed logins
Your total downloads have exceeded the daily threshold for this site

http://www.passprotector.com/blocked.html

You then have a popup for a porn site, http://www.teencocks.com when you close that page.


It took us months of research and so much work to come up with such a simple yet effective way to block bad accounts - I spent may late weary nights personally working through the apache mod_rewrite manual - and then guys like you come along and rip the hard work we've done. I've never been a guy to like Patents but next time it won't be so easy, let me assure you.

And regarding your challenge, I am not interested. They do this all the time :

apache vs IIS
linux vs freebsd
etc..

And it's never a constructive thing. I have no doubts that your product works and as a C module is probably a timy bit faster than a perl module. But one of my main goals with Pennywize is to make it extremely simple for the webmaster to install. There's no compiling of C code, adding any crontabs (other products), adding SSI to web pages etc.. You just FTP the script onto your server and call it up via. your browser. I've never come across as web server that hasn't had perl on it.

With your product, it has to comminucate backwards and forwards continually to get lists of 'bad proxies'. This i believe is counter effective and like i had said before, i wouldnt put it in Pennywize because it would block too many legitimate users. In fact, I think you've realised that too, because you have recently released your 'stand alone' version which doesnt do this anymore (and that was the whole point of proxy pass being different!)

Pennywize is a product that will continue to evolve and improve, as we are commited to making the best product possible. You are welcome to stay on these boards and slander it as much as you like, but I will be focussing my energies on our clients.

Good luck to you,

Steve.

[fiveyes]

Originally posted by pennywize_v3
...
PENNYWIZE

RewriteEngine on
RewriteCond $cgi/pennywize/$site/%{LA-U:REMOTE_USER} -f [OR]
RewriteCond $cgi/pennywize/$site/%{REMOTE_ADDR} -f
RewriteRule ^/*members.*\$ http://www.pennywize.com/blocked.html [L,R]

YOURS

RewriteEngine on
RewriteCond $p/filters/$s/%{LA-U:REMOTE_USER} -f [OR]
RewriteCond $p/filters/$s/%{REMOTE_ADDR} -f
RewriteRule ^/+members http://www.passprotector.com/blocked.html [L,R]
...

Both of you need to study a bit more on how to write a correct RewriteRule.

HINT: After "^/", what does the "*" or "+" actually do and why is it complete nonsense?

[pennywize_v3]

Hi Robert

Good question - Without it, the rewriterule would look like this:

RewriteRule ^/members.*\$ http://blah.......

The problem is that people can go to

http://www.somesite.com/////////members and it still works.. but because the rewriterule would specifically look for "/members" (only one slash), it wouldn't get matched. Therefore hackers were 'getting around' the protection.

This was a known bug for a while until it was corrected

Steve

[fiveyes]

Originally posted by pennywize_v3
Hi Robert

Good question - Without it, the rewriterule would look like this:

RewriteRule ^/members.*\$ http://blah.......

The problem is that people can go to

http://www.somesite.com/////////members and it still works.. but because the rewriterule would specifically look for "/members" (only one slash), it wouldn't get matched. Therefore hackers were 'getting around' the protection.

This was a known bug for a while until it was corrected

Steve

Hi Steve! (my name's fiveyes btw, Robert Woodruff is just a fellow I like to quote )
Since rewrite rules are checked first, before it's RewriteConds, they should be written in the most effecient manner possible. You're checking for access to all files within a directory, but that should already be determined by placement (either where the .htaccess file resides or, better, which Directory block within the config files the code is found).

In other words, you really don't want this code to apply to an entire site, where it's being parsed and evaluated for every file downloaded by a surfer taking a tour, but only for the files downloaded from the directory "/members/". Right? And to do that, you put an .htaccess file within that directory or, better, have a Directory block for it that has the code within it.

So, if the coding is correctly checking only the file downloads from the diectory "/members/", why are you having the rewrite engine check to see if the requested file is from there?!?

Much better is simply "RewriteRule $.*?", which is short-circuited to automatically apply to all files within that directory tree.

[pennywize_v3]

Hi Fiveyes,

Yep, you are absolutely right there - You only want it to be evaluated on calls that pertain to the members section only (or that directory).

The new version 3 shortcuts it by checking to see if the AUTH_TYPE = "basic" instead of having to get webmaster to try and modify a regular expression with their members section directory (not everyone uses /members, there's lots we come across like /protected, /vip, /access/members etc..)!!

For customers who have high volume traffic and want it optimized completely we work with them (doing things like the <directory> matching). Most of the big guys have entire virtualhosts devoted to members (eg. http://members.sometime.com) so the rule would apply to everything in that case.

Thanks for your comments!

Steve

[PxG]

Steve,

I think you have us mixed up with someone who uses an architecture similar to yours. We do not ever put RewriteCond rules into our apache config files for any reason. Sorry, but you are referring to another product, not ours, perhaps http://www.passprotector.com? We are ProxyPass.com. We built our own set of Apache directives and our architecture is an apache module that triggers before mod_auth(or similar) kicks in, it does not rely on a RewriteCond to trigger it. Either way, i am sure there are a lot of good ideas you guys have put into your product, I don't doubt that at all. And if you mean that we stole the concept of blocking password shares and brute force hackers, then yes, of course we are not the first to build such a system.

Secondly, we have a Makefile installation that is one simple step:
perl Makefile.PL
and you're done. And we do the installations for our customers anyway. There is no messy C file compilation or the like.

Lastly, the ProxyPass Standalone is actually for clients who do not wish to pay a montly recurring fee. However, very few of our clients would ever want to be without protection from open proxies.

We never meant to slander you or your products, we simply feel strongly that our product is superior in terms of performance. I was just suggesting a friendly comparison. At any rate, the best of luck to you and clients in this common struggle against brute force attacks and password sharing.

[Big E]

PxG, since your product is an Apache module, that means it deals with brute force at the Apache level, correct?

So any brute force attempts are still being dealt with in Apache - thus still susceptible to being DoS'ed due to excess load, given 'load x' correct?

[PxG]

Originally posted by Big E
PxG, since your product is an Apache module, that means it deals with brute force at the Apache level, correct?

So any brute force attempts are still being dealt with in Apache - thus still susceptible to being DoS'ed due to excess load, given 'load x' correct?

Big E,

First, let me give some background on your premise and then I'll try to do my best at explaining our solution to the problem. The problem you are describing is actually why we built this software. Its one thing to have your passwords stolen and to lose a customer here and there, it is an entirely different situation when, in the process of running a dictionary cracking attack, the brute forcer sends so many requests to your mod_auth, that your Apache hangs. The critical issue here is not that Apache is getting tons of hits (Apache can handle a very high number of simultaneous hits), its that mod_auth (that little grey box that pops up and asks you to enter your passwords) is a VERY poorly written and inefficient module in Apache. That is to say, that when a cracker brings your server to its knees it is because Apache is having to run mod_auth too many times in a short period of time. This essentially creates the DoS effect that you mention: your server's Apache hangs and your system crashes.

This problem would most likely go away (to some degree) if the major billing companies didn't insist on using mod_auth (the grey pop up box) as their authentication scheme. But they do, so this is the system which we must deal with.

So here is why ProxyPass alleviates the stress on your servers during a heavy brute force attack: We wrote a module that integrates with mod_auth that quickly and efficiently checks a variety of things and automatically severes the connection to the attacker BEFORE the brunt of mod_auth is run. To put it another way, we pre-empt mod_auth and do not require it to run it's course and therefore save your Apache the terrible scenario of manually checking and rejecting password attemtps through mod_auth. We have a small localized DB that it first checks for local blocks like password shares, username bursts, etc. Then, the module quickly analyzes the hit for flags that indicate a proxy is being used. We use a UDP connection to our central server at that point to quickly determine if the hit is an open proxy, and will automatically deny it, again without running mod_auth. We do a number of other things, but we prevent the DoS effect essentially by pre-empting mod_auth when it doesn't need to be called.
For example, if an attack "state" is reached, we simply start dropping connections from open proxies, this really screws with the attacker because their connections are never answered, in essence, they are black holed. All the meanwhile, your other users can successfully authenticate and enjoy your site.

Any apache server will eventually crap out given enough simultaneous traffic. It's just a fact of life, and there is no way around it. That's why the world doesn't run on one server alone. But mod_auth is particularly costly in terms of memory and CPU and it is more prone to crap out under a load that is far less than a system that authenticates through php or some other means.
That's why when you install ProxyPass the userload goes way down and your server's performance goes way up.

If you or anyone has any more questions, I would be happy to answer them.

[erotictrance]

Originally posted by OneHungLo
Let me tell you this...If your using ccbill you are fucked. Their scripts are wide open. I got friendly with this password cracker and has key generators for literally 1000's of ccbill sites so basically he can write paswords to your password file at free will. I contacted CCbill with this info and they were clueless, they were asking me to ask this 16 yr old how to fix it even after i sent them the whole convo WITH the script the kid made to be able to do this...what a joke. Another sad thing is that i HIGHLY doubt they contacted everyone about this.

You're right. I asked CCBill about this the other day, because I've been hearing about this same hack from other people. Naturally, CCBill said it wasn't a problem.

[Big E]

Originally posted by PxG

The critical issue here is not that Apache is getting tons of hits (Apache can handle a very high number of simultaneous hits), its that mod_auth (that little grey box that pops up and asks you to enter your passwords) is a VERY poorly written and inefficient module in Apache. That is to say, that when a cracker brings your server to its knees it is because Apache is having to run mod_auth too many times in a short period of time. This essentially creates the DoS effect that you mention: your server's Apache hangs and your system crashes.

Somewhat true, but also dependent on what kind of auth you're doing. DBM and Mysql are much, MUCH more efficient than NCSA.

This problem would most likely go away (to some degree) if the major billing companies didn't insist on using mod_auth (the grey pop up box) as their authentication scheme. But they do, so this is the system which we must deal with.

What would you have them do? (I think you mean DBM vs NCSA)

here is why ProxyPass alleviates the stress on your servers during a heavy brute force attack: We wrote a module that integrates with mod_auth that quickly and efficiently checks a variety of things and automatically severes the connection to the attacker BEFORE the brunt of mod_auth is run. To put it another way, we pre-empt mod_auth and do not require it to run it's course and therefore save your Apache the terrible scenario of manually checking and rejecting password attemtps through mod_auth. We have a small localized DB that it first checks for local blocks like password shares, username bursts, etc. Then, the module quickly analyzes the hit for flags that indicate a proxy is being used. We use a UDP connection to our central server at that point to quickly determine if the hit is an open proxy, and will automatically deny it, again without running mod_auth. We do a number of other things, but we prevent the DoS effect essentially by pre-empting mod_auth when it doesn't need to be called.
For example, if an attack "state" is reached, we simply start dropping connections from open proxies, this really screws with the attacker because their connections are never answered, in essence, they are black holed. All the meanwhile, your other users can successfully authenticate and enjoy your site.


Yeah, you're dropping connections but you're dropping them at the application layer. Why not /dev/null them at the network level via ipfilter/ipchains/iptable level? That way Apache (or any other application/service up the OSI model) never even sees the packet. Much more "clean" solution.

Any apache server will eventually crap out given enough simultaneous traffic. It's just a fact of life, and there is no way around it. That's why the world doesn't run on one server alone. But mod_auth is particularly costly in terms of memory and CPU and it is more prone to crap out under a load that is far less than a system that authenticates through php or some other means.
That's why when you install ProxyPass the userload goes way down and your server's performance goes way up.

I admire your solution but it's just too "busy" for me. Too many things to calculate and go wrong.

[Que?]

Had some problems with ccbill passes could potentially be surfers using the same login at many sites.

Otherwise the problem has been with AOL users as they(AOL) use different ip adressess and get logged as multiple users.

If you set your block limit at seven the AOL problem should disappear...