If a managed server is hacked ... who's at fault?

It's a two way street, while a managed host does do more work in maintaining your server and also accepting some more responsbility in terms of keeping things secure, the customer also has some liability. Since it is usually a script or something that the user put on the server that created the initial whole through which the user was able to get through.

It really depends on the variables of the situation to develop a more clear picture of the situation.

depends on what scripts you have running on that server

if you have something like phpbb, which is known for it's backdoor security holes, then it is your fault

Sorry to hear that... One of my sites got hacked this morning, but I had backups and had it running again in minutes.

I have to attribute this more to the amateur nature of the hacking than any skills of my own, though. I'm just glad I backed up.

The host have to keep it patched and updated. But its not always their fault since it depends on how they got in. Most of the time its because the scripts/cms is full of security holes, and you cant blame the host for that.

It is probably the script'(s') fault.

Data loss is ultimately your responsibility - either make sure your host provides backups, either as part of your hosting plan or as an add-on, or make sure you back your data up very frequently.

If a hacker wants in bad enough, he/she will find a way into any box.

As mentioned, some user created scripts open up huge security gaps. It can and does happen, regardless of how good the management team is. Situations like this are why its extremely important to have weekly back-ups at bare minimum. If you're updating your site often and it is even remotely important to you, shoot for daily back-ups. It may cost you a couple extra bucks every month but that added insurance will save you from potential mishaps and even hard drive failure.

Lol, so if it's managed you can't be hacked? How is it their fault?

backups perdiodically is the best we can do.
i dont make them myself but i should start doing it before something like that happends to me.

hackers fault

I'm sure you don't want to hear this but unless the hack was via an OS or system level vulnerability then you are responsible. (Now for the shameless plug) My company ScannerX scans your server, dedicated, managed or virtual doesn't matter, and identifies any and all vulnerabilities that a hacker can use to break in. We then generate a report detailing all the vulnerabilities and how to fix them. With our service you can scan your server monthly, weekly, daily, or even hourly if you want to. All for the same low, low price of $19.95 per month. http://www.scannerx.com/webmasters.html

BTW we also have an affiliate program

how can you possibly run a usefull scan on a virtual hosting server where you
don't have root uid. In order to do any usefull tests you need to check a lot
if files which are not accessible for non root users.

Please explain more about your scan, give us some technical info on how
you scan. And I don't mean the nice sales text that you have on your site.
Also your site says you use open source software together with custom stuff.
Are you aware that you cannot sell/make money of open source packages?

You can use them freely but you can't whore it out for money. Don't get me
wrong I'm not trying to bash you.....I just want some more informaion and
whenever I see stuff that sounds dodgy I point it out.....if you have a good
service excellent I don't mind any competition......in case you might think
that. But if it's dodgy and if any company uses false information to take
advantage of people.....then I share that aswell.

Does your scan run localy on the machine that is scanned? What program
language is used? Give me more info on what kind of checks. Does it check
if files/binaries have been tampered with and how?

No problem I don?t mind answering good honest questions.

Q: How can you possibly run a useful scan on a virtual hosting server where you don't have root uid.

A: On a virtual server we can identify any system or OS level vulnerabilities that could affect the entire server. Granted without root the client could only ask for those issues to be fixed by the provider. Also, just because you are on a virtual server and don?t have root that does not preclude our scan from checking your webapps for things like SQL inject and Xsite scripting among others.

Q: Please explain more about your scan, give us some technical info on how you scan. And I don't mean the nice sales text that you have on your site.

A: Our primary engine is based on Nessus but we have made many significant modifications like enhanced web crawling, five levels of critically per vulnerability, and a downloadable scanner iso for internal checks. (see bottom of post for all the open source tools we use)

Q: Also your site says you use open source software together with custom stuff. Are you aware that you cannot sell/make money of open source packages?

A: I?m not sure that you have your facts straight here. You can sell or make money off almost any open source package so long as you are in compliance with the licensing and release, as open source, any modifications that you have made.

List of open source tools that we have incorporated into our service

Arphound
A tool that listens to all traffic on a network interface. It reports IP/MAC address pairs as well as events such as IP conflicts, IP changes, IP addresses with no RDNS, various ARP spoofing, and packets not using the expected gateway.

Arping
A network tool to broadcast ARP packets and receive replies similar to "ping." Good for mapping a local network and finding used IP space.

ARPwatch
Keeps track of Ethernet/IP address pairings and can detect unusual behavior.

Bing
Bandwidth Ping. A point-to-point bandwidth measurement tool, based on ping. Can measure raw throughput between any two network links.

Bugtraq
A database of known vulnerabilities and exploits providing a large quantity of technical information and resources.

CVE
The Common Vulnerabilities and Exposures dictionary. CVE provides a large quantity of technical information and resources about thousands of vulnerabilities.

Dig
Performs detailed queries about DNS records and zones, extracting configuration and administrative information about a network or domain.

DNStracer
A tool to determine the data source for a given DNS server and follow the chain of DNS servers back to the authorative sources.

Dsniff
A network auditing tool to capture username, password, and authentication information on a local subnet.

Filesnarf
A network auditing tool to capture file transfers and file sharing traffic on a local subnet.

FindSMB
Used to find and describe SMB servers on the local network.

Fping
A utility similar to ping that performs parallel network discovery.

Fragroute
Intercepts, modifies, and rewrites egress traffic destined for a specified host, implementing several IDS evasion techniques.

Fragtest
Tests the IP fragment reassembly behavior of the TCP stack on a target.

Google
Internet search engine that can be used to help search for misconfigurations and/or exposed sensitive information on a network.

Hackbot
A host exploration tool, simple vulnerability scanner, and banner logger.

Hmap
Detailed fingerprinting of web servers to identify vendor, version, patch level, included modules, and much more.

Host
A utility to perform DNS queries, zone transfers, and more.

Hping
Hping and Hping2. A TCP/IP packet assembler and analyzer. Can perform firewall ruleset testing, port scanning, network TOS/QOS testing, MTU discovery, alternate-protocol traceroute, TCP stack auditing, and much more.

Httping
Similar to "ping" but for HTTP requests. Show how long a URL will take to connect, send a request, and receive a reply.

Hunt
A tool for exploiting well known weaknesses in the TCP/IP protocol suite.

LEAP Cracker
A suite of tools to break the NTChallengeResponse encryption technique of the LEAP authentication system used by various vendors of wireless devices.

Libwhisker
Application library designed to assist in scanning for CGI/web vulnerabilities.

Mailsnarf
A network auditing tool to capture SMTP and POP3 email traffic (including message headers, bodies, and attachments) on a local subnet.

Msgsnarf
A network auditing tool to capture instant message (Yahoo, MSN, ICQ, iChat, AIM, and many more) traffic on a local subnet.

NBTScan
A utility for scanning networks for NetBIOS information. Reports IP address, NetBIOS name, logged-in user name, and MAC address.

Nemesis
A network custom packet creation and injection utility.

Nessus
A powerful, fast, and modular security scanner that tests for many thousands of vulnerabilities. The Edgeos system can also be used to create custom Nessus reports.

Netcat
A utility to read and write custom TCP/UDP data packets across a network connection for network debugging or exploration.

NGrep
Similar functions to GNU grep, but applied to the network layer. A packet to sniff network packet payloads and match them against extended regular or hexadecimal expressions.

Nikto
A web server vulnerability scanner that tests over 2,600 potentially dangerous files/CGIs on over 625 types of servers.

Nmap
A port scanner, operating system fingerprinter, service/version identifer, and much more. Nmap is designed to rapidly scan large networks.

OSVDB
The open source vulnerability database providing a large quantity of technical information and resources about thousands of vulnerabilities.

Pathchar
A network tool for inferring the characteristics of Internet paths, including layer-3 hops, bandwidth capacity, and autonomous system (AS) information.

Ping
Standard network utility to send ICMP packets to a target host.

ScanSSH
ScanSSH supports scanning a list of addresses and networks for open proxies, SSH protocol servers, Web and SMTP servers. Where possible, ScanSSH displays the version number of the running services.

SinFP
SinFP is an OS fingerprinting tool that determines the target OS with used TCP frames.

SMBclient
A client to talk to a SMB (Samba, Windows File Sharing) server. Operations include getting files from the server, putting files on the server, retrieving directory information, and more.

SMBtree
A tool to discover and browse SMB (Samba, Windows File Sharing) services. Prints a tree with all the known domains, the servers in those domains, and the shares on the servers.

SMTPscan
A tool to determine the type and version of a remote SMTP mail server based on active probing and analyzing error codes of the target SMTP server.

SSL Certificate Check
ssl-cert-check checks the expiration status of digital certificates on SSL servers.

TCPdump
A network tool for monitoring, protocol debugging, and data acquisition.

TCPreplay
A utility to read captured tcpdump/pcap data and "replay" it back onto the network at arbitrary speeds.

TCPtraceroute
Similar to the "traceroute" network utility, but uses TCP SYN packets instead of ICMP or UDP, attempting to bypass firewalls and packet filters.

THC-Amap
A scanner to remotely fingerprint and identify network applications and services.

THC-Hydra
Network-based authentication/login cracking system supporting almost any service or protocol.

THC-RUT
A tool offering a wide range of network discovery utilities, like ARP lookup on an IP range, spoofed DHCP request, RARP, BOOTP, ICMP-ping, ICMP address mask request, OS fingerprinting, and high-speed host discovery.

THC-Vmap
A scanner to remotely identify version information about network applications and services.

Traceroute
Standard network utility to trace the logical path to a target host by sending ICMP or UDP packets with incrementing TTLs.

URLsnarf
A network auditing tool to capture HTTP traffic on a local subnet.

Whois
A tool to query both domain name and IP address registries to find owner and assignment information.

CHANGE YOUR SIG...IT VIOLATES THE RULES OF GFY

TEXASDREAMS IS GOING TO HACK YOUR SIG

ScannerX,

Sorry to rain on your parade... But hopefully I can offer some input on the side of the actual folks on the front lines here.

Nessus (no matter how modified) will be of fairly limited usefulness for any even remotely properly managed *NIX server. On windows, I'll give you that, since my expertise simply does not lie there.

I havn't seen actual OS-level or "daemon level" (e.g. apache, bind, sendmail, sshd, etc.) in-the-wild actual exploit on our network for a LONG while. In fact, I can count on one hand the number of local root exploits we've had lately even after customers left remote holes open. Nessus is great for finding those holes, however since they are rare the product simply doesn't offer too much for us other than a "oh shit" type of scan where someone REALLY screwed up and left something running accidently.

Now.. for something I absolutely *would* pay good money for. I want essentially a virus scan, which scans for ALL known exploitable PHP/perl/whatever files on the system. This means, it will keep signatures of all PHPbb files that can be exploited, etc. Remote scans are near-worthless in my opinion, as they simply "guess" at what pathnames a client may use. If I have a nightly scan going through the entire filesystems on my machines, I can be assured every file is checked. There is nothing keeping anyone from creating a product like this, save the work involved. Basically take clamav (or your favorite open source *nix AV scanner) and simply create your own definitions file. Watch all the security lists, test the exploits, and add signatures hourly/daily/whatever. I would absolutely subscribe to a "definitions feed" service that was reliable and trustworthy, and would be willing to pay at minimum multiple thousands/mo for the privilege. However, the service would absolutely have to be very complete and kept up to date.

If/when someone actually comes up with a workable, supported, and *good* product such that that, I think they'd find a whole lot of success selling to the hosting provider market. I would love nothing more than to be able to proactively contact customers and put in hotfixes for "zero day" random-script-of-the-week exploits. Currently it's very much a reactive process.

As for the original poster - sorry for threadjacking. But pretty much everyone has it more or less right. If the entry vector was a script you uploaded or requested to be installed, it would be your responsibility to keep it up to date. A host simply can in absolutely no way take responsibility for third party software. However, they should have decent tools/staff to help you out after the fact and try to come up with what happened. However, even that can be an excersize in futility depending on the "hack" used.

Peace,

-Phil

And by running all these tests you're giving your client lots of false positive results.
Hacking nowadays is more like vulnerable php and weak ssh passwords

prob via a script. if its crappy code and you can do a sql injection, you can do anything, create a user with root access. people need to write secure code.

A host? Take responsiblity? For something they fucked up or neglected to do?

What planet have you been on? Hosts arent responsible for anything. Even if your agreement says they were supposed to apply patches, etc. Not responsible. Never ever.

The word "hacked" is used much to loosely these days. Most of the time you should be using the term "script kiddies". They are usually the so called hacker. Anyone can find a hole, or exploit when they are using software someone else wrote. Go download it, scan the shit out of 50,000 IP's. Break it down to the few that are running the software version your little script is able to exploit, and go nuts. What did they really do but sit around watching things happen. A hacker is someone that you usually won't even know hit you, at least for awhile. They are after information 99% of the time. They find their way in, get the info they want, and cover their tracks on the way out. Script kiddies leave a trail that a blind person could see.

Was said before, but always needs to be said again. Back up your data.
Change your passwords every 30 days.
Know the software you are running, and make it a point to watch for exploits.
Back up your data!
Back up your data!
And last.. the most important thing of all. Back up your data!

What has happened to you sucks, I know, I have seen it. Just make sure this teaches you to never let it catch you off guard again. If someone wants into your box, there is no sure way to keep them out. There has been cases of machines being hacked where a brute force attack ran for months until they got in. There is only one sure way to be safe from someone hacking you on the net. Pull your Ethernet cable.

That's unacceptable with a good host... Good managed host will make sure backups of your data are available, and you should be back up running within a few hours...

but expect to pay extra for premium service like that, you shouldn't expect much more than getting help with creating a database from a $99/month "managed" dedicated server host...

No rain Phil21 but you do make a good point. Not properly managing a *nix server is the biggest problem we see. Many of our clients are either in a managed environment and think that the server is ?properly managing? or they are on dedicated server and don?t know what they don?t know. In either case what usually ends up happening is they get hacked/defaced and they are not sure how it happened or whom to blame i.e. this post. In a managed environment our clients often use us as a trusted third party to validate that their provider is doing their job correctly. In a dedicated environment our clients once again uses us to analysis their current security state, identify what holes need to be fixed, prioritize the fix implementation and then validate that the fix is in place.

As to false positives, of course they happen but we work diligently with our clients to eliminate the root cause. Additionally, our service includes a threat level editor so that if you find a false positive is popping up too much you can either lower the threat level or select to ignore it.

Finally, our service does find exploitable vulnerabilities remotely on hosts on a daily basis. These include OS-level and webapp level vulnerabilities. Because of the webcrawling feature we analysis and follow every link on a page for php, .net, asp and other vulnerabilities that could lead to SQL injections, xsite scripting and other exploits.

I?m happy to talk all day long about our services but I?m a firm believer that the proof is in the pudding. So, I offer everyone on this thread a test of our service for free. Shoot me an email, mpearson at scannerx.com, I?ll give you a free scan and if you still think I?m full of shit you?ll have the proof to back it up. Otherwise, if you find that what we offer is valuable that I hope you would convey that here as well.

I like that you are willing to back up what you claim.

= You are fucked!

To back up a claim like that would be foolish...

Fixing someones hacked server or providing protection so a server won't get hacked is not a full-proof job. But to do nothing and pretend you won't have problems is just plain stupid.

I think a lot of good things have been explained already. I would simply emphasize that anybody who truly depends on their web sites ought to be paying for appropriate backups - whether they are monthly full backups, weekly incrementals or daily backups on databases and configurations. "Hacking" aside, let us not forget that hard drives and even RAID configurations are entirely capable of complete failure and even with active monitoring of hard drive health these things can happen.

By our experience, most exploited servers are a result of a poorly written scripts. Best practices, I think, are to work closely with your managed host - not just to have them install scripts, but to have them help with script selection too.

Cheers,

Brad

I agree but I?m not saying that we can prevent any and all hacks but rather that we can help find the holes that a hacker can use to break in. And I?ll stand behind that claim all day long.

Haha here you go:

The Ultimately Secure DEEP PACKET INSPECTION AND APPLICATION SECURITY SYSTEM

Featuring signature-less anomaly detection and blocking technology with application awareness and layer-7 state tracking!!!



Installation Instructions

For best effect install the firewall between the CPU unit and the wall outlet. Place the jaws of the firewall across the power cord, and bear down firmly. Be sure to wear rubber gloves while installing the firewall or assign the task to a junior system manager. If the firewall is installed properly, all the lights on the CPU will turn dark and the fans will grow quiet. This indicates that the system has entered a secure state

For Internet use install the firewall between the demarc of the T1 to the Internet. Place the jaws of the firewall across the T1 line lead, and bear down firmly. When your Internet service provider's network operations center calls to inform you that they have lost connectivity to your site, the firewall is correctly installed.

The firewall above is the only 100% guaranteed secure solution.

(* May have a performance impact on traffic if prevention is enabled)

sorry here's the pic

Not at all entirely true. It depends what license the OSS is bundled under. The BSD license grants open source or proprietary use of its software (classic eg is Apple's OS X operating system), whereas Gnu Public License (GPL) requires any derivative works (which can be saleable) to be distributed in source code under GPL or compatible license. People get around the GPL license by daemonising their software, in order to avoid creating a derivatinve work.

BTW Apache is an open source package that most of us are making money using!

crap coding aside, I find that a really strict ruleset on a kernel-level firewall (pf my preference) using FreeBSD's daily security run output to immediately patch any server vulnerabilities on my installed software makes my servers quite adequately secure. Of course, I backup to be on the safe side.

Extra layers of security can be added by hosts.allow with tcpwrappers enabled and a little used but darn powerful daemon is DenyHosts to stop dead any brute force attempts

yeah and i was under the impression that is what i was paying for

before i went to a dedicated i used a simple virtual plan and when that box crashed they hda everyones data backed up

but ofcourse now when it comes to me wanting to leave they dont have shit for me

scannerx
ill take that free trial on the server that was just hacked

shoot me an icq 71462500
it is back up and running now and "fixed" or so they claim so it would be interesting

99 times out of a 100 I would sasy its the users fault...not the hosting company.

With that being said; they should have a backup from no longer than 30 days ago. I pay to have daily backups on our servers though

yeah i thought i was paying to
guess not my backups are gone

also it seems that another box on the same range from them was hacked aswell

flaw in there security

so is that still my fault that they didnt patch a managed box?

Most good hosts do run back ups at their expense on their virtual servers. A dedicated server is another story and back ups are usually only done if you request it and pay for that special service otherwise every server a host had online they would need a mirrored drive for and that would be ridicules and totally cost prohibitive.

The usual hacking culprit is olde code that has had a major weakeness public revealed getting exploited. The most common these days seems to be wordpress, vbulletin, and content mangement software such as joomla.

If your hosting company is managing their end even reasonably. There usually isn't many holes left open on their end. As soon as you install a third part piece of software, especially one that is popular and public exposed, you then get the lucky job of trying to keep up with all the security fixes.

I had the pleasure to work with Mike and ScannerX and it was a great experience..

He knows his stuff and i really learnt a lot of valuable information.

I think this is a great service for our industy!!

Where was the flaw in their security? It's quite possible, even probable, that the other hacked site/server was also running a dodgy script. Without details of how the hack happened you can't attribute blame. It's remains quite possibly your fault.

Even the backup thing you cannot blame the host for as you get what you pay for. A server with decent backup usually costs just a few more bucks and oddly few people want to pay it.

However, there is a major caveat which the sense of security backups give you - if you don't know where the exploit came from, and when, restoring from backup to a fresh install could leave you wide open again!

lets say it's some cgi script with a dodgy bit of coding - if you restore to a fresh system, that dodgy cgi script isstill there, leaving you wide open to another hacked session.
Also, lets say the hack occurred 2 weeks ago, but the hacked system was only exploited yesterday - there are lots of hackers that lie dormant for a good feww weeks/months, so that when you restore from your backup a week ago, you are effectively restoring the backdoor.....

It is absolutely essential that you know how the system was exploited, so that it won't happen again....

what he said

-deleted: double post

nomatter what, restoring a server to how it was is a right royal pain in the arse. It's not just your data, but all the system tweeks, custom kernels, configs etc etc that you've been adding over the years that need replacing. A right royal arse - Chris, it isn't your provider that's at fault. There is not a single provider that would guarantee you a hack-free system. If there was, stay well away from them, because they can't.

i am not wanting a full 100% restore
i want one domains file restored
the domain had no scripts
just a fwe html pages and about 200 images

thats it
nothing major...fuck all my other sites ... i was getting tired of baby sitting em just one site i want back

the funkin hackers fault!

On any given day there is a whole slew of bots running scans on random sites/ip ranges accessing known urls to find an array of exploitable scripts. Once it finds one, it then attempts to exploit the script, usually writing files to /tmp, and then executing them. These files they write and execute are usually back doors to the server.

The best way to protect against that is to set your /tmp dir to be noexec, and link your other tmp directories there as well.

But then they ocassionally go and find other writable directories. If you find a hackers script in your domains files, then your directory is probably set writable for the apache web service to right to it.

So not only do you always want to make sure you have the latest updates of all scripts you are running, but you want to make sure your directory permissions are also secure, don't allow writing if it doesn't need to be. Be careful with setting stuff to 777, etc.

I feel for ya - a simple gzip and ftp to your home puter would have save a lot of heartache. I use rsync to monthly backup to my home, in conjunction with dailies to the backup server (the backup server is currently in the same DC as the server, so essential to make offsite backups).

Sorry, but it's simply "live and learn"

If you are not running any scripts at all on your sites on the server, then it definately sounds like something was insecure on the box.

Most hacks these days are due to exploits in scripts, you don't see security issues as often.

Do you have any information on what was found on the server? What was running, etc? I'd be really curious to know.