TROJAN infected our WEBSERVER!
 

I mean, really.

WHAT INSANITY!

A trojan infected our WEBSERVER.

I'm just beside myself here. I've never had this happen to me at any other place I have had a server colocated.

A customer called in and complained that when they access our site through a google search their browser get's hijacked. Sure enough, a few seconds later the plethora of our sites on that server are completely in the bucket. Any hit to our sites attempts to download the trojan to the browser's computer.

Smoke is just coming outta my ears over here. Anyone with any information, experience, suggestions (besides the obvious = "look for a new web host" which will be done) - please speak up!

Get a good sys admin.

Who do you host with?

Man that sucks.

Yea i'd be interested in knowing who you host with too?

Who's your host?

Firewall? Server Anti-virus software ?

If it is just a file, find it and delete it.

If a process is running to do that, identify the process, kill it and try to remove it.

your host isn't doing shit about it?

mothafuckin POWERMEDIUM a.k.a. CANDID HOSTING

they have been trying to fix it for about a half an hour now! I am fucking livid that it happened in the FIRST PLACE

candid? lol

thought they kicked the bucket long ago..

get a real host

even geocities would be better

Ns1.candidhosting.com 64.159.90.4
Ns2.candidhosting.com 64.159.90.10

Why are you livid?

Having a virus has absolutely NOTHING to do with your host. And you don't even know if you have a virus or if someone hacked in and did this all manually which is usually the case.

Your host only handles the server itself and the processes needed to run inorder for it to be a web server. (the kernel, apache, mysql, PHP)...

You are responsible for EVERYTHING else. So when you people install phpBB 2.0.10 and get hacked, don't complain to the host... or when you fail to install software to monitor problems like these...

it all depends on what was hacked... you are responsible for maintaining software you have access to .. not your host, they are responsible for the things they control.. i wouldnt be so quit to blame them just yet

who the fuck uses the word plethora

Firstly, this is the 4th in a chain of problems that has me pissed with PowerMedium / Candid.

Secondly, it IS the responsibility of the host to maintain network security. Firewalls to the facility, server antivirus, etc. No?

I haven't installed phpBB or any other out-of-box software. I am unsure how this happened - and if the sysadmin contacts me before next year to let me know how they traced the issue and it points to something that is not their fault I will be a bit blush, not mad at them, and more than a bit surprised.

Tell about the other problems... You might just help someone NOT make a bad decision on their hosting for 2006.

When you say security it is pretty black and white. They are only responsible for THEIR networks security. Routers, switches, etc.. if someone writes a virus for a switch they use, they need to upgrade and patch its software.

As for your server, they only provide help if the entry point was a piece of software they are responsible for. Examples from above, the kernel, apache, mysql, sendmail, etc...

If it turns out the entry point was a script in your affiliate program or other software YOU have control over, then they are off the hook on this one.

candid wont do shit

Does a sysadmin come with a colo at Candid or whatever the new name is?

From tech support:

"Vic,

There were some older versions of PHP on your server which may have been
causing this. I've upgraded and I can't recreate the error that was
happening before."

a trojan? wtf is it a windows server?

Did they ever sen you php vulnerability notices for the versions your running?

That sucks man let me know if you need anything...

My other issues with them

- no a dedicated sysadmin doesn't come with a colo, you are lumped into a tech support pool which really is only 9to5, Monday - Friday unless you call in and demand an escalation on your ticket.

- our mailserver is out of control and they can't seem to do anything about it. by "out of control" i mean being used to send out email by spamers.

- they couldn't manage to get elmlm installed correctly for me

- they tried to charge me for october's bandwidth on november's (smaller) bandwidth contract (we took our content to Limelight Networks for distribution).

- now this.

So what do you think of their response? I had a half a fuckin hour complete downtime while waiting for them to decide that it was possibly caused by an older version of php. and that they can't trace the problem. My focus is blurry right now, so any advice you might offer about how to deal with the situation will be helpful

hire a system admin who knows wtf he's doing. No way can or should you rely on the colo people to take care of you. :2 cents:

An older version of PHP will not cause a nirus/trojan/scamware to create it self.

An older version will expose you to security risks.

Updated version from 4.1.1 to 4.1.2 ( just as an example ... ) are in fact security patches most of the time.

Get a good admin ( Dynaspain as example ) to close the ports not used, change the numbers of the one used to complicate the hackers task ...

But this situation is not really the responsability of your host. You have a CP on that box. YOU should update the OS , HHTP , etc .. software. :2 cents:

The spam issue might just be a highjack of another customers for instance guestbook script (assuming the box is shared). I know several hosts which has a "free" unsecure guestbook script. These are usually found by hackers by doing a simple search from google.

I guess thats where the line is drawn between pro hosts and, hmm... not pros. :disgust

Vic, hit these people up, they have been awesome for me for 6 years.

www.cyberwurx.com

I'm a host myself and I'll take my chances here as to what happened, as I've seen this on one of my customers servers.

The real culprit could be any of the following combination of things:

Old PHP version allowing an easy code exploit
Poorly written PHP/CGI code that allowed an exploit
Error Reporting turned on providing critical info to the attacker to exploit your system

Chances are no matter how good a sys admin, or how great a host you get -- you can still be affected by these things unless from the ground up you build your sites with security in mind which almost no one ever does as it greatly limits the suite of available applications.

For example with PHP you must run with safe mode enabled along with a slew of other options that will need to be enabled to ensure proper security, mind you though that a number of scripts will not run with safe mode turned on.

* As for the attack they most likely overwrote your html/php files and inserted a few lines of code, all you need to do is get an experienced tech to write a script which will go thru every file that has been affected on your system and remove the malicious lines of code.

Good Luck as its a serious Pain in the Ass!

pornguy, thanks for the lead.

so here's what it seemed to be:

We were running PHP 4.3.7 (Unix - not shared, dedicated)

Apparently someone took advantage of this to create the following effect: hits to our php page would (often, apparently, not always) redirect and attempt to get the client to click "OK" to download the browser hijacker.

I am led to hope that the upgrade of PHP to 4.3.11 closes this vulnerability. So far, it appears to. We've been running for about 15 minutes without a detected incident.

A QUESTION: Those of you with dedicated servers (sorry, I used "colocated" earlier, which is not exactly what we have) - do you receive notifications from your server when a php upgrade should be installed, etc?

some do , some dont, its always good to have someone on top of that sort of thing :) If you arent satisfied your host is adequately taking care of it in a timely fashion then its up to you.. it cant hurt to ask them .

if leaving error reporting on is a vulnerability (thanks zagi, i've turned it off) - i imagine leaving a file called "phpinfo.php" is a bit of a no no, right? But this is the second time tech support has left this file hanging around.

Also, I found no files modified besides those i modified myself.

It shouldn't be possible to fuck up the whole server, if the host is doing its job :2 cents:

from candid's tech support:

"It looks like someone may have used one of the numerous exploits in PHP
4.3.7 to overflow apache and install a new module into apache which
redirected traffic randomly from google."

Any comments on this?

How Have You Got A Trojan On Your Server?

Here's some advice. Your entire system has likely been rootkitted (what is a rootkit? Check here). My advice:

1. Migrate all data (data, files, pics, movies, config settings etc.) to a clean backup box
2. Reinstall the infected machine FROM SCRATCH
3. Upgrade all modules, firewall all unneeded ports
4. Copy your data back
5. Cron monitor all system and config files w/ md5 checksums

Takes half a day or even a whole day. Good luck.

it's not a trojan on the server... but it was redirecting traffic to download a trojan to the client.

David, thanks for the checklist, but rooting is done against WINDOWS boxes, right? We run on Unix.

That version still has some " holes" . Why didn't you update to 4.4.1 ? I did that progressively on all my boxes.

Do you have a control panel. If you have Cpanel ( as an example), you can select in WHM hahahaha=> software hahahaha=> Apache update . You will be presented with various choices ( GD library, Curl, etc .. ) including various versions of PHP.

If not, you can do in ssh command line an update by running the update apache command. Again, you will have choices available.


The host/datacentre will not inform you of that... They will inform you of kernel update, which they perform.

Once again I suggest you seek professional help :1orglaugh

For scripts that don't run with " safe_mode" on :

Activate Safe Mode Using Per Site Basis

Now if you have scripts that require safe mode off like Modernbill or any script doesn't work well with safe_mode on what you will do? Disable safe_mode on the entire server just for these scripts? This isn?t very practical when you can disable php safe mode per user account/site basis.

Let?s do it!
1) SSH to your server and login as root.

2) Then find the httpd.conf, normally it?s in /etc/httpd/conf/ or /usr/local/apache/conf/
If it?s not in either of those places try search for it: locate httpd.conf

3) Then find the site you wish to edit.
Ctrl+W and type in the domain name

You should see something like this:
4) Now add this line:

php_admin_flag safe_mode Off

We have also found that the following works as well if the above does not but DO NOT USE BOTH, pick one!

php_admin_value safe_mode 0

to be like this :
5) Good :) Now save the changes.
Ctrl + X then Y

6) Restart the Apache web server by
/etc/init.d/httpd restart

Final Words

PHP Safe mode should be on by default on all your servers for added security. However there are some scripts that are not compatible with it on so you have to make an exception to some client sites. Make sure you know why they?re requesting to have it turned off because it is much more secure for everyone to have it on.

If you run into trouble after editing httpd.conf you can run the apachectl configtest
in shell. This will test the Apache configuration for errors and report them back to you if you can?t start it, very handy indeed!

lol no it's done on any type of box. Linux, BSD, SunOS, Windows you name it.

Consider going through what I suggested, you are only asking for trouble if you just patch up PHP and Apache and bring the box(es) back online.

My tech at the hosting company: "I don't think your system was rootkitted. I don't see any trace of it. The nature of what happened to your server is that PHP or apache (both which run as unprivileged users) can be overflowed. The most the user has access to is the files the apache user owns, which were replaced when I recompiled PHP and apache."

I don't want to rub in, but if this was a windows machine, everyone on this board would have said something bad about windows...WOW UNIX and no one say "how great UNIX is"?

It funny how the first thing people will bash is Candid, without even getting all the facts.

I guess UNIX are not great perfect machines as anyone on this board try to make.

both unix and windows are ok as long as you don't hand them to a clueless
person.

it doesn't matter what OS you are using, if you don't know the first thing about security your system will be compromised.

well call me naieve (sp?), but in the past getting a dedicated server somewhere and having them set it up, the techs there are experts and set the machine up properly. I.E. they turn off error reporting, mount /tmp noexec, set up suexec domains, etc. etc. right from the start - part of the reason why I pay a fee to a hosting company is to take advantage of the expertise they have in running and setting up servers. I am dissatisfied with Candid - and I believe rightfully so - because they set this box up in ways that they now come back and tell me are not safe.

Lesson learned - I need to personally learn-up on all that shit and get up to speed with it, and make sure that my tech support people are doing what they should be. Reality is a bummer.

that's because you are using windows server! LOL! :D