Hacked Linux Server

[size13]

We've had one of our Linux web servers hacked into overnight.

It only has access from outside to ports 80 and 25
The hackers have copied over every index file (i.e. .htm, .php etc) with thir own.

Does anybody know how they got in?

There are only 3 active user accounts on the server with not-obvious password, e.g. a mix of numbers and letters

It's running Libranet Linux (always updated) and Apache

[JamieBeeston]

There are an infinate number of ways they could have got in.

SSH Exploit
Telnet Exploit
Sendmail / MTA Exploit
Apache Expoit etc

Apache and PHP have both recently had large security holes exposed.. PHP especially allowing access via a multitude of Bulletin Boards / forums / Gallery software.

Speak to your host about 'cleansing' the server, if you're lucky it might have been an 'automatic' attack, with no human intervention, as such easyier to repair..

look into getting a better firewall / Systems Admin on the server to keep the versions current.

Stay Safe.

[__LEE__]

Tbere is a whole host of ways someone could have penetrated your web server from using a know/unknown exploit in Linux or Apache to social hacking etc.

Maybe it would be a good idea to call in a security specialist to carry out forensics on the server and find the fault, because you can bet on the fact that if the whole goes unfixed the person will be back in the near future.

[size13]

Unfortunately we are the host and admin!

The server has been running fine for the past four years and this happens all of a sudden.

Doh, it's my fault

Although it is fully updated, I'll look into you suggestions thanks.

>> Edited by size13 on Tuesday 21st December 15:52

[Targarama]

Time to stick Solaris x86 on it ... if and when your apps have been ported.

[JamieBeeston]

size13 said:

Unfortunately we are the host and admin!

size13 said:

Although it is fully updated, I'll look into you suggestions thanks.

PHP 4.3.10 is just out (1 week) specifically for protection against this sort of intrusion.

We've tested on our boxes and rolled out the upgrade to all of our Managed Clients.

Are you running any bulliten boards? webmail? gallery software? anything php database/filebase driven?

www.hardened-php.net/advisories/012004.txt

For more info.

[size13]

The server only runs php db stuff with no galleries etc. I'm going to upgrade PHP tonight. (shutting the door after ...)

Thanks for the info, I've have a read.

[JamieBeeston]

size13 said:

The server only runs php db stuff with no galleries etc. I'm going to upgrade PHP tonight. (shutting the door after ...) Thanks for the info, I've have a read.

Hmm, Older apache? 1.3.33 is latest 1.3x branch.

Sendmail is another possibility, but unlikely given only webpages were exploited.

www.chkrootkit.org

Download, untar and execute the RookCheck script.

My money is on php still.

[TheExcession]

1st. If possible isolate the machine completely fro mthe netowrk
2nd. Change ALL your account passwords immeditaely and consider isolating the machine.
3nd. Check no other user accounts have been created.

Quick fix - you do have backups don't you?

If you are going to do a restore to get going again - consider doing a full backup of the system first so you can analyse what happened on anothe machine later.

At the very least take copies of the var/log and all apapche/php logs.

Now....

I don't know that flavour of Linux but what version of Apache/Php etc are you running? - check for patches on these.

Also do you have any firewall logs that might indicate when and where from the attack came.

Be very carefull not to destroy evidence that will help you solve the problem.

Check out the times all the files were modded and go looking in the logs around that time.

Check out whether any of the known user accounts we logged in at those times.

Was the html page content modified to anything interesting/informative?

It's a bugger when this happens - in our house the machine would get a full format of all disks and be built again from the bottom up but you may be happy that only html files have been compromised and all your binaries are safe.

Unfortunately you have to take it on the chin - it happens - the key is to remain cool, calm and collected.

Hopefully you haven't got a database of credit card numbers for an ecommerce site on there?

Don't rush into a course of action without thinking everything through.

Best of luck getting it fixed.
Ex

>> Edited by TheExcession on Tuesday 21st December 16:07

[size13]

Apache 2.0.51, but I'm guessing on PHP too

I'm just going to go through the access logs - do you know if it will show anything?

My main problem is that there's only me - it's difficult finding the time to keep on top of all the updates and alerts etc.

[size13]

TheExcession said:

1st. If possible isolate the machine completely fro mthe netowrk 2nd. Change ALL your account passwords immeditaely and consider isolating the machine. 3nd. Check no other user accounts have been created. Quick fix - you do have backups don't you? If you are going to do a restore to get going again - consider doing a full backup of the system first so you can analyse what happened on anothe machine later. At the very least take copies of the var/log and all apapche/php logs. Now.... I don't know that flavour of Linux but what version of Apache/Php etc are you running? - check for patches on these. Also do you have any firewall logs that might indicate when and where from the attack came. Be very carefull not to destroy evidence that will help you solve the problem. Check out the times all the files were modded and go looking in the looks around that time. Check out whether any of the known user accounts we logged in at those times. Out of interest was the html page content modified to anything interesting? It's a bugger when this happens - in our house the machine would get a full format of all disks and be built again from the bottom up but you may be happy that only html files have been compromised and all your binaries are safe. Unfortunately you have to take it on the chin - it happens - the key is to remain cool, calm and collected. Don't rush int oa course of action without thinking everything through. Best of luck getting it fixed. Ex

Do all this - it's now back up and running, I'm just going through everything - logs etc that I copied off it.

The hacked pages said "Owned by: XSupr3mo Segmentation Fault Group" and that was it - just text.

[JamieBeeston]

size13 said:

Apache 2.0.51, but I'm guessing on PHP too I'm just going to go through the access logs - do you know if it will show anything? My main problem is that there's only me - it's difficult finding the time to keep on top of all the updates and alerts etc.

It may well show a lot of gibberish suffixed to one of your PHP files.

If you can, work out the time the html files were modified, then check the logs 20 mins before that time to say 10 mins after.

Depends on what level of logging you have innstalled,

Its possible to get very forensic if you have the time and knowledge and inclination, depends how quickly you need the service back live.

Are you Defo only 25 and 80 are public facing? (no 53 for DNS, 22 for SSH, 23 for Telnet, 20/21 for FTP etc!)

if so, thats good.. limits the scope of the investigation.

Also, check the ownership of the files modified, see if they are owned by nobody/httpd/apache some process who shouldnt own them.

If you really want to look more closely, drop me a mail, and I will happily look around the box if you can sort accesss.. or.. just chalk it down to experience and proceed with whatever company policy you have.

[TheExcession]

size13 said:

Do all this - it's now back up and running, I'm just going through everything - logs etc that I copied off it. The hacked pages said "Owned by: XSupr3mo Segmentation Fault Group" and that was it - just text.

Sorry I was a bit late getting here! Still at least you've pulled the log files.

Let us know if you find anything interesting.

best
Ex

[JamieBeeston]

size13 said:

The hacked pages said "Owned by: XSupr3mo Segmentation Fault Group" and that was it - just text.

www.google.co.uk/search?q=XSupr3mo+Segmentation+Fault+Group&hl=en&lr=&safe=off&filter=0

The upside of this is that i am 99% sure this wasnt a targetted attack, most likely just a broadranging netblock scan seaching for vulverable versions.

You shoud turn off Apache Server Tokens / Server Signature / Extended Status.. this will help hide version numbers.

Showing versions to the public is asking for trouble.

Do the same with SSH / Sendmail (either by recompiling them with edited versions, or get a firewall that supports FixUp)

[size13]

JamieBeeston said:

size13 said: The hacked pages said "Owned by: XSupr3mo Segmentation Fault Group" and that was it - just text. www.google.co.uk/search?q=XSupr3mo+Segmentation+Fault+Group&hl=en&lr=&safe=off&filter=0 The upside of this is that i am 99% sure this wasnt a targetted attack, most likely just a broadranging netblock scan seaching for vulverable versions.

I did the very same thing this morning!

[size13]

I've got these from the log file, 4 minutes after the time of the files placed on the server

213.219.122.11 - - [21/Dec/2004:04:17:34 +0000] "GET / HTTP/1.0" 200 44 "-" "Wget/1.9.1"
213.219.122.11 - - [21/Dec/2004:04:17:35 +0000] "HEAD / HTTP/1.0" 200 - "-" "Sprint (safemode.org)"
213.219.122.11 - - [21/Dec/2004:04:17:36 +0000] "HEAD / HTTP/1.0" 200 - "-" "Sprint (safemode.org)"

I assume this is somebody taking a snapshot of the site after it was cracked

>> Edited by size13 on Tuesday 21st December 16:33

[TheExcession]

SamSpade WhoIs: said:

12/21/04 16:29:15 IP block 213.219.122.11 Trying 213.219.122.11 at ARIN Trying 213.219.122 at ARIN OrgName: RIPE Network Coordination Centre OrgID: RIPE Address: P.O. Box 10096 City: Amsterdam StateProv: PostalCode: 1001EB Country: NL ReferralServer: whois://whois.ripe.net:43 NetRange: 213.0.0.0 - 213.255.255.255 CIDR: 213.0.0.0/8 NetName: RIPE-213 NetHandle: NET-213-0-0-0-1 Parent: NetType: Allocated to RIPE NCC NameServer: NS-PRI.RIPE.NET NameServer: NS3.NIC.FR NameServer: SUNIC.SUNET.SE NameServer: AUTH00.NS.UU.NET NameServer: SEC1.APNIC.NET NameServer: SEC3.APNIC.NET NameServer: TINNIE.ARIN.NET Comment: These addresses have been further assigned to users in Comment: the RIPE NCC region. Contact information can be found in Comment: the RIPE database at www.ripe.net/whois RegDate: Updated: 2004-03-16 # ARIN WHOIS database, last updated 2004-12-20 19:10 # Enter ? for additional hints on searching ARIN's WHOIS database.

Following the who is referral:
www.ripe.net/whois?form_type=simple&full_query_string=&searchtext=213.219.122.11&do_search=Search

there is an email adddress there to report abuse.

Even more interesting is

http://213.219.122.11/


plot thickens...

Anything just before the pages were modified, does it look like the logs have had any offending entries removed?

best
Ex

>> Edited by TheExcession on Tuesday 21st December 16:36

[size13]

also got

80.255.42.105 - - [21/Dec/2004:04:44:15 +0000] "GET / HTTP/1.1" 200 44 "www.zone-h.org/defacements/onhold" "Mozilla/5.0 (Windows; U; Windows NT 5.1; pt-BR; rv:1.7.5) Gecko/20041118 Firefox/1.0"

lookes like another defacement snapshot

Nothing else shows in the logs except search engine robots

[JamieBeeston]

Most likely the kiddies are 'in a competition' and ar simply submiting 'proof' of the sites they have defaced, the 'security' site then catalogues it for reference.

Wouldnt surprise me if the site was run by associates of the hackers, but I think the link ends there..

The logs just show copies of the page being grabbed, most likely from a Script at the other end, called by the script that defaced you.

Scripts feeding scripts feeding scripts.

[TheExcession]

size13 said:

also got 80.255.42.105 - - [21/Dec/2004:04:44:15 +0000] "GET / HTTP/1.1" 200 44 "www.zone-h.org/defacements/onhold" "Mozilla/5.0 (Windows; U; Windows NT 5.1; pt-BR; rv:1.7.5) Gecko/20041118 Firefox/1.0" lookes like another defacement snapshot Nothing else shows in the logs except search engine robots

that address looks a bit more likely - big message about reporting abuse:
www.ripe.net/whois?form_type=simple&full_query_string=&searchtext=80.255.42.105&do_search=Search

Might be worth dropping thema line and see if they have anything in their logs - In my experience ISPs are usually cooperative in these situations.

Might also be worth a call to your own ISP and see if they can spot anything/help.

Personaly I'd try a few emails t othe bove people as in the USA they take this stuff very seriuosly now.

But as Jamie has popinted out - it could just be a script kiddie that finally hit your IP Address.

You just need to get your system patched up.

best
Ex