Have we been hacked?
Discussion
Been having some very odd problems with our web/database server over the past day or so.
Yesterday morning I got in to find none of our websites that used the MySQL database were working. Further investigation revealed that the root password had been changed. Couldn't do anything to change it back and when I rebooted the server it wouldn't come back up. After calling the hosting company to get it back up again I reinstalled MySQL and restored the data tables, and all was well and good, though still very worrying.
Today, while managing one of the databases, my colleage noticed a new UDF entry on all the databases that we're sure wasn't there before. We have the same version of the software running on some local machines here in the office and none of these have the UDF, or the DLL it calls - reverse.dll.
I've trawled through the MySQL website and googled for reverse.dll but I can't find anything about it. But I have this horrible suspicion we've been hacked and it's some kind of exploit. The dll is sitting in c:mysql in and C:windows
The server is running Windows Server 2003 and up to date with all security patches. MySQL is version 4.1.7.
Help!
Yesterday morning I got in to find none of our websites that used the MySQL database were working. Further investigation revealed that the root password had been changed. Couldn't do anything to change it back and when I rebooted the server it wouldn't come back up. After calling the hosting company to get it back up again I reinstalled MySQL and restored the data tables, and all was well and good, though still very worrying.
Today, while managing one of the databases, my colleage noticed a new UDF entry on all the databases that we're sure wasn't there before. We have the same version of the software running on some local machines here in the office and none of these have the UDF, or the DLL it calls - reverse.dll.
I've trawled through the MySQL website and googled for reverse.dll but I can't find anything about it. But I have this horrible suspicion we've been hacked and it's some kind of exploit. The dll is sitting in c:mysql in and C:windows
The server is running Windows Server 2003 and up to date with all security patches. MySQL is version 4.1.7.
Help!
Reverse.dll is listed here - www.dynamiclink.nl/htmfiles/rframes/dll-r03.htm
as Reverse Video Detection for Asian OCR. Not sure how that works but it sounds most odd.
as Reverse Video Detection for Asian OCR. Not sure how that works but it sounds most odd.
The file on the server is 40kb in size. What is worrying me is that none of the installations of XP/MySQL we have locally have this file and it is called by a UDF in MySQL, the exact syntax is:
CREATE FUNCTION cbShell RETURNS INTEGER SONAME "reverse.dll";
This, alongside the yesterday's problems, makes me very, very suspicious.
Oh, and the file was created yesterday evening long after we'd finished reinstalling everything...
>> Edited by judas on Tuesday 14th December 15:54
CREATE FUNCTION cbShell RETURNS INTEGER SONAME "reverse.dll";
This, alongside the yesterday's problems, makes me very, very suspicious.
Oh, and the file was created yesterday evening long after we'd finished reinstalling everything...
>> Edited by judas on Tuesday 14th December 15:54
Well, to be on the safe side I've deleted the dlls and removed the reference to them in the d/b's master function table. I pulled a copy of the dll off to study and there are internal references to the .NET framework in the error messages. 
Bah! I hate computers.
Update: Dependency walker shows it calls a whole heap of other dlls to open up the shell. Sounds like a trojan to me, considering there's real information about it anywhere.
>> Edited by judas on Tuesday 14th December 16:17
Bah! I hate computers.
Update: Dependency walker shows it calls a whole heap of other dlls to open up the shell. Sounds like a trojan to me, considering there's real information about it anywhere.
>> Edited by judas on Tuesday 14th December 16:17
When you restored your DB, did you ensure you changed *ALL* passwords (MySQL/System Accounts/Administrator etc)
Also, you should ensure 3306 (MySQL) is added to your Firewall config (assuming you are firewalled! I literally dont know of ONE client running an unfirewalled 2003 who hasnt been rooted!)
Defo sounds like someone has gotten in, but not well enough to compromise the system entirely. You should look at a Decent Hardware firewall (Checkpoint if funds permit), speak to your hosting provider for more info.
Feel free to mail via profile if you need any more info.
J
Also, you should ensure 3306 (MySQL) is added to your Firewall config (assuming you are firewalled! I literally dont know of ONE client running an unfirewalled 2003 who hasnt been rooted!)
Defo sounds like someone has gotten in, but not well enough to compromise the system entirely. You should look at a Decent Hardware firewall (Checkpoint if funds permit), speak to your hosting provider for more info.
Feel free to mail via profile if you need any more info.
J
judas said:
Thanks Jamie. The server is behind a firewall but I think we need to look at the configuration again in depth 
Time to check all the passwords again. Oh joy.
If so, they lock it down totally, ONLY allowing to the public those ports EXPLICITLY needed, ie 80
RemoteDesktop/MySQL/FTP lock down to your office Static IP and that of Trusted Home Workers.
Even then its still a gamble, A decent Firewall (chckpoint) will inspect each packet, and filter out any naughtyness it identifies before it even hits your server (Codered/nimbda/SQL Injection Exploits/ASP Exploits/SSL Exploits etc) but doesnt come cheap.
You should also consider a VPN only access policy for non-public facing services.
And dont trust a Windows Software based firewall as far as you can spit IMHO!
Enjoy.
J
Jamie
Many moons ago we found one of our systems had been accessed in an unauthorised manner.
What a bastard that was seeing as the systems were in Perth and we're all based in London.
We had to rebuild 8 linux servers and two NT boxes remotely get oiperators to put in the CDs for use whilst we used console access via an ISDN line.
In a situation like this the only option is to start with a reformated drive and build the whole lot from the ground up or backups if you know they are safe - you just cannot be certain what has been done.
We tracked our intrusion down to a hole in the firewall.
Once you get everything up and running again try http://grc.com for a security scan check - use the shields up test - he's added a loadof new tests now by the looks of things.
Also bear in mind that now adays I would not even plug in a network cable to a PC until it has had all the patches installed from a safe CD.
Grim news I know hope, you get it sorted.
best
Ex
What a bastard that was seeing as the systems were in Perth and we're all based in London.
We had to rebuild 8 linux servers and two NT boxes remotely get oiperators to put in the CDs for use whilst we used console access via an ISDN line.
In a situation like this the only option is to start with a reformated drive and build the whole lot from the ground up or backups if you know they are safe - you just cannot be certain what has been done.
We tracked our intrusion down to a hole in the firewall.
Once you get everything up and running again try http://grc.com for a security scan check - use the shields up test - he's added a loadof new tests now by the looks of things.
Also bear in mind that now adays I would not even plug in a network cable to a PC until it has had all the patches installed from a safe CD.
Grim news I know hope, you get it sorted.
best
Ex
TheExcession said:
Also bear in mind that now adays I would not even plug in a network cable to a PC until it has had all the patches installed from a safe CD.
35 seconds is how long a 2000 Server typically lasts on first boot before it gets RPC'd into a reboot. After the first 2-3 times we concluded that it was just not viable, and so locally downloaded the Windows Update Hotfixes and apply by hand.
Not Fun out there for Windows or Nix Vanilla nowadays!
JamieBeeston said:
Also bear in mind that now adays I would not even plug in a network cable to a PC until it has had all the patches installed from a safe CD.
After the first 2-3 times we concluded that it was just not viable, and so locally downloaded the Windows Update Hotfixes and apply by hand.
Not Fun out there for Windows or Nix Vanilla nowadays!
Wouldn't it be easier to maintain a local Windows Update server (or software update server, depending on what MS are calling it this week) and have the machines get their patches from that before making them Internet accessible. Sounds like the manual method would take a while if you building the number of servers a hosting company is likely to build....
D
JamieBeeston said:
Why the
Windows boxes still install and boot up the first time with any network ports open is still beyond my comprehension. At least with most NIX distros they install and boot up pretty tight now.
best
Ex
TheExcession said:
JamieBeeston said:
Why the
At least with most NIX distros they install and boot up pretty tight now.
best
Ex
I'd imagine it's because most machines are used on a LAN and therefore the aforementioned RPC service is not shut down on startup because it's used for a lot of communications within the Windows environment.
Microsoft have reduced the number of services and ports that are open in Windows 2003 to address this issue, but it's the usual compromise between security and convenience.
I'm afraid to say that if you asked most Windows admins what RPC is and does you'd not get the correct answer.
Ironically, if I started a client or server up internally then it'd be less secure if RPC was disabled (security policies are appled using group policy when a machine first boots following it joining the AD).
If you're that concerned about security you'd think that the Windows standard automated build would deal with all that, and have a policy to block all traffic to the machine initially if that's what you want.
Also, JB mentioned Windows 2000. That product was released prior to Microsoft's security 'crack down' so it's more open that both XP and 2003.
Darren.
>> Edited by _deejay_ on Wednesday 15th December 13:12
>> Edited by _deejay_ on Wednesday 15th December 13:13
Gassing Station | Computers, Gadgets & Stuff | Top of Page | What's New | My Stuff