Any network/hardware specialists in the house?

Any network/hardware specialists in the house?

Author
Discussion

fade2grey

704 posts

249 months

Monday 31st March 2008
quotequote all
I'd have thought a root kit unlikely unless you are particularly badly configured.

Run the usual suspects through first - spyware, av etc - full system scans so might be worth doing over night. I've no experince of NOD32 - I know some gamers recommend it because it's light weight - I used to do a lot of AV stuff, & light weight is not what I'd want on an IIS\Exchange box.. Might be worth looking at Mcafee or trend, even if it's just for trials.

mondeoman

11,430 posts

267 months

Monday 31st March 2008
quotequote all
I reckon you shoul ddownload and run Process Explorer - it'll show youexactly what processes are running and you cna thne identify if you have a virus problem.

Might also be worth checking the registry for programms that are run at startup - could be a couple of nasties in there too.

I say this cos I've had similar problems before on a server....

judas

Original Poster:

5,992 posts

260 months

Tuesday 8th April 2008
quotequote all
Update on the server problem: I think it's safe to rule out hardware/driver problems now. I've identified a pattern of behaviour but I'll be damned if I can find the cause. Whatever's at the root of this problem is something to do with Exchange/IIS. I've been monitoring ping times against activity in TCPView and every so often the inetinfo process opens up a connection to port 25 of an external server, almost always to some random server in another country. At this point ping times go through the roof as it presumably starts flooding the connection with packets. If I kill this connection the ping times drop immediately, but it does try to reconnect again straight away until I kill it again and then it goes quiet for a bit.

I'm no expert in malware, but I'm guessing that as it's connecting on port 25 it's some kind of spambot. However, I've run four different virus scanners, SpyBot, Hijack This and a couple of rootkit detectors and can't find anything. Short of a server rebuild, which I really don't want to do, I'm stumped... frown

Plotloss

67,280 posts

271 months

Tuesday 8th April 2008
quotequote all
25 is an email port usually.

Sounds like you've got some sort of SMTP malware on one of the machines would be my first guess I reckon.

HRG

72,857 posts

240 months

Tuesday 8th April 2008
quotequote all
idea

Do you have an upstream ISP providing anti-spam? You can set your firewall up to only allow port 25 from the ISP's mail drop servers and it will stop the problem in it's tracks. If you look at the exchange logs you'll see it denying relays at the same times the problem occurs.

If what you say is right the problem is external and no amount of rebuilding will fix it!

judas

Original Poster:

5,992 posts

260 months

Tuesday 8th April 2008
quotequote all
While it's possible it's one of the desktop machines rather than the server, I think it's unlikely:
1) TCPView is showing the connection originating from the server's IP address (though it could be just traffic being routed through the server from another machine).
2) The simple pull-out-the-network-cable test to see if traffic drops when any particular machine is disconnected only worked when I pulled the server's network cable.

judas

Original Poster:

5,992 posts

260 months

Tuesday 8th April 2008
quotequote all
HRG said:
idea

Do you have an upstream ISP providing anti-spam? You can set your firewall up to only allow port 25 from the ISP's mail drop servers and it will stop the problem in it's tracks. If you look at the exchange logs you'll see it denying relays at the same times the problem occurs.

If what you say is right the problem is external and no amount of rebuilding will fix it!
The problem is with outgoing traffic, not incoming spam (that's a whole different bucket of squid).
When the problem occurs two inetinfo process start sending traffic, using what look like random port numbers this side (usually in the range 30000 to 40000) to port 25 (smtp) on the remote server.

HRG

72,857 posts

240 months

Tuesday 8th April 2008
quotequote all
judas said:
HRG said:
idea

Do you have an upstream ISP providing anti-spam? You can set your firewall up to only allow port 25 from the ISP's mail drop servers and it will stop the problem in it's tracks. If you look at the exchange logs you'll see it denying relays at the same times the problem occurs.

If what you say is right the problem is external and no amount of rebuilding will fix it!
The problem is with outgoing traffic, not incoming spam (that's a whole different bucket of squid).
When the problem occurs two inetinfo process start sending traffic, using what look like random port numbers this side (usually in the range 30000 to 40000) to port 25 (smtp) on the remote server.
Random originating port numbers is normal. Do you send NDR's?

judas

Original Poster:

5,992 posts

260 months

Tuesday 8th April 2008
quotequote all
HRG said:
Do you send NDR's?
Que?

HRG

72,857 posts

240 months

Tuesday 8th April 2008
quotequote all
judas said:
HRG said:
Do you send NDR's?
Que?
Non Delivery Reports. It could be your server responding to spam attempts by sending NDR's back to the originating server.

Edited by HRG on Tuesday 8th April 10:26

judas

Original Poster:

5,992 posts

260 months

Tuesday 8th April 2008
quotequote all
Currently, no. If I turn it on then I should see my inbox swamped with NDRs if we have a spambot on the server, yes?

HRG

72,857 posts

240 months

Tuesday 8th April 2008
quotequote all
Nope, the NDR's will go back to the originating server. There's no need to have them turned on these days, in fact most people find them annoying as they can act as a reverse DDOS attack biggrin

If you ramp up the logging level on all the SMTP components on your Exchange server you might get a clue from the event viewer.


judas

Original Poster:

5,992 posts

260 months

Tuesday 8th April 2008
quotequote all
Are NDRs normally sent straight away or batched for block sending? Given the volume of spam we get (a lot but still only in the hundreds per hour) unless NDRs are batched I can't see how sending them could cause the kind of sustained network slowdown I'm seeing

HRG

72,857 posts

240 months

Tuesday 8th April 2008
quotequote all
Without looking in the logs it's hard to say. I've seen cirka 19k NDR's sat in an outbound queue before! If you've got them turned off then it's not likely to be NDR's.

I'd definitely turn up the Diagnostic Logging level on your Exchange server's MSExchange Transport tab to see what's occurring.

judas

Original Poster:

5,992 posts

260 months

Tuesday 8th April 2008
quotequote all
In c:\Program Files\Exchsrvr\Mailroot\vsi 1\Queue\ there are currently a grand total of 9 items. I'll keep an eye things. I'm using the Intelligent Message Filter to move suspected spam to a separate folder and then have a quick look through it using IMFCompanion before deleting it - so all the spam coming in should not necessarily be sending out NDRs. Whether that makes any difference remains to be seen.

ETA: in a 19 hour window there have been 2400 or so emails flagged as spam and dumped in the spam trap. That should give you an idea of volume.

Edited by judas on Tuesday 8th April 11:49

league67

1,878 posts

204 months

Tuesday 8th April 2008
quotequote all
Proc explorer as mentioned earlier is your friend. I doubt that exchange is responsible for your outgoing traffic. Can you check your temp folder and see if there are any BN?.TMP files there (where ? = number from 0-9.)


judas

Original Poster:

5,992 posts

260 months

Tuesday 8th April 2008
quotequote all
league67 said:
Proc explorer as mentioned earlier is your friend. I doubt that exchange is responsible for your outgoing traffic. Can you check your temp folder and see if there are any BN?.TMP files there (where ? = number from 0-9.)
I've been using process explorer and so far there's nothing I can see that's out of place - but as I've said, I'm not a server admin, just the poor sap who knows a bit more than everyone else about computers so gets landed with the job rolleyes

No BN*.tmp files to be found. What's the deal with them?

league67

1,878 posts

204 months

Tuesday 8th April 2008
quotequote all
BN?.TMP was the origin of last couple of spyware bots churning out colossal amounts of emails to random servers. With yours thats obviously not the case. Can you run proc explorer, capture the screen and post it?

judas

Original Poster:

5,992 posts

260 months

Tuesday 8th April 2008
quotequote all
Ah - right. This problems been ongoing for a while now though frown

Proc Explorer screenshot during normal network activity:


league67

1,878 posts

204 months

Tuesday 8th April 2008
quotequote all
Can't see anything obviously wrong. Keep this and take snapshot when you start getting mails out.