Any network/hardware specialists in the house?

[judas]

We've had an ongoing problem with our main file server for a while with what appears to be packet flooding. We updated the network card drivers and that seemed to fix things but it's now started doing it again and it's completely crippling our internet connection - ping times are usually somewhere near 2000ms.

The server is a Dell Poweredge 2800 running Windows Server 2003 SBE and has two Intel Pro/1000 MT cards. Switching over from one card to another doesn't help. I've checked for spyware thinking that may be the cause - nothing found. It's got NOD32 antivirus and that hasn't found anything either.

Pulling the network connection and reconnecting resets things for a half a minute or so, but then the ping times start rising again.

I'm really out of my depth on this one - any help would be appreciated!

[judas]

Definitely the server. Turn off all other PCs - problem still there. Pull server network cable out - bingo! Ping times drop immediately to normal levels.

Bear in mind that I'm not a network/systems admin - I'm a web monkey who's been landed with the job because "you know about computers, don't you?"

I did get a mate in who is a network admin to have a look at it. He's the one that surmised that it was the network card or driver and updated the driver and fiddled with some settings, which (for a while at least) made the problem go away.

I've tried all kinds of traffic analysers, but the main problem is I haven't got a clue what the results mean!

Edited by judas on Thursday 27th March 19:28

[judas]

league67 said:

More info needed, I'd start with sysinternals and proc explorer/monitor. My 2p is going to be that you have virus on the server. Never had a problem with Dell where is randomly chucking out packets because of network driver problem.

Just ran a virus scan over the server - nothing found. But I realise that doesn't mean there isn't one there...

[judas]

Thanks for the suggestions chaps!

I've downloaded the sysinternals suite and have it standing by for when it kicks off again.

On the auto-negotiate front - how/where do I do this? I've checked all the tabs/settings in the network card properties panel and in the hardware explorer but there's nothing about this?

[judas]

HRG: Rob - please stop changing your username! Didn't realise it was you!

Anyhoo - Switch is a Netgear JGS516, Firewall/Router is a Netgear FVS318.

fade2grey: Still can't find any auto-negotiate settings on the server network card properties panel.

I've spent most of the afternoon going cross-eyed watching TCPView. I've turned off NetBios on the network card and blocked ports 135, 137-139 as well after seeing some unusual EPMAP activity.

Network's been mostly behaving itself today so it's not been possible to pin down anything specific yet.

The struggle will continue on Monday...

ETA: Anyone know how to get Sysinternals' RootkitRevealer working? The instructions are sparse and decidedly cryptic

Edited by judas on Friday 28th March 17:35

[judas]

theboss said:

You shouldn't struggle to find speed/duplex settings with an Intel NIC - try looking in the device manager, finding the NIC and viewing its properties, there's usually an advanced tab that has a load of settings you can alter. Both the NIC and the switch port should be set to auto negotiate.

ETA - sorry just noticed fade2grey said almost exactly the same thing a few posts ago!!

Believe me, I have looked - there's nothing. The only setting on the Advanced tab is 'Windows Firewall - Protect my computer by limiting or preventing access to this computer from the Internet.' When I click the settings button I get the following popup message:

Windows said:

Windows Firewall cannot run because another program or service is running that might use the network address translation component (Ipnat.sys).

We're running a hardware firewall, so lack of a Windows firewall is not a problem, thought whether this is at all related to our ongoing issue is beyond me...

theboss said:

You mentioned two NICs which I presume are embedded on the 2800, are they both connected to the switch and if so how have they been configured? Are you using any NIC teaming utilities?

Only one is connected. I thought it may have been a hardware fault, so I disconnected from one, disabled it, and reconnected to the second port.

theboss said:

Was the server built from scratch or cloned from another machine?

Built from scratch.

[judas]

Oh, what fun! Our web server fell over as well to add to the joy

Anyhoo - I've just reinstalled the network card drivers to get all the extra options. Speed and duplex are set to Automatic rather than Auto Negotiate. I'll leave it at that for now as I'm getting grief from the MD for the server being up and down like a tart's knickers.

[judas]

Update: those who suggested a virus, I think you may be right. After another session of watching TCPView I noticed a lot of inetinfo processes opening to external IP addresses on port 25 coinciding with high ping times - stopping IIS immediately dropped the ping times back to normal. A whois lookup on the terminating address shows random foreign servers that I can guarantee no one in the office would be sending mail to. Also, there seems to be a thread popping up quite frequently connecting to emailsrvr.com.

My guess now is that we have some kind of rootkit spambot buried somewhere in IIS or Exchange.

Arse

I could really do with some help on getting RootkitRevealer working!

Edited by judas on Monday 31st March 12:35

[judas]

Update on the server problem: I think it's safe to rule out hardware/driver problems now. I've identified a pattern of behaviour but I'll be damned if I can find the cause. Whatever's at the root of this problem is something to do with Exchange/IIS. I've been monitoring ping times against activity in TCPView and every so often the inetinfo process opens up a connection to port 25 of an external server, almost always to some random server in another country. At this point ping times go through the roof as it presumably starts flooding the connection with packets. If I kill this connection the ping times drop immediately, but it does try to reconnect again straight away until I kill it again and then it goes quiet for a bit.

I'm no expert in malware, but I'm guessing that as it's connecting on port 25 it's some kind of spambot. However, I've run four different virus scanners, SpyBot, Hijack This and a couple of rootkit detectors and can't find anything. Short of a server rebuild, which I really don't want to do, I'm stumped...

[judas]

While it's possible it's one of the desktop machines rather than the server, I think it's unlikely:
1) TCPView is showing the connection originating from the server's IP address (though it could be just traffic being routed through the server from another machine).
2) The simple pull-out-the-network-cable test to see if traffic drops when any particular machine is disconnected only worked when I pulled the server's network cable.

[judas]

HRG said:

Do you have an upstream ISP providing anti-spam? You can set your firewall up to only allow port 25 from the ISP's mail drop servers and it will stop the problem in it's tracks. If you look at the exchange logs you'll see it denying relays at the same times the problem occurs.

If what you say is right the problem is external and no amount of rebuilding will fix it!

The problem is with outgoing traffic, not incoming spam (that's a whole different bucket of squid).
When the problem occurs two inetinfo process start sending traffic, using what look like random port numbers this side (usually in the range 30000 to 40000) to port 25 (smtp) on the remote server.

[judas]

HRG said:

Do you send NDR's?

Que?

[judas]

Currently, no. If I turn it on then I should see my inbox swamped with NDRs if we have a spambot on the server, yes?

[judas]

Are NDRs normally sent straight away or batched for block sending? Given the volume of spam we get (a lot but still only in the hundreds per hour) unless NDRs are batched I can't see how sending them could cause the kind of sustained network slowdown I'm seeing

[judas]

In c:\Program Files\Exchsrvr\Mailroot\vsi 1\Queue\ there are currently a grand total of 9 items. I'll keep an eye things. I'm using the Intelligent Message Filter to move suspected spam to a separate folder and then have a quick look through it using IMFCompanion before deleting it - so all the spam coming in should not necessarily be sending out NDRs. Whether that makes any difference remains to be seen.

ETA: in a 19 hour window there have been 2400 or so emails flagged as spam and dumped in the spam trap. That should give you an idea of volume.

Edited by judas on Tuesday 8th April 11:49

[judas]

league67 said:

Proc explorer as mentioned earlier is your friend. I doubt that exchange is responsible for your outgoing traffic. Can you check your temp folder and see if there are any BN?.TMP files there (where ? = number from 0-9.)

I've been using process explorer and so far there's nothing I can see that's out of place - but as I've said, I'm not a server admin, just the poor sap who knows a bit more than everyone else about computers so gets landed with the job

No BN*.tmp files to be found. What's the deal with them?

[judas]

Ah - right. This problems been ongoing for a while now though

Proc Explorer screenshot during normal network activity:

[judas]

No, that was my first thought as we were having some problems getting email through to some customers. Turned out that a now-redundant secondary MX record was pointing to a mailserver that had been blacklisted.