Any network/hardware specialists in the house?

[judas]

We've had an ongoing problem with our main file server for a while with what appears to be packet flooding. We updated the network card drivers and that seemed to fix things but it's now started doing it again and it's completely crippling our internet connection - ping times are usually somewhere near 2000ms.

The server is a Dell Poweredge 2800 running Windows Server 2003 SBE and has two Intel Pro/1000 MT cards. Switching over from one card to another doesn't help. I've checked for spyware thinking that may be the cause - nothing found. It's got NOD32 antivirus and that hasn't found anything either.

Pulling the network connection and reconnecting resets things for a half a minute or so, but then the ping times start rising again.

I'm really out of my depth on this one - any help would be appreciated!

[cottonfoo]

Run something like tcpdump on the interface to see exactly what's going on. Depending on your network it could be anything from a spanning tree protocol problem to a faulty client, so you'll have to have a look.

[mondeoman]

Are you sure its the server??

I've had similar and eventually tracked it down to a virused PC.

Mind you, also had the server do it and that was eventually tracked down to some network monitoring software that was, ermmmm, having a bit of a dickie fit.

Try installing Process Explorer, AVG, Spybot, CCleaner - might help identify the problem

[judas]

Definitely the server. Turn off all other PCs - problem still there. Pull server network cable out - bingo! Ping times drop immediately to normal levels.

Bear in mind that I'm not a network/systems admin - I'm a web monkey who's been landed with the job because "you know about computers, don't you?"

I did get a mate in who is a network admin to have a look at it. He's the one that surmised that it was the network card or driver and updated the driver and fiddled with some settings, which (for a while at least) made the problem go away.

I've tried all kinds of traffic analysers, but the main problem is I haven't got a clue what the results mean!

Edited by judas on Thursday 27th March 19:28

[HRG]

Check the duplex settings on both the switch and NIC(s) are still set to autonegotiate.

[league67]

More info needed, I'd start with sysinternals and proc explorer/monitor. My 2p is going to be that you have virus on the server. Never had a problem with Dell where is randomly chucking out packets because of network driver problem.

[judas]

league67 said:

More info needed, I'd start with sysinternals and proc explorer/monitor. My 2p is going to be that you have virus on the server. Never had a problem with Dell where is randomly chucking out packets because of network driver problem.

Just ran a virus scan over the server - nothing found. But I realise that doesn't mean there isn't one there...

[buggalugs]

As said get a tcpdump or wireshark on the case, find out what the packets actually are, which direction they're going in and work back form there. Also netstat -abn will tell you what's got which sockets open on the box which might also give you a clue, you can google the port numbers and process names to try and spot the rat. Good luck!

[fade2grey]

not an easy one to troublshoot remotely but have a look at what ports are open & what process are running - you may see something strange there. Get a packet sniffer & have a look at the traffic coming out of the server when the ping time is high.. you'll be able to see what it's doing from there - if you are unfamiliar with that type of stuff it may be pretty confusing so google or possibly get someone in.

Gut feeling is a virus\spyware - but not nesc on that server, but it's hard to tell. As pointed out earlier, it could easily be auto negotiate settings on the NIC's\Switches having an argument.

[judas]

Thanks for the suggestions chaps!

I've downloaded the sysinternals suite and have it standing by for when it kicks off again.

On the auto-negotiate front - how/where do I do this? I've checked all the tabs/settings in the network card properties panel and in the hardware explorer but there's nothing about this?

[HRG]

Usually on the Configuration tab but where precisely it's hard to say. What model of switch are you plugged into?

[fade2grey]

open device manager, look at the properties of the NIC card, there's normally an advanced type tab - speed & duplex is what you are looking for. The correct setting will depend on your config, but set it the same as the switch ideally.

[judas]

HRG: Rob - please stop changing your username! Didn't realise it was you!

Anyhoo - Switch is a Netgear JGS516, Firewall/Router is a Netgear FVS318.

fade2grey: Still can't find any auto-negotiate settings on the server network card properties panel.

I've spent most of the afternoon going cross-eyed watching TCPView. I've turned off NetBios on the network card and blocked ports 135, 137-139 as well after seeing some unusual EPMAP activity.

Network's been mostly behaving itself today so it's not been possible to pin down anything specific yet.

The struggle will continue on Monday...

ETA: Anyone know how to get Sysinternals' RootkitRevealer working? The instructions are sparse and decidedly cryptic

Edited by judas on Friday 28th March 17:35

[jimmyb]

I am miles from a server expert but have you tried simply moving the server to a different switch port as I have come across issues with switch ports for no apparent reason causing issues (even after being checked over and having no errors found). Could it be faulty wiring causing the problem? I hesitate to suggest it though as wiring very rarely goes wrong unless b***ered around with.

[HRG]

judas said:

HRG: Rob - please stop changing your username! Didn't realise it was you!

Anyhoo - Switch is a Netgear JGS516, Firewall/Router is a Netgear FVS318.

fade2grey: Still can't find any auto-negotiate settings on the server network card properties panel.

I've spent most of the afternoon going cross-eyed watching TCPView. I've turned off NetBios on the network card and blocked ports 135, 137-139 as well after seeing some unusual EPMAP activity.

Network's been mostly behaving itself today so it's not been possible to pin down anything specific yet.

The struggle will continue on Monday...

ETA: Anyone know how to get Sysinternals' RootkitRevealer working? The instructions are sparse and decidedly cryptic

Edited by judas on Friday 28th March 17:35

I like to keep people guessing

Switch is auto negotiate and can switch polarity too, so can the NIC. I've seen this combination get it's knickers in a twist before, but only once. Have you got and old switch you can throw in the middle? While it's imperfect it'll rule out any autoneg issues.

[theboss]

judas said:

HRG: Rob - please stop changing your username! Didn't realise it was you!

Anyhoo - Switch is a Netgear JGS516, Firewall/Router is a Netgear FVS318.

fade2grey: Still can't find any auto-negotiate settings on the server network card properties panel.

I've spent most of the afternoon going cross-eyed watching TCPView. I've turned off NetBios on the network card and blocked ports 135, 137-139 as well after seeing some unusual EPMAP activity.

Network's been mostly behaving itself today so it's not been possible to pin down anything specific yet.

The struggle will continue on Monday...

ETA: Anyone know how to get Sysinternals' RootkitRevealer working? The instructions are sparse and decidedly cryptic

Edited by judas on Friday 28th March 17:35

You shouldn't struggle to find speed/duplex settings with an Intel NIC - try looking in the device manager, finding the NIC and viewing its properties, there's usually an advanced tab that has a load of settings you can alter. Both the NIC and the switch port should be set to auto negotiate.

ETA - sorry just noticed fade2grey said almost exactly the same thing a few posts ago!!

You mentioned two NICs which I presume are embedded on the 2800, are they both connected to the switch and if so how have they been configured? Are you using any NIC teaming utilities?

Was the server built from scratch or cloned from another machine?

Edited by theboss on Monday 31st March 00:16

[judas]

theboss said:

You shouldn't struggle to find speed/duplex settings with an Intel NIC - try looking in the device manager, finding the NIC and viewing its properties, there's usually an advanced tab that has a load of settings you can alter. Both the NIC and the switch port should be set to auto negotiate.

ETA - sorry just noticed fade2grey said almost exactly the same thing a few posts ago!!

Believe me, I have looked - there's nothing. The only setting on the Advanced tab is 'Windows Firewall - Protect my computer by limiting or preventing access to this computer from the Internet.' When I click the settings button I get the following popup message:

Windows said:

Windows Firewall cannot run because another program or service is running that might use the network address translation component (Ipnat.sys).

We're running a hardware firewall, so lack of a Windows firewall is not a problem, thought whether this is at all related to our ongoing issue is beyond me...

theboss said:

You mentioned two NICs which I presume are embedded on the 2800, are they both connected to the switch and if so how have they been configured? Are you using any NIC teaming utilities?

Only one is connected. I thought it may have been a hardware fault, so I disconnected from one, disabled it, and reconnected to the second port.

theboss said:

Was the server built from scratch or cloned from another machine?

Built from scratch.

[fade2grey]

How are you configuring your NIC? are you using Intel Proset? If so, the duplex settings will be in there.

[judas]

Oh, what fun! Our web server fell over as well to add to the joy

Anyhoo - I've just reinstalled the network card drivers to get all the extra options. Speed and duplex are set to Automatic rather than Auto Negotiate. I'll leave it at that for now as I'm getting grief from the MD for the server being up and down like a tart's knickers.

[judas]

Update: those who suggested a virus, I think you may be right. After another session of watching TCPView I noticed a lot of inetinfo processes opening to external IP addresses on port 25 coinciding with high ping times - stopping IIS immediately dropped the ping times back to normal. A whois lookup on the terminating address shows random foreign servers that I can guarantee no one in the office would be sending mail to. Also, there seems to be a thread popping up quite frequently connecting to emailsrvr.com.

My guess now is that we have some kind of rootkit spambot buried somewhere in IIS or Exchange.

Arse

I could really do with some help on getting RootkitRevealer working!

Edited by judas on Monday 31st March 12:35