Interesting firewall or hardware issue - Linux and iptables
Discussion
I have a really odd issue with my firewall here at home. I believe this is a hardware issue.
Uploads from Linux boxes on the LAN are slow, but from Windows boxes they are fine.
If I run a Linux VM on a Windows host, the upload speeds from that Linux box are fine.
Uploads via a site to site VPN are fine both on Linux and Windows.
Download speeds are fine on all boxes.
Fiddling with different MTU sizes does not improve the upload speeds.
However - swapping out the firewall hardware fixes things.
I have many years of experience with this stuff, but I've not encountered anything like this.
The only thing I can think of, is a problem with the Linux Realtek drivers on the firewall.
Uploads from Linux boxes on the LAN are slow, but from Windows boxes they are fine.
If I run a Linux VM on a Windows host, the upload speeds from that Linux box are fine.
Uploads via a site to site VPN are fine both on Linux and Windows.
Download speeds are fine on all boxes.
Fiddling with different MTU sizes does not improve the upload speeds.
However - swapping out the firewall hardware fixes things.
I have many years of experience with this stuff, but I've not encountered anything like this.
The only thing I can think of, is a problem with the Linux Realtek drivers on the firewall.
I'm guessing your downloads are tcp
I would look at trying a udp speed test, particularly if you vpn is udp and unaffected.
There is a broad array of tcp kernel parameters that default to behaviour that is different enough to windows for there to be significant performance differences under the right circumstances.
My best guess would be tcp_congestion_control
I would look at trying a udp speed test, particularly if you vpn is udp and unaffected.
There is a broad array of tcp kernel parameters that default to behaviour that is different enough to windows for there to be significant performance differences under the right circumstances.
My best guess would be tcp_congestion_control
e-honda said:
I'm guessing your downloads are tcp
I would look at trying a udp speed test, particularly if you vpn is udp and unaffected.
There is a broad array of tcp kernel parameters that default to behaviour that is different enough to windows for there to be significant performance differences under the right circumstances.
My best guess would be tcp_congestion_control
I've been through more TCP parameters than I care to remember - even restoring to defaults.I would look at trying a udp speed test, particularly if you vpn is udp and unaffected.
There is a broad array of tcp kernel parameters that default to behaviour that is different enough to windows for there to be significant performance differences under the right circumstances.
My best guess would be tcp_congestion_control
The thing is - on other hardware the problem goes away...
It's also just uploads that are affected - downloads are absolutely fine.
This is a really, really strange issue - the only thing I can think of, is that the Realtek NIC drivers are doing something odd (and given the reputation of the Realtek NIC hardware on Linux this wouldn't surprise me). Sadly, the NICs are on board so swapping them out is not an option (it's a small form factor machine).
xeny said:
Have you tried Linux VMs both with NAT'd and bridged connections? Wondering if NAT is good and bridged will exhibit the problem.
That's an interesting idea - I'll try the bridged connection in a VM tomorrow.I've reached the stage where I'm happy to swap out the hardware - but curiosity is getting the better of me and I want to know what it is...
This is where you need to get tcpdump/wireshark out, and look at the TCP streams in detail. Presumably there will be some difference in the handshake parameters. As an example, if SACK isn’t used, that can make a big difference to recovery from lost packets. Window sizing is another etc.
duff-man said:
Anything show in a packet capture?
outnumbered said:
This is where you need to get tcpdump/wireshark out, and look at the TCP streams in detail. Presumably there will be some difference in the handshake parameters. As an example, if SACK isn’t used, that can make a big difference to recovery from lost packets. Window sizing is another etc.
Replying to you both at once...I've not tried packet capture, and I'm not sure if I want to put that much effort into it!
@outnumbered makes a couple of interesting points. I suspect my (in depth) knowledge of networking is not quite good enough to investigate down to this level.
eltawater said:
Have you considered bunging a cheap usb Wifi dongle onto one of the linux boxes to eliminate the onboard nic / cabling from the equation?
I have a USB to Ethernet dongle which I could try, along with a Wireless dongle.eltawater said:
And when you swapped out the firewall hardware and all is fine, what is the "firewall hardware" ?
Linux distro is vanilla Debian, current version is 10.11, running kernel: Debian 4.19.194-3 (2021-07-18) x86_64 GNU/LinuxThe hardware that works:
Intel Celeron N2940 Quad-core Small Computer with Windows/Linux, 4x Intel Gigabit Ethernet for Network Firewall Gateway Router, 4G RAM 64G SSD/USB 3.0/2.4G WiFi/HDMI/VGA
The hardware that DOESN'T work:
The current firewall hardware is a FUJITSU D2963-S2 (2GB RAM)
cpu:
AMD Athlon(tm) Neo X2 Dual Core Processor L325, 1500 MHz
AMD Athlon(tm) Neo X2 Dual Core Processor L325, 1500 MHz
network:
enp2s0 Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
enp3s0 Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
bridge:
AMD K8 [Athlon64/Opteron] Miscellaneous Control
AMD K8 [Athlon64/Opteron] Address Map
ATI RS690/RS740 PCI to PCI Bridge (Internal gfx)
ATI SB600 PCI to LPC Bridge
ATI RS690 PCI to PCI Bridge (PCI Express Port 3)
ATI RS690 Host Bridge
AMD K8 [Athlon64/Opteron] DRAM Controller
ATI RS690 PCI to PCI Bridge (PCI Express Port 2)
AMD K8 [Athlon64/Opteron] HyperTransport Technology Configuration
ATI SBx00 PCI to PCI Bridge
I have an update on this.
My connection was previously: Linux firewall --> Cisco 887 router --> VDSL.
The Cisco failed the other day, and I went back to using the BT Modem and the Linux PPP stack on my firewall.
And now suddenly I have full speed uploads via SFTP.
All rather bewlidering!
My connection was previously: Linux firewall --> Cisco 887 router --> VDSL.
The Cisco failed the other day, and I went back to using the BT Modem and the Linux PPP stack on my firewall.
And now suddenly I have full speed uploads via SFTP.
All rather bewlidering!
TonyRPH said:
I have an update on this.
My connection was previously: Linux firewall --> Cisco 887 router --> VDSL.
The Cisco failed the other day, and I went back to using the BT Modem and the Linux PPP stack on my firewall.
And now suddenly I have full speed uploads via SFTP.
All rather bewlidering!
Might explain everything, but it's very dependent on the configuration of the Cisco device. There are any number of features in IOS that could be messing with TCP in one way or another.My connection was previously: Linux firewall --> Cisco 887 router --> VDSL.
The Cisco failed the other day, and I went back to using the BT Modem and the Linux PPP stack on my firewall.
And now suddenly I have full speed uploads via SFTP.
All rather bewlidering!
outnumbered said:
Might explain everything, but it's very dependent on the configuration of the Cisco device. There are any number of features in IOS that could be messing with TCP in one way or another.
The Cisco config is here on PastebinTonyRPH said:
The Cisco config is here on Pastebin
OK, so that's a bare minimum config with nothing turned on. In that case, I still have no idea.Gassing Station | Computers, Gadgets & Stuff | Top of Page | What's New | My Stuff