LAN, subnet, router and firewall fun!
Discussion
After much head scratching and hair pulling, I've run out of ideas on an issue I'm currently having at work. 
I've had some very helpful advice here in the past so I'm hoping there are some gurus able to help me out!
To set the scene... I work in a body in white factory, the shopfloor is split into zones. Each zone is split into two discrete subnets - controls for PLCs, HMIs etc. and the archive network for backing up robot data, rivet gun data etc.
These two networks were built as physically separate networks (no idea why! and it's not something I can change so we have to deal with it as is) so we have all the controls for each zone on a LAN and the archive network for each zone on a LAN. The zones each have their own subnet, for example:-
Zone 2 Controls - 10.0.2.xx
Zone 2 Archive - 10.0.3.xx
Zone 3 Controls - 10.0.6.xx
Zone 3 Archive - 10.0.7.xx
And so on, throughout the shop. These subnets are linked on fibre ring networks (two separate ones, remember) both with the subnet 10.0.100.xx (so both have the same subnet numbering but are different physical networks).
Each subnet has a firewall/router as its gateway at 10.0.yy.1 which sits on the corresponding 10.0.100.xx network. When we have need for one zone to communicate with another, we add the routes and firewall rules to the relevant routers and all is well.
Now comes the issue. We have need to communicate between the controls and archive networks in order to interrogate the robots from the HMIs. So to achieve this we have set up some routers on two networks set up in our office:-
Office Controls - 10.0.70.xx
Office Archive - 10.0.71.xx
The new routers sit at 10.0.70.10 and 10.0.71.10 respectively and communicate via a 10.0.100.xx subnet (this subnet contains only these two routers - I just wanted to stick with the standard already set for linking routers)
So the route from 10.0.2.21 to 10.0.3.21 goes
10.0.2.1
10.0.100.92 (10.0.70.1)
10.0.70.10
10.0.100.71 (10.0.71.10)
10.0.71.1
10.0.100.35 (10.0.3.1)
10.0.3.21
Which works brilliantly. However, if I want to go to 10.0.3.91, the path gets lost at 10.0.100.92! It's a similar story throughout the shop and it doesn't seem to make sense. All the routes have a CIDR of /24 (255.255.255.0 mask) to cover the entire subnet so I have no idea why some addresses will work but others won't within the same subnets.
Has anyone had issues like this before or knows what would cause such a thing? I've been messing about with this for a couple of weeks now and just don't know where next to turn.
Apologies for the wall of text, I wanted to get as much info in as I thought was required! Thanks for reading this far if you made it!
I've had some very helpful advice here in the past so I'm hoping there are some gurus able to help me out! To set the scene... I work in a body in white factory, the shopfloor is split into zones. Each zone is split into two discrete subnets - controls for PLCs, HMIs etc. and the archive network for backing up robot data, rivet gun data etc.
These two networks were built as physically separate networks (no idea why! and it's not something I can change so we have to deal with it as is) so we have all the controls for each zone on a LAN and the archive network for each zone on a LAN. The zones each have their own subnet, for example:-
Zone 2 Controls - 10.0.2.xx
Zone 2 Archive - 10.0.3.xx
Zone 3 Controls - 10.0.6.xx
Zone 3 Archive - 10.0.7.xx
And so on, throughout the shop. These subnets are linked on fibre ring networks (two separate ones, remember) both with the subnet 10.0.100.xx (so both have the same subnet numbering but are different physical networks).
Each subnet has a firewall/router as its gateway at 10.0.yy.1 which sits on the corresponding 10.0.100.xx network. When we have need for one zone to communicate with another, we add the routes and firewall rules to the relevant routers and all is well.
Now comes the issue. We have need to communicate between the controls and archive networks in order to interrogate the robots from the HMIs. So to achieve this we have set up some routers on two networks set up in our office:-
Office Controls - 10.0.70.xx
Office Archive - 10.0.71.xx
The new routers sit at 10.0.70.10 and 10.0.71.10 respectively and communicate via a 10.0.100.xx subnet (this subnet contains only these two routers - I just wanted to stick with the standard already set for linking routers)
So the route from 10.0.2.21 to 10.0.3.21 goes
10.0.2.1
10.0.100.92 (10.0.70.1)
10.0.70.10
10.0.100.71 (10.0.71.10)
10.0.71.1
10.0.100.35 (10.0.3.1)
10.0.3.21
Which works brilliantly. However, if I want to go to 10.0.3.91, the path gets lost at 10.0.100.92! It's a similar story throughout the shop and it doesn't seem to make sense. All the routes have a CIDR of /24 (255.255.255.0 mask) to cover the entire subnet so I have no idea why some addresses will work but others won't within the same subnets.
Has anyone had issues like this before or knows what would cause such a thing? I've been messing about with this for a couple of weeks now and just don't know where next to turn.
Apologies for the wall of text, I wanted to get as much info in as I thought was required! Thanks for reading this far if you made it!
jpringle819 said:
Do you have static routes on these routers 10.0.70.10, 10.0.71.10? What devices are you using as routers as the setup looks overly complicated the route you show is only 2 hops shorted than the route from my PC to google.
Yes, all routers have static routes. Those two route to either their main gateway or the other one of the pair, with a whole list for all the zones.We're using Hirschmann Eagle firewall routers. It wouldn't be my choice as they are very complicated and overly secure for our use but some things can't be changed.
I know the routes are ridiculously long but I have to go from the zone's subnet, to the controls main ring network, to the office subnet, over the two routers we have up there and down the archive ring network to the zone. The link in our office is the only link between the two networks, hence the overly complicated routes.
ging84 said:
I don't really understand this notation, but it looks like you have some sort of natting going on
Apologies, I should've explained what I'd written - one of those things that looks obvious when you're the one writing it down! 
The IPs in parenthesis are the 'other side' of the router, so for 10.0.100.92 (10.0.70.1) - the packet enters the router through the interface at 10.0.100.92 and leaves the router from 10.0.70.1. Hopefully that makes more sense of what I was trying to explain.
There is no NAT set up at present.
you have made it annoyingly complicated by having 3 different 10.0.100.0 subnets
I understand 2 were already in place, but you could have picked litterally anything else for your new one
But what i really don't understand is why the need for 2 routers with thier own internal subnet, why could you not have just have 1 acting as a bridge ?
I understand 2 were already in place, but you could have picked litterally anything else for your new one
But what i really don't understand is why the need for 2 routers with thier own internal subnet, why could you not have just have 1 acting as a bridge ?
Start simple and work your way along.
If source is 10.0.2.21
and destination is 10.0.3.91
but path gets lost at 10.0.100.92
What does the routing table and static routes look like at the device which answers for 100.92 ? Does it know it needs to pass this packet onwards down the correct interface?
If source is 10.0.2.21
and destination is 10.0.3.91
but path gets lost at 10.0.100.92
What does the routing table and static routes look like at the device which answers for 100.92 ? Does it know it needs to pass this packet onwards down the correct interface?
WinstonWolf said:
Two physically separate fibre rings with same IP addressing scheme? 
I think you're going to have to re-address one so your routing tables are sane.
Yes, it is a bit odd I know. No option to change it I'm afraid. The saving grace is that the device addresses are unique across both networks (i.e. if they were the same network, there would be no conflicts). I think you're going to have to re-address one so your routing tables are sane.
ging84 said:
you have made it annoyingly complicated by having 3 different 10.0.100.0 subnets
I understand 2 were already in place, but you could have picked litterally anything else for your new one
But what i really don't understand is why the need for 2 routers with thier own internal subnet, why could you not have just have 1 acting as a bridge ?
Good point on the 100.0 subnets. I'll look to changing that if necessary - like you say, no point complicating things even further.I understand 2 were already in place, but you could have picked litterally anything else for your new one
But what i really don't understand is why the need for 2 routers with thier own internal subnet, why could you not have just have 1 acting as a bridge ?
Re: the subnet that only contains the two routers, I did have it set up as you suggest with one acting as a bridge (sat at 70.10 and 71.10). That setup had the same issues so a colleague suggested adding the extra router in an attempt to eliminate some of the traffic management issues caused by the weird handling of the external/internal ports by the firewall. I just wanted to describe the network exactly as it's sitting now but that will probably be returned to just the one router.
eltawater said:
Start simple and work your way along.
If source is 10.0.2.21
and destination is 10.0.3.91
but path gets lost at 10.0.100.92
What does the routing table and static routes look like at the device which answers for 100.92 ? Does it know it needs to pass this packet onwards down the correct interface?
Yes, that's the frustrating thing. Especially since I can ping 10.0.3.96 from the same source!If source is 10.0.2.21
and destination is 10.0.3.91
but path gets lost at 10.0.100.92
What does the routing table and static routes look like at the device which answers for 100.92 ? Does it know it needs to pass this packet onwards down the correct interface?
Not at work anymore so don't have access to the pretty architecture pictures. Have a rough as a
holes sketch instead 
Thanks to all so far for taking the time to reply.
yorkshireegg said:
WinstonWolf said:
Two physically separate fibre rings with same IP addressing scheme?
I think you're going to have to re-address one so your routing tables are sane.
Yes, it is a bit odd I know. No option to change it I'm afraid. The saving grace is that the device addresses are unique across both networks (i.e. if they were the same network, there would be no conflicts). I think you're going to have to re-address one so your routing tables are sane.
ging84 said:
you have made it annoyingly complicated by having 3 different 10.0.100.0 subnets
I understand 2 were already in place, but you could have picked litterally anything else for your new one
But what i really don't understand is why the need for 2 routers with thier own internal subnet, why could you not have just have 1 acting as a bridge ?
Good point on the 100.0 subnets. I'll look to changing that if necessary - like you say, no point complicating things even further.I understand 2 were already in place, but you could have picked litterally anything else for your new one
But what i really don't understand is why the need for 2 routers with thier own internal subnet, why could you not have just have 1 acting as a bridge ?
Re: the subnet that only contains the two routers, I did have it set up as you suggest with one acting as a bridge (sat at 70.10 and 71.10). That setup had the same issues so a colleague suggested adding the extra router in an attempt to eliminate some of the traffic management issues caused by the weird handling of the external/internal ports by the firewall. I just wanted to describe the network exactly as it's sitting now but that will probably be returned to just the one router.
eltawater said:
Start simple and work your way along.
If source is 10.0.2.21
and destination is 10.0.3.91
but path gets lost at 10.0.100.92
What does the routing table and static routes look like at the device which answers for 100.92 ? Does it know it needs to pass this packet onwards down the correct interface?
Yes, that's the frustrating thing. Especially since I can ping 10.0.3.96 from the same source!If source is 10.0.2.21
and destination is 10.0.3.91
but path gets lost at 10.0.100.92
What does the routing table and static routes look like at the device which answers for 100.92 ? Does it know it needs to pass this packet onwards down the correct interface?
Not at work anymore so don't have access to the pretty architecture pictures. Have a rough as a
holes sketch instead Thanks to all so far for taking the time to reply.
- If* I'm reading that correctly it will get lost at 10.0.100.92 because 10.0.100.94 is theoretically on the network the packet just came from. I still think you need to readdress your rings so they have a sane addressing structure.
WinstonWolf said:
- If* I'm reading that correctly it will get lost at 10.0.100.92 because 10.0.100.94 is theoretically on the network the packet just came from. I still think you need to readdress your rings so they have a sane addressing structure.
ging84 said:
Should all work
The only thing i can see what would give you that sort of behaviour would be if you had setup the routes as /26 subnets instead of /24 but i would assume you have checked that already.
I will double check in the morning. Different subnets allow me access to different machine addresses though; some I can access .21, some I can't - it seems quite random. The only thing i can see what would give you that sort of behaviour would be if you had setup the routes as /26 subnets instead of /24 but i would assume you have checked that already.
I had considered ARP table issues on the routers but I can see said machines from within the same subnet and from the office subnet on the same fibre ring. Am I right in thinking this eliminates that possibility as the ARP tables are held on the gateway router for each subnet?
That's an interesting setup!
What kind of devices are the 10.0.100.70 & 71 boxes? Most systems won't let you configure overlapping subnets on different interfaces like that as you'll end up with all kinds of wierd connectivity problems like load balancing all traffic out both interfaces or sending wherever the first ARP response comes from. Using a single router there connected to both 10.0.100.0/24 networks will have the same problem.
If 'twere me then I'd re-address that link between the two rings to something that doesn't overlap as first step.
What kind of devices are the 10.0.100.70 & 71 boxes? Most systems won't let you configure overlapping subnets on different interfaces like that as you'll end up with all kinds of wierd connectivity problems like load balancing all traffic out both interfaces or sending wherever the first ARP response comes from. Using a single router there connected to both 10.0.100.0/24 networks will have the same problem.
If 'twere me then I'd re-address that link between the two rings to something that doesn't overlap as first step.
ging84 said:
you have made it annoyingly complicated by having 3 different 10.0.100.0 subnets
I understand 2 were already in place, but you could have picked litterally anything else for your new one
But what i really don't understand is why the need for 2 routers with thier own internal subnet, why could you not have just have 1 acting as a bridge ?
That allows it to work. If there was only one router it would have a 10.0.100.0/24 network on two different interfaces which cant normally be configured.I understand 2 were already in place, but you could have picked litterally anything else for your new one
But what i really don't understand is why the need for 2 routers with thier own internal subnet, why could you not have just have 1 acting as a bridge ?
yorkshireegg said:
WinstonWolf said:
- If* I'm reading that correctly it will get lost at 10.0.100.92 because 10.0.100.94 is theoretically on the network the packet just came from. I still think you need to readdress your rings so they have a sane addressing structure.
ging84 said:
Should all work
The only thing i can see what would give you that sort of behaviour would be if you had setup the routes as /26 subnets instead of /24 but i would assume you have checked that already.
I will double check in the morning. Different subnets allow me access to different machine addresses though; some I can access .21, some I can't - it seems quite random. The only thing i can see what would give you that sort of behaviour would be if you had setup the routes as /26 subnets instead of /24 but i would assume you have checked that already.
I had considered ARP table issues on the routers but I can see said machines from within the same subnet and from the office subnet on the same fibre ring. Am I right in thinking this eliminates that possibility as the ARP tables are held on the gateway router for each subnet?
ffc said:
ging84 said:
you have made it annoyingly complicated by having 3 different 10.0.100.0 subnets
I understand 2 were already in place, but you could have picked litterally anything else for your new one
But what i really don't understand is why the need for 2 routers with thier own internal subnet, why could you not have just have 1 acting as a bridge ?
That allows it to work. If there was only one router it would have a 10.0.100.0/24 network on two different interfaces which cant normally be configured.I understand 2 were already in place, but you could have picked litterally anything else for your new one
But what i really don't understand is why the need for 2 routers with thier own internal subnet, why could you not have just have 1 acting as a bridge ?
Changed my mind, if 'twere me then I'd put a switch (or bridge, or router in bridge mode depending on your chosen nomenclature) between the two rings and run a dynamic routing protocol on all the gateways so they can just learn where every other subnet is. Given none of the 10.100.x.x addresses on the individual gateways overlap I would suggest that's probably how it was designed to work in the first place, two physical rings but one logical network.
wombleh said:
He mentioned doing it as a bridge so that would be OK. I was thinking the OP had done that already because as you say most devices won't let you overlap like that, but then those two routers between the rings wouldn't be showing up in the traceroutes. Must have every gateway device with dozens of static routes on, both to all the other subnets and to every 10.100.x.x devices on the other ring. Very high chance of errors creeping in to that setup IMO.
Changed my mind, if 'twere me then I'd put a switch (or bridge, or router in bridge mode depending on your chosen nomenclature) between the two rings and run a dynamic routing protocol on all the gateways so they can just learn where every other subnet is. Given none of the 10.100.x.x addresses on the individual gateways overlap I would suggest that's probably how it was designed to work in the first place, two physical rings but one logical network.
If dynamic routing were used the two 10.0.100.0/24 networks would break everything. I think ultimately dynamic routing after renumbering would simplify everything as long as the firewall/router devices that connect to each subnet support dynamic routing of some kind and it's not too much of a pain to renumber one of the rings.Changed my mind, if 'twere me then I'd put a switch (or bridge, or router in bridge mode depending on your chosen nomenclature) between the two rings and run a dynamic routing protocol on all the gateways so they can just learn where every other subnet is. Given none of the 10.100.x.x addresses on the individual gateways overlap I would suggest that's probably how it was designed to work in the first place, two physical rings but one logical network.
Gassing Station | Computers, Gadgets & Stuff | Top of Page | What's New | My Stuff