LAN, subnet, router and firewall fun!

[yorkshireegg]

After much head scratching and hair pulling, I've run out of ideas on an issue I'm currently having at work. I've had some very helpful advice here in the past so I'm hoping there are some gurus able to help me out!

To set the scene... I work in a body in white factory, the shopfloor is split into zones. Each zone is split into two discrete subnets - controls for PLCs, HMIs etc. and the archive network for backing up robot data, rivet gun data etc.

These two networks were built as physically separate networks (no idea why! and it's not something I can change so we have to deal with it as is) so we have all the controls for each zone on a LAN and the archive network for each zone on a LAN. The zones each have their own subnet, for example:-

Zone 2 Controls - 10.0.2.xx
Zone 2 Archive - 10.0.3.xx

Zone 3 Controls - 10.0.6.xx
Zone 3 Archive - 10.0.7.xx

And so on, throughout the shop. These subnets are linked on fibre ring networks (two separate ones, remember) both with the subnet 10.0.100.xx (so both have the same subnet numbering but are different physical networks).

Each subnet has a firewall/router as its gateway at 10.0.yy.1 which sits on the corresponding 10.0.100.xx network. When we have need for one zone to communicate with another, we add the routes and firewall rules to the relevant routers and all is well.

Now comes the issue. We have need to communicate between the controls and archive networks in order to interrogate the robots from the HMIs. So to achieve this we have set up some routers on two networks set up in our office:-

Office Controls - 10.0.70.xx
Office Archive - 10.0.71.xx

The new routers sit at 10.0.70.10 and 10.0.71.10 respectively and communicate via a 10.0.100.xx subnet (this subnet contains only these two routers - I just wanted to stick with the standard already set for linking routers)

So the route from 10.0.2.21 to 10.0.3.21 goes

10.0.2.1
10.0.100.92 (10.0.70.1)
10.0.70.10
10.0.100.71 (10.0.71.10)
10.0.71.1
10.0.100.35 (10.0.3.1)
10.0.3.21

Which works brilliantly. However, if I want to go to 10.0.3.91, the path gets lost at 10.0.100.92! It's a similar story throughout the shop and it doesn't seem to make sense. All the routes have a CIDR of /24 (255.255.255.0 mask) to cover the entire subnet so I have no idea why some addresses will work but others won't within the same subnets.

Has anyone had issues like this before or knows what would cause such a thing? I've been messing about with this for a couple of weeks now and just don't know where next to turn.

Apologies for the wall of text, I wanted to get as much info in as I thought was required! Thanks for reading this far if you made it!

[yorkshireegg]

jpringle819 said:

Do you have static routes on these routers 10.0.70.10, 10.0.71.10? What devices are you using as routers as the setup looks overly complicated the route you show is only 2 hops shorted than the route from my PC to google.

Yes, all routers have static routes. Those two route to either their main gateway or the other one of the pair, with a whole list for all the zones.

We're using Hirschmann Eagle firewall routers. It wouldn't be my choice as they are very complicated and overly secure for our use but some things can't be changed.

I know the routes are ridiculously long but I have to go from the zone's subnet, to the controls main ring network, to the office subnet, over the two routers we have up there and down the archive ring network to the zone. The link in our office is the only link between the two networks, hence the overly complicated routes.

ging84 said:

I don't really understand this notation, but it looks like you have some sort of natting going on

Apologies, I should've explained what I'd written - one of those things that looks obvious when you're the one writing it down!

The IPs in parenthesis are the 'other side' of the router, so for 10.0.100.92 (10.0.70.1) - the packet enters the router through the interface at 10.0.100.92 and leaves the router from 10.0.70.1. Hopefully that makes more sense of what I was trying to explain.

There is no NAT set up at present.

[yorkshireegg]

WinstonWolf said:

Two physically separate fibre rings with same IP addressing scheme?

I think you're going to have to re-address one so your routing tables are sane.

Yes, it is a bit odd I know. No option to change it I'm afraid. The saving grace is that the device addresses are unique across both networks (i.e. if they were the same network, there would be no conflicts).

ging84 said:

you have made it annoyingly complicated by having 3 different 10.0.100.0 subnets

I understand 2 were already in place, but you could have picked litterally anything else for your new one

But what i really don't understand is why the need for 2 routers with thier own internal subnet, why could you not have just have 1 acting as a bridge ?

Good point on the 100.0 subnets. I'll look to changing that if necessary - like you say, no point complicating things even further.

Re: the subnet that only contains the two routers, I did have it set up as you suggest with one acting as a bridge (sat at 70.10 and 71.10). That setup had the same issues so a colleague suggested adding the extra router in an attempt to eliminate some of the traffic management issues caused by the weird handling of the external/internal ports by the firewall. I just wanted to describe the network exactly as it's sitting now but that will probably be returned to just the one router.

eltawater said:

Start simple and work your way along.

If source is 10.0.2.21
and destination is 10.0.3.91
but path gets lost at 10.0.100.92

What does the routing table and static routes look like at the device which answers for 100.92 ? Does it know it needs to pass this packet onwards down the correct interface?

Yes, that's the frustrating thing. Especially since I can ping 10.0.3.96 from the same source!

Not at work anymore so don't have access to the pretty architecture pictures. Have a rough as aholes sketch instead



Thanks to all so far for taking the time to reply.

[yorkshireegg]

WinstonWolf said:

• If* I'm reading that correctly it will get lost at 10.0.100.92 because 10.0.100.94 is theoretically on the network the packet just came from. I still think you need to readdress your rings so they have a sane addressing structure.

I see what you're saying and I would be inclined to agree with you but why can I access some machines on a given subnet but not others?

ging84 said:

Should all work

The only thing i can see what would give you that sort of behaviour would be if you had setup the routes as /26 subnets instead of /24 but i would assume you have checked that already.

I will double check in the morning. Different subnets allow me access to different machine addresses though; some I can access .21, some I can't - it seems quite random.

I had considered ARP table issues on the routers but I can see said machines from within the same subnet and from the office subnet on the same fibre ring. Am I right in thinking this eliminates that possibility as the ARP tables are held on the gateway router for each subnet?

[yorkshireegg]

Apologies for not replying to this thread, things have been rather hectic.

Latest news is we have joined the two fibre rings over the weekend and it seems to be working much better. I was reluctant to do this at first as ownership of the network is a bit complicated and I'd rather not take that decision on myself! We're going to run a few tests to see how it goes with traffic levels etc. Is it actually likely to be an issue?

Regarding the previous setup with the two separate 10.0.100.0 rings for clarity... The routing tables are all static (and yes, I now have to go round today updating them all!) and held within the industrial firewall/routers that act as the gateway to each subnet. Because each gateway could only see the 100.0 ring that it sits on, the routes actually worked for some machines. Using tracert showed this. ( All masks were set at /24, all machines able to see other machines on the same subnet and the servers sat on the same ring. I can only assume an obscure hidden firewall rule was stopping some packets to some addresses due to the crazy routing necessary to make it 'work' on that setup.

Fingers crossed this new setup works... the routes are much simpler so issues will be easier to trace at least!

Thanks all for your help. Always good to see different ideas for solutions to a problem.