Random Active Directory problem

[aranell]

Bit of a random one.....

Having spent a miserable week having problems with our Active Directory domain controllers, Microsoft have provided us with a "solution" which to my mind is not a proper one. I think maybe I don't understand the solution fully, so was wondering if anyone could shed some light? :confused:

Essentially, we are moving our NT domain onto an AD domain, and we've moved a few people as "test" users. After a month with no problems, last week, we had problems logging onto our machines - we either didn't get to the ctrl+alt+del prompt, or we'd be able to get to enter our passwords, and wouldn't be able to get any further (it would get stuck on "applying computer settings" either before or after you'd entered your password)

As it turns out, the routers were having problems with the Kerberos authentication - the packet size was too big hence they were getting fragmented. Now, what puzzled me was that some PCs were ok, and others weren't, even "identical build" PCs didn't act the same way. We had to add a registry hack to the PCs which couldn't log in: The hack consisted of adding two entries - the MTU size and also the MaxPacketSize, and then everyone who was having problems was able to login OK. However, a couple of people do not have these hacks in their registry and yet their PCs are logging in fine - there is no correlation between these PCs: they are different builds, with different OS's and different software. Can anyone suggest why some PCs need their registries hacked whilst others don't? I really don't want to have to change 300 PC registries be hand (the MTU entry is interface specific therefore I can't just write a little file which I can double click to add the entry as the interface ID changes from PC to PC :( )

Thanks,

- Sarah -

[BliarOut]

Not sure about the problem and I'm just nipping out, but you definitely don't have to edit the registries by hand. Find the key's you want and export it/them to a .reg file. Then in the logon script (assuming they can get logged on ) run

regedit /s yourregfile.reg

that will import the values into the registry silently.

Another option is to take apart an ADM file and build your own custom one that you can tweak to your hearts content. Stick your custom ADM in a group policy at the right level. Bit of Googling should show you the way on that one.

Have you tried netdiag and dcdiag? If not, do so as it will point you to the problem. Run in /V (Verbose) mode if you aren't seeing enough info.

HTH

Bugger, interface specific. Read the whole post Rob. Doh.

>> Edited by BliarOut on Sunday 28th August 10:56

[aldi]

How very odd - this just randomly started after a month?

Has a setting been changed on the routers? if its genuinley a problem with them and kerberos maybe theres a new firmware out for them or something?
I could be wrong but are the packets getting fragmented before they reach the router, then the router is dropping the fragments (since random fragments are used in a lot of denial of service type shinanigins). Can the router be reconfigured? It would seem easier than running regedit 300 times

[aranell]

No, nothing (to our knowledge) has been changed on the routers. The only thing that has happened is some patches were applied to the server, however these were applied after the problems started....

I've just read my work emails and the interesting thing is that the US believe that there have been problems for some time, and are surprised that all the problems started happening on the same day. Apparently, we also had a DNS problem which when fixed, has also helped cure the problem (I'm not so sure they were even related....but I'm smiling and nodding at this point )

The US have closed the ticket with MS so it looks like a) they have no answer for why some machines need the hack and others don't and this leads to b) I'm going to have to manually hack 300 machines registries....gah

Does anyone have any other thoughts on the whole registry hack thing? I've tried reading about Kerberos, but I haven't read anything which has helped my understanding. I'm sure I'm missing something pretty fundamental, and it worries me because I don't know the exact way things happen which means it makes it a lot harder to support and troubleshoot.

Any insights or thoughts welcome

Thanks,

- Sarah -

[zumbruk]

Could this be a TCP/UDP problem? Kerberos uses UDP until the maximum UDP packet size is exceeded, then it switches to TCP.

There's some reading here;

www.microsoft.com/technet/prodtechnol/exchange/guides/E2k3FrontBack/f9733398-a21e-4b40-8601-cfb452da82ad.mspx

(This relates to Exchange and firewalls, but the comments about the underlying technology are useful.)

(This site hooks the "Back" button in Firefox, preventing one from backing out of the site - more evidence, were it required, that Microsoft are evil.)

There seems to be some relevant information, and a non-interface related fix, here;

www.sfu.ca/ad/kerberos/

Bung the registry fix in their login script.

[GreenV8S]

zumbruk said:

(This site hooks the "Back" button in Firefox, preventing one from backing out of the site - more evidence, were it required, that Microsoft are evil.)

The original page appears to be framing itself within an index page, I'd say they're trying to be helpful rather than awkward. (They're evil right enough, but I don't think that's why they're doing this.)

[UpTheIron]

Before you start making changes...

Have you identified if the problem is specific to the USER or the MACHINE?

Take one of your PC's that has been fine along - get a user who has had problems to log onto it.

And vice versa - reverse the changes you have made, recreate the problem with a user who has had the problem previously, then try the same PC with a user without problems.

Then start troubleshooting from there...my money is on it build OS specific (a particular build revision/hotfix level even), or more likely, user specific - are your users members of an exceptionally large number of AD groups? Something to bloat the size of packets - e.g. SID history? Even the length of username / devicename.

Also, you mention that you had a router problem - do the unaffected PC's not go via the same router as affected ones?

>> Edited by UpTheIron on Sunday 28th August 21:48

[zumbruk]

GreenV8S said:

zumbruk said: (This site hooks the "Back" button in Firefox, preventing one from backing out of the site - more evidence, were it required, that Microsoft are evil.) The original page appears to be framing itself within an index page, I'd say they're trying to be helpful rather than awkward. (They're evil right enough, but I don't think that's why they're doing this.)

Frames are evil, too.

[agent006]

Have you eliminated trends to do with NIC makes? I.e. problems only occurring on one model? Maybe there's a firmware issue on the cards.

[aranell]

Yes - MS related us to that page on their site too, but I'm not so sure it helps.

It isn't NIC specific - I checked this. A lot of the laptops are identical - same batch, network cards etc. It's not OS specific - it affects XP Pro SP1, XP Pro SP2 and Windows 2000 SP4. It's not even manufacturer specific

Everyone goes via the same routers - and everyone logs onto the same domain controller.

It is a machine specific problem and not a user specific problem - this makes sense as we are adjusting properties in the registry for the interface and Kerberos settings rather than individual user.

I wish I knew why some need the reg hacks and others don't. I can't even narrow it down to software - all our users are in different departments and use completely separate software. I can't see a common factor in the PCs which don't need the hack vs the ones who do.

We are one of the "smaller" user groups to be migrated, so it can't be a problem there - same goes for SID history. I'm one of the newest users there, yet I had the logging in problem, so I'm sure it isn't SID history length (some of the other users have been there yonks, so isn't a case that it's the "new" users with problems).

Actually....I think I've heard someone mention we use VLans on that segment - could that be the cause? They've been there for a long while without change (as long as the AD DCs have been up) so it would surprise me that it would take months to show up a fault like this?

Thanks for all the suggestions so far

[dilbert]

aranell said:

Yes - MS related us to that page on their site too, but I'm not so sure it helps. It isn't NIC specific - I checked this. A lot of the laptops are identical - same batch, network cards etc. It's not OS specific - it affects XP Pro SP1, XP Pro SP2 and Windows 2000 SP4. It's not even manufacturer specific Everyone goes via the same routers - and everyone logs onto the same domain controller. It is a machine specific problem and not a user specific problem - this makes sense as we are adjusting properties in the registry for the interface and Kerberos settings rather than individual user. I wish I knew why some need the reg hacks and others don't. I can't even narrow it down to software - all our users are in different departments and use completely separate software. I can't see a common factor in the PCs which don't need the hack vs the ones who do. We are one of the "smaller" user groups to be migrated, so it can't be a problem there - same goes for SID history. I'm one of the newest users there, yet I had the logging in problem, so I'm sure it isn't SID history length (some of the other users have been there yonks, so isn't a case that it's the "new" users with problems). Actually....I think I've heard someone mention we use VLans on that segment - could that be the cause? They've been there for a long while without change (as long as the AD DCs have been up) so it would surprise me that it would take months to show up a fault like this? Thanks for all the suggestions so far

I don't know anything about your particular software, but I have half an idea about network cards, IP.

Have you considered that different network chipsets will have different hardware buffer sizes. When you create a packet, the upper limit on the size is usually set by the hardware buffer, if no other value has been specified.

It's also worth considering that Datagram, and TCP packets have different rules as to how they handle incoming and outgoing packets, and their sizes.

Could it be that your server side software patches, as someone has already mentioned, ahve moved from TCP to datagram. Shunting datagrams about has far more hardware compatability issues, but if you are also dealing with some kind of security issue, is probably more secure from an external vulnerability perspective.

I'd put money on the issue being related to client side hardware, specifically network chipset, buffer size.

[UpTheIron]

aranell said:

It is a machine specific problem and not a user specific problem - this makes sense as we are adjusting properties in the registry for the interface and Kerberos settings rather than individual user.

Sorry, just to clarify (or just to humour me)...changing machine settings doesn't necessarily make the problem machine specific...have you proven that the problem doesn't "follow" an affected user to a previously working machine, and that a previously affected machine still fails with a non-affected user?

What happens if you take an affected machine (without reg changes), and try with a brand new AD user who is a member of only Domain Users.

Going back to your original post, you say that sometimes machines would fail to even reach the Ctrl-Alt-Delete screen - that doesn't make a great deal of sense as they should be able to do that without any network connectivity...are you able to reproduce this problem?

[BliarOut]

Machines can hang for ages getting to Ctrl-Alt-Del if there's a DNS issue....

[aranell]

UpTheIron said:

Sorry, just to clarify (or just to humour me)...changing machine settings doesn't necessarily make the problem machine specific...have you proven that the problem doesn't "follow" an affected user to a previously working machine, and that a previously affected machine still fails with a non-affected user? What happens if you take an affected machine (without reg changes), and try with a brand new AD user who is a member of only Domain Users. Going back to your original post, you say that sometimes machines would fail to even reach the Ctrl-Alt-Delete screen - that doesn't make a great deal of sense as they should be able to do that without any network connectivity...are you able to reproduce this problem?

Yes - it is definitely machine specific....we did a few tests at the time (swapping machines and logging on)

Yes - I can reproduce the problem - if I take my registry hacks out, my machine will revert to having problems. My machine wouldn't get to the ctrl alt del prompt - it would be stuck on "Applying computer settings" before the prompt came up. Interestingly, a couple of PCs had one of the two hacks already applied (must be a piece of software had added it in???) and they were able to get to the ctrl alt del prompt and enter their passwords, however it would then hang after this on the "Applying computer settings".

If we took the network cable out the back of the PCs and used cached credentials to logon, and then put the network cables back in again, everything was pretty much ok.

Dilbert - I hear what you are saying (I think!). Are you referring to the way that different hardware may handle the requests? Is there any way I can check this? Will have to double check tomorrow as there is an identical machine to mine on the network which had no problems and I didn't think it had the registry stuff....but I need to double check that one.

Thanks for the help everyone

[aldi]

Standard machines logging onto a standard AD don't have to mess about with MTU's, its got to be a network configuration problem. Or maybe you've got a malfunctioning switch or router? Have the switches/routers involved been rebooted? I've seen switches do strange stuff thats been cured by a reboot.

Can you try plugging a troublesome machine into a different port on your switch? or a different switch entirely, different bit of the LAN etc.? Maybe you could use a packet sniffer like ethereal to try and isolate exactly where things are getting foobar'd?

IMO for MS to fob you off with this MTU thing is just them trying to get the ticket closed. But if you do get forced into using regedit as a temporary measure, you could use a VB script to change the setting rather than just a reg file, then it should be able to cope with different interfaces.

[dilbert]

aranell said:

UpTheIron said: Sorry, just to clarify (or just to humour me)...changing machine settings doesn't necessarily make the problem machine specific...have you proven that the problem doesn't "follow" an affected user to a previously working machine, and that a previously affected machine still fails with a non-affected user? What happens if you take an affected machine (without reg changes), and try with a brand new AD user who is a member of only Domain Users. Going back to your original post, you say that sometimes machines would fail to even reach the Ctrl-Alt-Delete screen - that doesn't make a great deal of sense as they should be able to do that without any network connectivity...are you able to reproduce this problem? Yes - it is definitely machine specific....we did a few tests at the time (swapping machines and logging on) Yes - I can reproduce the problem - if I take my registry hacks out, my machine will revert to having problems. My machine wouldn't get to the ctrl alt del prompt - it would be stuck on "Applying computer settings" before the prompt came up. Interestingly, a couple of PCs had one of the two hacks already applied (must be a piece of software had added it in???) and they were able to get to the ctrl alt del prompt and enter their passwords, however it would then hang after this on the "Applying computer settings". If we took the network cable out the back of the PCs and used cached credentials to logon, and then put the network cables back in again, everything was pretty much ok. Dilbert - I hear what you are saying (I think!). Are you referring to the way that different hardware may handle the requests? Is there any way I can check this? Will have to double check tomorrow as there is an identical machine to mine on the network which had no problems and I didn't think it had the registry stuff....but I need to double check that one. Thanks for the help everyone

If you're lucky enough to have a PCI network card in both a machine that exhibits the problem and one that doesn't, just swap the network cards, and see if the problem follows the network card. This is more difficult if the network hardware is integrated onto the motherboard of the offending machine.

The other possible way to solve the problem, without having to modify registry settings on individual machines, would be to make sure that the server network hardware has a bigger buffer than any of the clients. Again, this may be difficult if the server doesn't have a removable network card. It could be that the actual hardware on the server has a really big buffer, but something in the software on the server is constraining the usable size of the buffer.

Obviously the whole of that depends on me being correct about my assumptions.

[_DeeJay_]

How about adding to the default domain policy to make those settings on all of the machines (then you can just forget about it).

If not, do a network trace to identify what's different between the packets on a known working and not-working config.

It sounds seriously odd though - if the routers are in fact limiting MTU size, then you may see other problems relating to that with other applications. Possibly better to treat the cause than the symptom?

[malman]

A bit of sniffing with ethereal or similar might be needed here.

Capture and compare working and non working.


The not getting to CTRL - ALT -DEL does sound like DNS as already mentioned. (using the cached credentials gets round this generaly)

[agent006]

I think you might do well to ckeck out the VLAN and or subnets these clients are on. 95% of the network issues i come across are VLAN related.

[aranell]

_DeeJay_ said:

It sounds seriously odd though - if the routers are in fact limiting MTU size, then you may see other problems relating to that with other applications. Possibly better to treat the cause than the symptom?

My thoughts exactly....which is why I want to know WHY this is happening and not just how to fix it. Can you elaborate on any potential problems we might see??

My guess is that someone has changed something on the routers, and not told us, and realised that it's caused Worldwide problems and has decided not to own up....it wouldn't be the first time

Well, I've been told that we are pressing ahead with migrating all other users, even though we don't know what caused this, and we still don't have a proper workaround. This worries me, however, once again, I'm smiling and nodding

So, onto my next question - is it theoretically possible to write a VBS script which would pick out the interface I need and add the MTU entries into it?? How hard is this? (bearing in mind I'm the World's Worst Programmer )

In the meantime, will keep an eye out for ethereal and see if I can find something that way....

Thanks a lot for all the help/suggestions