Greetings All. Eric here again. Recently I was doing an ADRAP remediation and one of the High Risk findings that the tool found was “Multiple Copies of a Primary zone Stored in Different Locations”. In this environment all of the DC’s were Windows Server 2008 R2. I’ve seen this finding on the ADRAP report a number of other times where the same zone was in the ForestDNSZones, DomainDNSZones, and Domain partitions, among other file based zones across different DC’s all at the same time in some pretty heinous environments. Obviously that’s a pretty big problem because the data would be inconsistent across the zones (even though it’s the same namespace) and clients pointing to the different DNS servers would likely get different answers to their queries, which could have a pretty major impact. This time however, none of the zones were really in different partitions… In looking at a DNSCMD /enumzones output however, it showed that 800+ zones on one DC were in a Custom Application Partition (AD-Custom) and the same 800+ zones on another DC were in the DomainDNSZones Partition (AD-Domain) or the ForestDNSZones Partition (AD-Forest). This was the case on a number of DC’s. If you looked at the DC’s that had the zones configured correctly, it would look like this:
If you looked at that same zone on a DC that was incorrectly configured (but not by the administrators), it would look like this:
The FQDN of the Application Partition seen in the “To all domain controllers in the scope of this directory partition” section of the above screenshot is the same FQDN as the default DomainDNSZones application partition, but for some reason some of the DC’s had the Default Application Partitions defined as Custom Partitions (This shouldn’t happen). As a matter of fact, not only should this not happen, but you can’t even manually create a Custom Partition with the same FQDN of either of the default partitions. If you try, then it fails with the following error:
Create directory partition failed: domaindnszones.contoso.com status = 9902 (0x000026ae) Command failed: DNS_ERROR_DP_ALREADY_EXISTS 9902 0x26AE
Just to be sure that there wasn’t some sort of duplicate or Custom Partition, I went looking in the configuration container (CN=Partitions,CN=Configuration,DC=Contoso,DC=Com) for additional Partitions / CrossRef objects. There were none. So now, I was just trying to figure out what other differences I could find to try to come up with a decent plan for fixing the issue. When doing a dnscmd /enumdirectorypartitions on a “good” DC, I saw the following output:
When doing the same command on a “bad” DC, I got a similar output, but missing some key elements….“Auto Domain” and “Auto Forest” were missing, which is what you see missing when you run the command and have custom application partitions present.
So the next thing that they allowed me to try, on a reverse lookup zone that had no data in it, was to change the scope of replication from “To all domain controllers in the scope of this directory partition” to “To All DNS Servers running on Domain Controllers in this Domain”. At first, it looked like it was going to work, but then it failed with the following error:
So after that failed, I wanted to try a different approach. Since the DC thought it belonged to a custom application partition, I thought, why not try to see if I could unenlist it from that custom application partition. By default, you cannot unenlist DC’s from the built in application partitions, so I wasn’t sure what to expect…
I ran the following command:
DNSCMD “BadDC” /unenlistdirectorypartition DomainDNSZones.Contoso.com
To my surprise, the command ran successfully. After that, I cycled the DNS service, and ran an updated DNSCMD /enumzones to see if everything was still showing AD-Custom. I figured that the zones that were in the partition that I just unenlisted the DC from would possibly either be fixed or impacted in some other way. To my surprise, all zones were now showing up correctly in the partitions that they belonged to, to include those in the ForestDNSZones partition, regardless of me unenlisting the DC from the DomainDNSZones partition.
We tested this on one DC after hours and left it over night to see if there would be any adverse impact that we didn’t know about, but there were no issues. The next day we did the same thing to all of the rest of the “bad” DC’s in the environment and it worked on all of them. Later in the day however, we got a call saying that one of the ones in the bunch was having an issue. All of the zones that were in the “DomainDNSZones” partition were missing from the DC. The odd thing, was that this was the only DC that it happened to. After doing a bit of digging around, apparently the partition was there, but nothing as in it. After doing a DCDiag, we saw that the issue appeared to be a replication latency issue. There were replication pre-emptions that prevented the data from replicating back to the DC….however, there should have been no reason that data should have had to replicate to it in the first place, since the data already existed on the DC. We kept re-running DCDiag and verbose output showed us specifically that data was being pulled for the DomainDNSZones partition and the percentage of the data that was pulled / how much was left until it was finished. Once it finished, we refreshed DNS and all of the data was back.
I’m not exactly sure why that one box had the issue, but it fixed itself for the most part. I will add that we ended up removing / re-installing DNS, but even after the re-install, the zones were still missing. I think we would have had the same result without removing / re-installing DNS. I just wanted to add that piece into the discussion should someone find the same thing occurring in their environment and choose to use the “fix” described above. Just keep a watchful eye on the DC’s where the “fix” is implemented.
In regards to the original issue, where half of the DC’s had zones that were showing up in the correct AD-Domain or AD-Forest partitions, and the other half of the DC’s had those same zones showing up in AD-Custom Partitions, I don’t know why or how this issue occurred or what the root cause is. I’ll post updates to the scenario when I get them, regarding root cause, how to repro it, if it’s a bug, etc..