Friday, June 28, 2013

Safely Demote a Windows 2008/r2 Core Domain Controller

Have you installed shiny new 2012 domain controllers? Do you have old, musty 2008r2 domain controllers laying around? Do you want some french fries?

Me too. Let's take a holistic look at decommissioning a domain controller. This article will not only walk through the actual demotion process but also list important pre and post steps to any successful domain controller removal. Most of the steps outlined here should actually work on 2000 up, but I've focused on 2008r2. Note that doing this in a production environment is something you may want to do over the course of a couple weeks due to reasons that will shortly become apparent. That said, let's go.


Assumptions

  • New domain controllers are installed and ready to go
  • Domain Admin Privs
  • A backup of the server should something go wrong. Note a snapshot will not suffice; as a general practice one should never use snaps on a DC.

Ensure Replication is Working Correctly


Removing a domain controller can be dangerous if your replication isn't appropriately healthy.Check the following event logs on EACH Domain Controller for issues regarding replication/DNS:
  • Application
  • Applications and Services Logs->DFS Replication
  • Applications and Services Logs->Directory Service
  • Applications and Services Logs->DNS Server
If there are any issues, sort them out prior to continuing.



Re-Point Clients


Next we need to ensure all clients are pointing to DNS servers other than the one you intend on taking down. Do the following:
  • Update your DHCP scopes with the addresses of the DNS servers you intend on carrying forward
  • Update static "clients" (probably servers) with the appropriate DNS settings
  • Update devices on your network appropriately as well. 
Your DHCP lease duration will determine how long you need to wait for the next step. The DHCP RFC tells us that clients will attempt to re-up their lease at the halfway point of the lease. Therefore, if your lease is 7 days you'll need to wait 3.5 days from when you make the change (Assuming the DHCP is fully available). Once you think all the clients have been moved, then move to the next step.



Enable DNS Logging on the DNS Controller to be Decommissioned


Now that we think the DNS server isn't used, we enable debug logging to ensure that nobody is referencing this server for DNS look-ups. Avoiding outages is more fun than fixing them. :)
  1. Edit the DNS server properties for the server to be decommissioned.
  2. Click the "Debug Logging" tab. 
  3. Check the "Log packets for debugging" and set a location for the log. Generally the defaults for this application will be sufficient. Click "OK"
  4. Restart the DNS server service or reboot the server when your environment permits to start logging.
Wait however long you would like before continuing; the longer you wait the more opportunity you give for devices still pointing to this server to issue a query. You make the call here depending on your environment.

After that wait period: Open the log you specified above; here we'll be reviewing the log for clients that still need their DNS server settings updated. At the top of the log you'll see a couple pages of initialization information that can be safely ignored. After that there will be a log of the DNS queries that the server has received. The first IP address listed in each entry is the IP of the requesting client. You may see some requests from other domain controllers; so long as the other DCs have their "client" settings configured correctly you can ignore these entries as some DC->DC DNS traffic is normal. Taking that into account, look for machines querying this DNS server and re-mediate them prior to moving on.




Transfer FSMO roles off of the DC


Before demoting the DC you need to ensure there are no FSMO roles served by it. I won't cover this in a step by step at this time to keep the article focused. To do this, refer to one of the two following links:

Demote the DC


We'll use an unattended answer file with dcpromo. You can do this on the command line but it makes it a bit easier to prepare and review the file.
  1. Create a new text file. 
  2. On the top line, enter
    [DCINSTALL]
  3. Enter the following lines after [DCINSTALL]:
    
    administratorpassword=NewLocalAdminPassword(required)
    removeapplicationpartitions=yes
    removeDNSDelegation=yes
    You may need to add other options as well, especially if this is the last DC in the domain, your Domain Admin privs are associated with a different account, or similar for the DNS service. Also, this assumes you have a DNSDelegation to remove. If you do not you should take that line out. Refer to this technet article for a full guide.
  4. Save the file to the domain controller to be demoted.
  5. Log on to the DC to be demoted. 
  6. At an elevated command prompt (default on core) execute:
    dcpromo /unattend:"(path to file)"
    where path to file=the file you just saved, i.e. dcpromo /unattend:c:\temp\demote.txt
  7. After successful execution the machine will successfully reboot. If you intend on re-using the machine for other operations you can log back in and execute dcpromo /uninstallbinaries to remove unnecessary files. If taking the server down don't bother.
Note: If you cancel a demote operation using this file the administratorpassword field will be intentionally blanked out by the installer. You'll need to reset if if you start again.


After the demotion, open up %SystemRoot%\debug\DCPROMO.LOG on the old DC and ensure the operation didn't have any major errors. After confirming the demotion was successful...

Clean Up DNS


After waiting for a replication cycle (should be 15 minutes, but wait an hour to be safe) it's time to ensure AD DNS was cleaned up successfully. Open the DNS manager and point it to one of your new domain controllers. Browse through your zones, including your _msdcs.domain.ext zone. Pay particular attention to the listed NS records in each zone. Those can also be manipulated by right clicking the zone->properties->Name Servers. Ensure only the new DNS servers are listed. Fix any other records that are incorrect; common clean up items are the _msdcs delegation in the primary zone and the sites container under the _msdcs zone. Before making changes to anything other than the the NS records (Name Servers GUI) make sure you fully understand what function the records serve. If you're unsure, this will take some study; here's a good place to start.



It should also be safe to remove the machine from the domain now should you choose to do so. To remove the machine, execute the following on a command prompt:

netdom remove /d:%USERDOMAIN% %COMPUTERNAME%

Also, be mindful that if you were using the old DC in your Windows Time infrastructure you may want to revisit that. Here is an article on how to setup an authoritative time source.

Test


Now that the machine is gone and DNS has been cleaned up, it's time to test your domain. To do so, perform the following tasks:

  • Run DCDiag on a new DC with the following command line: dcdiag /e /c and review the output for any issues. 
  • Review the event log on each domain controller. Refer to the "Ensure Replication is Working Correctly" section above for the log locations. 
That's it! You've successfully removed a DC and made sure nothing will come back to bite you. If you have questions/comments please feel free to leave the below. 

Additional Reading

 

Technet: Demote a Domain Controller
Technet: DCDiag

Monday, June 17, 2013

Upgrade Windows Server Core 2008r2 to 2012 In Place..Unless it's a Domain Controller

I would never recommend an in place upgrade of any Windows version, but sometimes it may be required in certain situations. Should this be necessary I figured we could take a few minutes to walk through it. Note that doing this on a Domain Controller presents additional challenges listed at the end of this article. (TLDR: Don't do an in-place upgrade of a core 2k8r2 DC to 2012)

Let's do a core upgrade from 2008r2 to 2012:

Assumptions

  • Your 2008r2 Filesystem is healthy and you don't have any major issues with the services on the node. 
  • Local Admin Access
  • A valid license

Upgrade


  1. Insert the 2012 disk in the physical machine or VM you intend to upgrade. 
  2. Change directory to the disk and execute setup.exe, i.e. "j:" (ENTER) "setup.exe"


  3. When prompted, choose "go online and install updates now" if you've got internet access on this machine. If not, more forward without updating. 


  4. Enter your product key for this install. Most enterprise customers will either use a volume license key or datacenter key. 
  5. Here you can select to either stick with the core install or install the GUI as part of the upgrade. I'm assuming that because you are upgrading the core version you'll want to stick with core. 


  6. Accept the license terms. 
  7. Select "Upgrade" unless you want to do a custom install. I won't cover that here since the options would be specific to your implementation.


  8. At the next screen you will be warned if there are any compatibility issues; often times virtualization components will be flagged. Oft times you can ignore this warning, but be sure you read it in full and consider how easy it would be to double back now and correct the issue. 


  9. The upgrade process will now proceed; there will be a few reboots. 


  10. After the upgrade is complete, ensure you update virtualization components if applicable and run Windows Update. 

Upgrading a domain controller on a core installation of 2008r2 to 2012 doesn't really work without relatively major workarounds. See this Microsoft support article for more information. This fix took quite awhile to be developed, and leaves a bit to be desired. It's probably easier to do the right thing anyhow; stand up new 2012 DCs and decommission the old 2008r2 servers. After you decommission the old ones, make sure you ensure the AD sites as well as the _msdcs and primary DNS zones are cleaned up correctly. (that's an article in and of itself) There is a long discussion thread regarding this workaround on the Technet forums.

The Dreaded Black Screen of Life!


It should also be noted that the Win2012 certification exams (70-41x) are not aware of this issue. Until next time!

Tuesday, June 11, 2013

Windows 2012 NAP (NPS) with DHCP

After my last article on DHCP I decided to flesh out the Microsoft offerings and expand into Network Policy Server. NPS is actually a set of different features from Microsoft including RADIUS and what the rest of the industry knows as NAC, (Network Access Control) which Microsoft calls NAP (Network Access Protection). This allows you to grant or deny network access to clients based on criteria such as:
  • Domain/Group Membership
  • Client source network
  • Time of day
  • Client health (as determined by an SHV, see below for more)

3 out of 4 Doctors are very serious about Microsoft NAP.

These rules must be introduced to the clients at an insertion point (think of it as an entry gate). The supported insertion points include:
  • IPSec
  • 802.1X
  • VPN
  • DHCP
  • RDGateway

I'll be covering integration with DHCP since it is by far the most cost effective method considering the required role of Microsoft based services in the environment. For information on the others, see Technet: NAP Enforcement Points.First, let's set up NAP:

Assumptions

  • At least one Windows 2012 server ready to go. Note that NPS is not supported on server core. Most of these instructions are applicable to 2008/r2 as well.
  • DHCP installed and ready to go.
  • Sufficient Privileges (Domain Admin generally)
  • I assume you'll be installing NAP on the DHCP server itself. You can have these roles on separate servers should you desire, but you'll need to install the NAP piece on the DHCP server as a RADIUS proxy. I won't be covering that piece here. For more on that, see this Technet article.

Installing NAP

  1. Using server manager, select "Manage"->"Add Roles or Features"
  2. Navigate through the Add Roles and Features Wizard, selecting the target server and "Network Policy and Access Services".


  3. When prompted to select role services, you need keep only the default "Network Policy Server" selected and continue through the wizard.


Repeat this step for the DHCP server as well if it's a different server than the one you installed NAP on.

Configuring NAP

The NAC portion of NAP is actually a collection of several different elements. The following elements make up a NAP policy that can be used by DHCP; numbers in front represent the default number of that item for one overall policy:

  • Connection Request Policy (Created by Wizard)
  • (3) Network Policies (Created by Wizard)
  • (2) Health Policies (Created by Wizard)
  • Windows Security Health Validator
  • At least one Remediation Server Group

There is a (seemingly hidden) wizard to guide you through the process of creating most of these elements, but we're going to create the unguided ones first and then circle back and use the Wizard since we'll want to point to those during the Wizard portion. For all these sections save the DHCP and client sections we'll be working in the Network Policy Server management tool.

Windows Security Health Validator

The Security Health Validator is the policy for defining what a healthy windows client is. The built-in Windows SHV gives the NAP server the capability to interact with the Network Access Protection Agent service on Windows clients (XP SP3 and newer) to determine the health status of that given client. The Security Health Validator determines what criteria the client must pass for the client to be considered healthy. There is a default configuration that can be utilized but for the sake of experience we'll configure the Windows SHV. To do so:

  1. First, open the NAP management tool by selecting "Tools"->"Network Policy Server" from Server Manager.
  2. Expand "Network Access Protection"->"System Health Validators"->"Windows Security Health Validator"->"Settings"
  3. Right click "Settings"->"New"
  4. Give the SHV a name; make it something meaningful to describe the target client base, i.e. "Standard Client Set"
  5. Set your settings as desired; they're split into settings for Windows XP and Vista or higher. The settings are quite straightforward though it should be noted that if you would like to utilize WSUS to update out of compliance clients (assuming you select being out of date makes for a failed check) you'll need to check the box on the very bottom of the options dialog. Click "OK" when complete.


  6. Click on the "Error Codes" under "Settings" and take note of the Error Code Configurations. Generally the default state of "Noncompliant" for each setting is desirable, but depending on your clients and equipment there may be situations where you would want to change some of these SHV check failures to compliant.

Note that third parties can also create SHVs to plugin to the NAP architecture for use with other products. (Old list of some others here)

Remediation Server Groups

The remediation server groups are the servers that will be made accessible to non-compliant clients if you choose to do so. These servers can be used to patch clients to a compliant status. Note that any services needed to contact the remediation services (WSUS, Antivir FTP, etc.) need to be available for the clients to update properly. (DNS, etc.) Under the DHCP enforcement model, these servers are made available via static routes. Also note that you can provide a help URL to a website instructing access-limited clients on how to repair their machines. The server that hosts this URL and all resources required to resolve it must be part of the remediation server group as well. For more information about RSGs, see Technet: Planning the Placement of a NAP Remediation Server. Assuming you have determined which servers need be part of the remediation group, let's set them up:

  1. Expand "Network Access Protection"->"Remediation Server Groups"
  2. Right Click->"New" (Note that these can be stored as templates for use elsewhere as well)
  3. Give the group a meaningful name; note that each health policy can use only one group, so everything for that client base will need be in that group. Something like "Sitename Remediation Group"
  4. Click the "Add" button to add a server.
  5. Enter a friendly name for this server, i.e. "Minneapolis WSUS 01" or just the server name and then enter the DNS name and click "Resolve". Hit "OK" when complete.
  6. Repeat steps 4 and 5 for each server in this group. Then hit "OK" on the "New Remediation Server Group" page.


Setup Remaining Items with The Whiz

  1. Click "NPS" on the top of the Network Policy Server management tool.
  2. Click "Configure NAP" to launch the secret hidden Wizard.


  3. Select "Dynamic Host Configuration Protocol (DHCP)" under "Network Connection Method". Note this is where you would select a different option should you want to use a different insertion point. If you wish to enable this policy on only specific scopes then give the policy a name, I.E. "Minneapolis NAP DHCP".  If you wish this policy to be effective on all scopes do not change it from the default name. Click "Next".
  4. On the next screen, click add if the DHCP server is not on the same server as the main NAP server. If that is the case, enter a Friendly Name, Address, and Shared Secret and click "OK". If not, no action is necessary. After this is complete, click "Next".
  5. On the next screen, "Specify DHCP Scopes", we need only add scopes if we want this policy to apply to a specific set of scopes. If we do not specify any scopes it will apply to all NAP enabled scopes. Either add the name of all specific scopes to which this policy will apply or just click "Next" with the scopes empty to have it apply to all.


  6. The "Configure Machine Groups" screen, like the previous DHCP Scopes screen, is only for limiting access to a set of computers. Should you choose, specify the group(s) of computers you would like to receive IP addresses. In most cases you should leave this blank, but in the event you would restrict via group click "Add" and enter the groups you would like to have access. Click "Next" when you are done with this screen.


  7. On the next "Specify a NAP Remediation Server Group and URL" screen select the server remediation group we created earlier. If you would like the clients to have access to a web site describing how to re-mediate their machines enter that under "Troubleshooting URL". Note you must setup this site, it is not included with NAP since the instructions will be different depending on your software selection. After entering the required information, click "Next".


  8.  "Define NAP Health Policy" is the final screen. You should see the "Windows Security Health Validator" selected and you can go ahead and enable auto-remediation of client computers with the applicable check box. I'll address this a bit more in closing.
  9. As for "Network access restrictions for NAP-ineligible client computers", select whichever you prefer. In most cases if you've bothered to come this far you'll be selecting to deny full network access since that's usually the point. Click "Next".


  10. You will be presented with a summary screen; review the information and click "Finish".


Configure Clients

We need to configure two important elements to make the clients functional: Enable the service and configure it to enforce via DHCP. To accomplish that, we'll use group policy. You will need to ensure your group policy objects are targeted appropriately via something like OU linking or security filtering. If you need more information on how to target group policy objects, see this link. Also note that according to some Microsoft documentation, the Wired and/or Wireless Autoconfig services need to be set to automatic, but in my testing they worked when set to manual. Keep that in mind if you have issues.

  1. Edit the group policy object you plan to use for NAP client enforcement using the Group Policy Management tool.
  2. Navigate to Computer Configuration->Policies->Windows Settings->Security Settings->System Services
  3. Double click "Network Access Protection Agent" and check "Define this policy setting" and select "Automatic". Click "OK" to save the setting.


  4. Navigate to Computer Configuration->Policies->Windows Settings->Security Settings->Network Access Protection->NAP Client Configuration->Enforcement Clients
  5. Double click "DHCP Quarantine Enforcement Client", check "Enable this enforcement client" and click "OK" to save the setting.


  6. Provided you're ready, take any steps necessary to apply the GPO to the desired clients. (Link, etc.)

Enable in DHCP

You can either enable globally or on a per-scope basis. I will give the instructions on how to enable globally. If you want to enable on a per-scope basis substitute right-clicking on the "IPv4" below with the scope(s) you desire instead. In that case, you'll need to specify the custom name of the policy you created in step 3 above.

  1. Open the DHCP Manager and point it at the server in question.
  2. Right click "IPv4" and click "Properties"
  3. Click "Network Access Protection"
  4. Select what you would like to happen should the NAP server become unavailable, then click "Enable on all scopes".


  5. You will be notified that your NAP settings will be overridden on all scopes. Assuming this is what you would like to do, click "Yes".
  6. Click "OK".

If you're using a 2012 failover/loadbalance configuration the NAP settings will replicate for each scope, but make sure communication to the NAP server is configured correctly for each DHCP server. That connection information is not replicated. Here is additional info should you need it: Technet: Configure a DHCP Server for NAP. No restart of anything is needed; you should be using NAP now!

Usage Notes/Troubleshooting

  • If using a custom profile name for a specific scope, you'll need to provide the custom profile name. This name is the name of the connection request policy you would like to use. For more info see: Configuring Custom NPS Policies Per DHCP scope


  • The event log location relevant to NPS authentication is in the security log with the task category of "Network Policy Server". A filtered view of this can be found under "Custom Views->Server Roles->Network Policy and Access Server".
  • Know that this solution won't keep out anyone trying to infiltrate your network; anyone with a moderate amount of savvy can take steps to determine what IP to assign themselves should they have physical access to your network.
  • What references what?
    • * Compliant and * Non-Compliant Network Policies reference Health Policies
    • Health Policies reference Security Health Validators
    • DHCP scopes reference Connection Request Policies
  • You can add several other types of criteria to your policy should you desire. Take a look at your policies under NPS->Policies->Network Policies for more options.


  • If you enabled auto-remediation, clients will try to repair themselves for simple issues. For example: if a client's firewall is off and the policy requires it, the client will re-enable the firewall and attempt to pass the health check again. Should this fail manual intervention will be necessary. For more information on how to troubleshoot the client, see "Configure NAP Tracing" on Technet.
  • Make sure you monitor the load on your NPS servers; the last thing you want is for them to get overwhelmed and prevent proper servicing of DHCP requests. There are some performance counters that can help you with this task; for more info see Technet: Load Balancing with NPS Proxy. Also, be sure to read up on Technet: Best Practices for NPS which covers performance as well as other important info.
  • For some information regarding how to use Powershell to configure/manipulate NPS, see my post here.

Prologue

Obviously we're just scratching the surface here; there is quite a bit more that we could dig into but I'm going to stop here in the interest of time. NAC solutions aren't particularly popular right now in a regular office scenario, but as issues continue to arise with malware, etc. more companies may determine these sorts of solutions are necessary. If you already have a Microsoft based infrastructure you probably have nearly everything you need to implement this solution. If you have questions/concerns/comments please feel free to comment. Thanks!

References

Network Policy Server

Network Policy and Access Services Overview

Network Access Protection in NPS

Checklist: Implementing a DHCP Enforcement Design

Networking and Access Technologies: Network Policy and Access Services

Planning the Placement of a NAP Health Requirement Server

NAP Client and Server-side Component Communication