Cloudy I/O Performance - Increasing Azure IOPS (Part 2 of 2)

Notes: This is part 2 of a 2 part post. You can find part 1 here. Since publication of this article Microsoft has introduced new machine types with higher performing storage. More information can be found here.

Foreword

In the last article we discussed a repeatable testing methodology to quantify storage performance in the cloud, and in this article we'll put that methodology into practice. I've done substantial testing in Azure and aim to illustrate what your options are for scaling performance at this point in time.

Scope

I undertook this project to see what can be done to increase disk I/O in Windows Azure IaaS. Upon researching the topic I found several interesting articles. Among those are:

• Timothy Khouri: Windows Azure IaaS Performance – SQL and IOps (Link removed, site was exploited)
  
• Timothy Khouri: Configuring Disks in Azure for More Performance (Link removed, site was exploited)
  
• MSDN: Performance Guidance for SQL Server in Windows Azure Virtual Machines
• MSDN: Getting Started with SQL Server in Windows Azure Virtual Machines
• Azure: Storage Pricing Details
• Azure: VM Pricing Details
• Virtual Machine and Cloud Service Sizes for Windows Azure
• Azure Blogs: Windows Azure’s Flat Network Storage and 2012 Scalability Targets
  

There seems to be little consensus regarding disk striping in Windows Azure IaaS. Some blogs recommend this while some of Microsoft's own writing seems to discourage it. After combing through the options the following points stand out:

• Disk Striping (Software RAID 0) may or may not increase performance based on your workload.
• Striping will increase I/O capacity to a degree (which we'll test here).
• What software striping solution works better: legacy (Windows software RAID from 2000 to present) or Storage Spaces (new software "RAID" in Windows 2012 and up)?
• How does NTFS cluster size impact performance?
• If striping, disable geo-replication as Microsoft explicitly warns against the use of geo-replication with this solution.
• If possible, use native application load distribution rather that disk striping to split I/O.  (For example, split DB files in SQL across disks)
• Some articles reference needing to use multiple storage accounts to get maximum performance. This is not true; as of 6/7/2012 storage account targets are 20,000 IOPS per account. Unless you will exceed the 20,000 keep all your disks on one account for the sake of simplicity. We will prove that does not have an impact on performance.

With that said, I want to quantify the solution for my given scaling problem with the notion that if the tests are simple enough to run, this approach can be used for any future scaling problem as well.

Putting it All Together

We'll use the testing methodology outlined in part 1 of this article to collect the results. In this case we need to first add disks and set up stripes in Azure Windows VMs.
Note: To jump straight to Azure disk performance tests, scroll to the bottom of this article.

Create New Disks and Attach to Designated VM

In order to run all the tests listed below, you need to know how to create new disks and attach them to your virtual machine. My favorite solution to this is to use a locally created dynamic VHD and upload it to the location you would like using PowerShell. Let's go through the process of attaching one disk as a primer:

1. Decide which storage account you will use for these disks. If you plan on doing striping of any kind, ensure the storage account is set to "Locally Redundant" replication (Storage->Desired Storage Account->Configure), as "Geo Redundant" is not supported. Since the replication setting applies to all blobs (Azure's terminology; disks) in that account you may want to have a dedicated account for these disks to keep your others Geo Redundant.


 


2. Determine what container you would like to store your Azure disk blob by opening the Azure management portal and navigating to Storage->Desired Storage Account->Containers and copy the URL to your clipboard. To keep things simple you may want to create a new storage container, so do so now and use that URL if desired.
3. Using Hyper-V (On Windows 2008 or higher including Windows 8) create an empty dynamically expanding VHD disk of your desired size. For my testing I have been using 10GB disks. Note 1: Do not create a VHDX; Azure uses the older VHD format. Note 2: You'll need to re-create the VHD for each disk if you intend on using Storage Spaces as each disk must have a unique ID. 

#create a dynamically expanding 10GB VHD; change size as appropriate
New-VHD –Path $sourceVHD –SizeBytes 10GB -Dynamic

4. This disk will be uploaded to the container we selected in step 1. Determine the name you want the disk to be referenced by in Azure and execute the following script:

#import Azure cmdlets
import-module azure.psd1
#specify your subscription so PS knows what account to upload the data to
select-azuresubscription "mysubscriptionname"
#$sourceVHD should be the location of your empty vhd file
$sourceVHD = "D:\Skydrive\Projects\Azure\AzureEmpty10G_Disk.vhd"
#$destinationVHD should be the URL of the container and the name of the vhd you want created in your account. Obviously for subsequent disks you need to change the VHD name. 
$destinationVHD = "http://yourstorageacctname.blob.core.windows.net/vhds/data02.vhd"
#now upload it. 
Add-AzureVhd -LocalFilePath $sourceVHD -Destination $destinationVHD






5. Add this new disk as available to VMs by navigating to Virtual Machines->Disks->Create





6. Enter the desired management name for this disk and input or browse to the URL of the VHD you just uploaded and click the check box.
7. Attach the disk to your VM by navigating to Virtual Machines->Ensure your desired VM is highlighted->Attach->Attach Disk





8. Select the disk we just added. Your cache preference will depend on the application. In my case this is off but you will want to use the methodology outlined in the first part of this article to test caching impact for your application. Note a change of cache status requires a VM reboot.

Now for a brief tutorial on how to set up our two types of striped disks; you'll likely only be using one of the two but I'll cover both just in case. Performance results of each are outlined later in this article.

Set Up a Traditional Software Stripe in Windows

Setting up a traditional software stripe is easy. I've tested this on Windows 2003 and higher.

1. Logon to your VM as an admin and open the Disk Management tool.
2. If prompted, allow the initialization of the disks.
3. Right-click on one of the newly created empty volumes and select New Striped Volume.





4. Select the desired disks and continue.





5. Create and format a new NTFS disk using your striped volume. Make sure to pay attention to the cluster size (results below).

Setup a Storage Spaces Software Stripe in Windows 2012 or Higher

Microsoft introduced a new approach to disk pooling in Windows Server 2012 and Windows 8 called Storage Spaces. This interesting new tech allows for a myriad of different configuration options including disk tiering which can be useful for on-premise servers. In this case we'll be using the "simple" pool type which is similar to disk striping.

1. Open Server Manager and navigate to File and Storage Services -> Volumes -> Storage Pools
2. Under Storage Pools you should see "Primordial". (As opposed to "Unused Disks". I'm guessing someone was pretty proud of that.) Right click it and select "New Storage Pool".





3. Walk through the Wizard selecting each disk you would like to be part of the pool.





4. On the results page, ensure "Create a virtual disk when this wizard closes" is selected and click "Close".





5. Walk through the Virtual Disk Wizard, specifying a meaningful name and selecting simple storage layout and fixed provisioning.





6. On the results page, ensure "Create a volume when this wizard closes" is selected and click "Close".
7. Complete the New Volume Wizard specifying your desired drive letter and desired NTFS cluster size.

Run Tests/Collect Results!

Now that we have our disks configured, we need to run our tests. For instructions how how to do so, see part 1 of this topic here.
When analyzing the IOMeter output you will want to pay special attention to the following metrics:

• IOPS (Cumlative, Read, Write, Higher is better)
• MBps (Cumlative, Read, Write, Higher is better)
• Response Time (Avg, Avg Read, Avg Write, lower is better) 

If putting the data together for a report, Excel works nicely as I'll display below.

Results

Now for the most important part, the findings. Tests performed:

Sector Size Tests:

• 1 Disk, 4k Sector Size (default)
• 1 Disk, 8k Sector Size
• 1 Disk, 16k Sector Size
• 1 Disk, 32k Sector Size
• 1 Disk, 64k Sector Size
• 3 Disks, 4k Sector Size (results confirmation test)
• 3 Disks, 32k Sector Size (results confirmation test)

Table 1-Cluster Size Tests

Table 2-Cluster Size Verification

Sector size tests echo what others have observed with Azure; since IOPS are capped at 500 (or 300 for basic VMs) larger sector sizes can result in higher throughput. In my case 32k was the sweet spot; depending on your workload your results will vary slightly. I have seen consistently (albeit slightly) higher performance with larger sector sizes in Azure.

Legacy Disk Striping Tests:

• 1 Disk, 32k Sector Size
• 2 Disks, Striped Volume, 32k Sector Size
• 2 Disks in 2 Storage Accounts, Striped Volume, 32k Sector Size (Multiple Storage Account Test)
• 3 Disks, Striped Volume, 32k Sector Size
• 4 Disks, Striped Volume, 32k Sector Size

<See Bar Charts Below Under Disk Striping Methodology>

Table 3-Legacy Striping and Storage Account Tests

You can see with one disk we get 500 IOPS as expected. From there we can see a scaling trend that is most definitely not linear. Two disks result in 33% higher performance, while three disks add an additional 23% (64% from one disk). Adding the fourth disk actually results in a drop from three disks, coming it at 5% lower than three and 56% higher than one.
Additionally, we also see that splitting disks across storage accounts makes no appreciable difference.  Note: Bar charts for this results section have been combined into the graphs below.

Disk Striping Methodology Tests:

• 2 Disks, Striped Volume 32k Sector Size
• 2 Disks, Storage Spaces Simple, 32k Sector Size
• 3 Disks, Striped Volume, 32k Sector Size
• 3 Disks, Storage Spaces Simple, 32k Sector Size
• 4 Disks, Striped Volume, 32k Sector Size
• 4 Disks, Storage Spaces Simple, 32k Sector Size

Table 4-Legacy Striping vs. Storage Spaces Test

Now we compare legacy striping to the newly introduced Storage Spaces. Two disk scaling is a definitive win for Storage Spaces, while beyond that legacy striping generally performs better (save max latency). In my opinion two disk Storage Spaces stripe is the sweet spot here (56% IOPS improvement!) when considering that the with more disks we add complexity that doesn't pan out on the performance side.

Conclusion

I hope you have found these results interesting; I certainly have. Even if you choose not to run these tests yourself I hope my results prove helpful when sizing your machines. Since the access pattern I used is relatively universal it should be applicable in most scenarios.
Software level disk striping works relatively well in Microsoft Azure to increase per-disk performance in lieu of a provider level solution similar to Amazon EBS provisioned IOPS. Splitting the workload across logical disks or VMs is preferred but not applicable to all workloads. When employing this solution make sure you select only locally redundant replication because Micrsoft warns that geo-redundant replication may cause data consistency issues on the replication target.
For additional information see the links near the top of this article. Thanks for reading!

Cloudy I/O Performance - Deciphering IOPS in IaaS (Part 1 of 2)

Note: This is part 1 of a 2 part post. Part 2 can be found here.

Foreword

Disk performance scaling options in the public cloud seem limited (particularly in Azure as of this writing), but there are ways to increase your IOPS in IaaS solutions. To add to the performance problem, transactional costs of running application tests can be not only time consuming but expensive. To tune your storage performance reliably you will need a fast, consistent way to test different configurations. This article will cover that methodology and lead into a results/guidance article for Azure (but applicable to others) IaaS storage performance.

We'll be doing this testing on Windows, but you could also easily do this on Linux and the results that I'll be sharing are just as applicable there. To accomplish this testing we'll be using the following tools:

• IOMeter (for Linux try iorate or iometer on Wine)
• PowerShell cmdlets for Azure
• Disk Management

Let's begin!

Execution

We will proceed in the following order:

1. Analyze Workload
2. Create Test Scenarios
3. Collect and Analyze Results (Mainly in Part 2)
4. Findings (In Part 2)

Assumptions

If you plan on emulating my tests you'll need to have access to the following:

• Microsoft Windows Azure account (note this methodology will worth with EC2 or any other platform including standard hardware/on-prem VMs)
• IaaS VM Configured. A medium size is recommended for testing 4 disks or fewer to limit the available memory. More on that below.
• Administrator access to your VM.
• Your workload is in fact disk I/O bound. If you're not sure of that you may want to start with this article.
• Awareness that you will incur additional storage transaction costs by running these tests.

Analysis/Create Workload

Note: If you're just trying to get a general sense for your VM I/O performance capability, you don't need to collect data for a custom access specification. IOMeter includes several tests you can use so skip to the "Install IOMeter..." section below.

The first thing we need to do is create our workload. By using IOMeter we can develop custom access patterns that model common workloads and have the tool and workloads installed and configured in minutes on any machine. There is nearly endless information on this topic, so I won't attempt to create a definitive source here. For details on how to configure and use IOMeter, see the following videos/articles:

• Video Tutorial from iTechStorm
• Kristau.net Short Tutorial/Intro
• IOMeter Users Guide
• VMWare Community:Storage System Performance Analysis with Iometer

 To create an accurate workload you will need a good understanding of the access pattern of your application. If you don't have that information you can use a tool like Perfmon to do analysis on a fully configured platform. The following counters will be of interest when creating your access specification:

• Physical or Logical Disk: Average Disk Bytes per Read
• Physical or Logical Disk: Average Disk Bytes per Write
• Physical or Logical Disk: Disk Read Bytes/sec
• Physical or Logical Disk: Disk Write Bytes/sec
• Physical or Logical Disk: Disk Reads/sec
• Physical or Logical Disk: Disk Writes/sec

For further information, see this excellent Technet Article.

By collecting this data during the access pattern you wish to emulate you can accurately estimate (with one caveat) the information needed to create the IOMeter access specification. That caveat is determining the sequential vs. random access pattern of the platform since Perfmon analysis will reveal the rest. To determine that, you'll need an understanding of how the platform stores and accesses/writes data. In my case I'm tuning my VM for Splunk, which uses a Map/Reduce functionality that has a highly sequential read/write pattern. If you are unsure of your access pattern then err on the side of configuring for mostly random access (90% or so) since it is generally more common and demanding of the underlying storage subsystem. 

Install IOMeter and Config Access Specification

The following actions can be done on your target testing platform or a different machine to stage settings. We'll be saving our settings for quick use later.

1. Download and install IOMeter on your server. There are a series of ways to stage files on any VM, but if you're looking for a quick way in the Microsoft ecosystem check out my Onedrive/Azure post.





2. Open IOMeter as administrator.





3. Under "Topology" configure your workers. Each worker represents one thread generating I/O. By default it will create one per CPU thread available, but in most cases you will only want one worker per process you are emulating. In my case I'm assuming one large query at a time (and we'll scale from there), so I'll be testing with one worker. If you are unsure stick to one worker and you can move up from there when you become more familiar.





4. Under "Disk Targets" select the disk you wish to test. This can change in later runs so if the disk you want to test isn't present here select a placeholder.
5. Under "Disk Targets" configure your "Maximum Disk Size". This configures the size of your test file in sectors, which are considered to be 512 bytes each. To lessen the impact of OS caching you need to ensure this value exceeds the amount of RAM present on the machine to be tested. In my case I'll be testing on a 6GB RAM machine with a (approx) 7.5GB file, so I've configured it for 15000000 sectors. (15000000 sectors * 512 bytes per sector=7,680,000,000 bytes)  To do this quickly take your total desired size (in bytes!) and divide it by 512. (If you aren't certain you got it right, check the size iobw.tst file created at the root of your target drive after the first test is complete)





Testing T: With a 4.5GB Test File



6. Under "Disk Targets" configure your maximum outstanding I/O. This varies depending on access spec and OS, but I've had consistent (with real application access) results testing with 16 maximum outstanding I/O on windows. 
7. Under "Test Setup" configure your "Ramp Up Time" and "Run Time". Ramp up need only be about 20 seconds for most scenarios and run time is best between 1 and 10 minutes. My results are based on (many per config) 5 minute tests. 
8. Under "Access Specification" select your access spec. There is far too much to get into here; either select one or many existing access specifications that suit you needs ("4k 75% read" is a good start if you don't care) or create your own based on your findings from the Analysis/Create workload section. For the purposes of my test I made a "_Splunk" access spec with the following characteristics ascertained from my earlier performance testing:
1. Transfer Request Size: 32kB (NOTE: My access spec may not reflect yours. Most won't be this large)
2. Percent Read/Write Distribution: 53% Write/47% Read (NOTE: My access spec may not reflect yours. Most specs won't be this write heavy)
3. Percent Random/Sequential Distribution: 75% Sequential/25% Random (NOTE: My access spec may not reflect yours. Most specs won't be this sequential)




9. Add your access specification to the list of queued tests if you haven't done so already (removing all others).





10. Click the disk icon to save the settings to an ICF file. This file will save all your settings including custom access specifications if applicable. Since this file is what you'll use to shortcut future testing, save it somewhere easy to transfer to other VMs such as OneDrive, Dropbox, SpiderOak, etc.

Run the Test

After setting up or loading your test settings, all you need do is click the green flag to start the test and then select where you would like to save the results. Make sure you don't overwrite any previous results and give the file a meaningful name so you remember what this test represents later, i.e. "results_3disk_1_StorAcct_Striped_32k_sectors_noCache_run1.csv" or similar. 

The test will run for the configured time and then you will be able to run additional tests or analyze results. Since the output is in CSV format, the natural place to look at this data is Excel. When IOMeter starts for the first time on a given disk it needs to create the test file. This will take quite awhile in both Amazon EC2 and Azure. (15 mins for my 7.5G for example) I believe this is due to the way space is allocated on the backend storage. Once this is created, however, you can run subsequent tests on the same volume without needing to wait for the test file to be created. Once the run is done I recommend running several more to ensure your tests aren't subject to wild performance swings. More on analysis in part 2 of this article.

How Much Will This Cost?

Since you're charged by transaction I'm sure you will be wondering how much this will cost. Let's break down your above baseline (system running) cost in Azure:

IOPS are currently capped at 500 for standard tier machines (300 for basic). Storage transactions are currently $0.01 per 100,000. (halved on 3/14/14) For every 5 minute test per disk you access you will then execute a maximum of 150,000 transactions. As a one time per configuration cost, you will need to build the test file which will be (test file size/volume sector size) transactions. For example, a 7.5 Gib test file will be approximately 1,875,000 transactions assuming a default 4kb sector size. (7,500,000,000/4000)

Test transactions + creation transactions = 2 million or so IOPS, or $0.20 @ .01 per 100,000. So... not much. The amount is generally trivial on Amazon EC2 as well. While this methodology will save you some in transaction costs, the main savings will be in time & labor. (which is usually our real cost anyhow!)

Further Optimization

Once you are comfortable with this process I would advise doing the following to further optimize this process. After doing so you may be able to automate the whole routine!

• Create standard Perfmon counter sets for disk access and save/import them as a template. 
• Script the Perfmon analysis with PowerShell. 
• Create or download IOMeter templates for common access routines and include them with your set.
• Script the installation and running of IOMeter, including multiple runs and uploading results to a common location. Easy to do with PowerShell and refer to the IOMeter manual for command line options (page 75 or so).
• Package up all your assets with a custom installer and put it in an easy to get location. (mmmm... Chocolatey)
• If you want angry followers and think digital bits are out there to be wasted, auto tweet your results! (maybe not this)

In Closing

I/O testing in the cloud is certainly feasible but requires a little extra discipline. With several access specifications in your toolkit you can conquer most performance problems quickly. What to do if your cloud platform doesn't provide your desired IOPS? Coming up in part 2!

Windows Server Failover Clustering Quorum Behavior Guide

“A Republic Quorum, if you can keep it.”  -  Ben Franklin.

Your WSFC Quorum is like a Republic, or more accurately, a Democracy. There are many articles out there regarding Quorum voting logic but most are somewhat lengthy. I decided to set out to see how few words I could effectively explain Quorum rules in, so here we go. Don't count this part. Or this. Wait... er... start counting.... NOW.

Windows 2008 and higher:

• A Quorum is the act of n nodes agreeing on a majority. 
• A node is a cluster member, shared disk, or fileshare witness. 
• A cluster can have a shared disk or fileshare, but not both. 
• A majority is defined by greater than 50% consensus. (a tie is not majority)
• Fractions (only) are rounded up to the nearest integer. (2.5=3)
• There is a legacy quorum method called "disk only" wherein one (defined quorum) disk is the only vote. This is considered obsolete because it creates a single point of failure. 

Windows 2008/r2 with Hotfix 2494036 or Higher:

"Nodeweight" was added to revoke a node of its voting privileges (NodeWeight=0). You can use this for nodes in a different site or to ensure that shared disk/fileshare casts the deciding vote. This is generally used in cross-site clusters.

Windows 2012 or Higher:

Dynamic Clustering

"Dynamic Clustering" changes the nodeweight of downed cluster member and effectively reduces the number of participating nodes by one. This works under the following circumstances:

• Prior to the outage, the cluster has achieved quorum (normal under most circumstances)
• Nodes must go down one at a time so the remaining nodes can agree to removed the downed member. If multiple nodes go down simultaneously the dynamic removal will not take place. 

Examples/Illustration:

Closing

Windows Server Failover Clustering is an excellent option for SQL Server, Hyper-V, and other services. Hopefully this understanding of cluster failover behavior enables you to design solutions that better meet the needs of your clients.

Note: It is important to consider how a Quorum is formed when considering patching strategy. 

References

TechNet: Understanding Quorum Configurations in a Failover Cluster
Aeval Shah's Blog: Windows Server 2012 Failover Clustering Dynamic Quorum
Configure Cluster Quorum NodeWeight Settings
Microsoft Support: Cluster NodeWeight hotfix for 2008/r2

Safely Demote a Windows 2008/r2 Core Domain Controller

Have you installed shiny new 2012 domain controllers? Do you have old, musty 2008r2 domain controllers laying around? Do you want some french fries?

Me too. Let's take a holistic look at decommissioning a domain controller. This article will not only walk through the actual demotion process but also list important pre and post steps to any successful domain controller removal. Most of the steps outlined here should actually work on 2000 up, but I've focused on 2008r2. Note that doing this in a production environment is something you may want to do over the course of a couple weeks due to reasons that will shortly become apparent. That said, let's go.

Assumptions

• New domain controllers are installed and ready to go
• Domain Admin Privs
• A backup of the server should something go wrong. Note a snapshot will not suffice; as a general practice one should never use snaps on a DC.

Ensure Replication is Working Correctly

Removing a domain controller can be dangerous if your replication isn't appropriately healthy.Check the following event logs on EACH Domain Controller for issues regarding replication/DNS:

• Application
• Applications and Services Logs->DFS Replication
• Applications and Services Logs->Directory Service
• Applications and Services Logs->DNS Server

If there are any issues, sort them out prior to continuing.

Re-Point Clients

Next we need to ensure all clients are pointing to DNS servers other than the one you intend on taking down. Do the following:

• Update your DHCP scopes with the addresses of the DNS servers you intend on carrying forward
• Update static "clients" (probably servers) with the appropriate DNS settings
• Update devices on your network appropriately as well. 

Your DHCP lease duration will determine how long you need to wait for the next step. The DHCP RFC tells us that clients will attempt to re-up their lease at the halfway point of the lease. Therefore, if your lease is 7 days you'll need to wait 3.5 days from when you make the change (Assuming the DHCP is fully available). Once you think all the clients have been moved, then move to the next step.

Enable DNS Logging on the DNS Controller to be Decommissioned

Now that we think the DNS server isn't used, we enable debug logging to ensure that nobody is referencing this server for DNS look-ups. Avoiding outages is more fun than fixing them. :)

1. Edit the DNS server properties for the server to be decommissioned.
2. Click the "Debug Logging" tab. 
3. Check the "Log packets for debugging" and set a location for the log. Generally the defaults for this application will be sufficient. Click "OK"
4. Restart the DNS server service or reboot the server when your environment permits to start logging.

Wait however long you would like before continuing; the longer you wait the more opportunity you give for devices still pointing to this server to issue a query. You make the call here depending on your environment.

After that wait period: Open the log you specified above; here we'll be reviewing the log for clients that still need their DNS server settings updated. At the top of the log you'll see a couple pages of initialization information that can be safely ignored. After that there will be a log of the DNS queries that the server has received. The first IP address listed in each entry is the IP of the requesting client. You may see some requests from other domain controllers; so long as the other DCs have their "client" settings configured correctly you can ignore these entries as some DC->DC DNS traffic is normal. Taking that into account, look for machines querying this DNS server and re-mediate them prior to moving on.

Transfer FSMO roles off of the DC

Before demoting the DC you need to ensure there are no FSMO roles served by it. I won't cover this in a step by step at this time to keep the article focused. To do this, refer to one of the two following links:

• Microsoft KB: Transfer FSMO roles using the command line
• Technet Wiki: Transfer FSMO roles using the GUI

Demote the DC

We'll use an unattended answer file with dcpromo. You can do this on the command line but it makes it a bit easier to prepare and review the file.

1. Create a new text file. 
2. On the top line, enter

   [DCINSTALL]

3. Enter the following lines after [DCINSTALL]:
   

   
   administratorpassword=NewLocalAdminPassword(required)
   removeapplicationpartitions=yes
   removeDNSDelegation=yes

   You may need to add other options as well, especially if this is the last DC in the domain, your Domain Admin privs are associated with a different account, or similar for the DNS service. Also, this assumes you have a DNSDelegation to remove. If you do not you should take that line out. Refer to this technet article for a full guide.
   
4. Save the file to the domain controller to be demoted.
5. Log on to the DC to be demoted. 
6. At an elevated command prompt (default on core) execute:

   dcpromo /unattend:"(path to file)"

   where path to file=the file you just saved, i.e. dcpromo /unattend:c:\temp\demote.txt
7. After successful execution the machine will successfully reboot. If you intend on re-using the machine for other operations you can log back in and execute dcpromo /uninstallbinaries to remove unnecessary files. If taking the server down don't bother.

Note: If you cancel a demote operation using this file the administratorpassword field will be intentionally blanked out by the installer. You'll need to reset if if you start again.

After the demotion, open up %SystemRoot%\debug\DCPROMO.LOG on the old DC and ensure the operation didn't have any major errors. After confirming the demotion was successful...

Clean Up DNS

After waiting for a replication cycle (should be 15 minutes, but wait an hour to be safe) it's time to ensure AD DNS was cleaned up successfully. Open the DNS manager and point it to one of your new domain controllers. Browse through your zones, including your _msdcs.domain.ext zone. Pay particular attention to the listed NS records in each zone. Those can also be manipulated by right clicking the zone->properties->Name Servers. Ensure only the new DNS servers are listed. Fix any other records that are incorrect; common clean up items are the _msdcs delegation in the primary zone and the sites container under the _msdcs zone. Before making changes to anything other than the the NS records (Name Servers GUI) make sure you fully understand what function the records serve. If you're unsure, this will take some study; here's a good place to start.

It should also be safe to remove the machine from the domain now should you choose to do so. To remove the machine, execute the following on a command prompt:

netdom remove /d:%USERDOMAIN% %COMPUTERNAME%

Also, be mindful that if you were using the old DC in your Windows Time infrastructure you may want to revisit that. Here is an article on how to setup an authoritative time source.

Test

Now that the machine is gone and DNS has been cleaned up, it's time to test your domain. To do so, perform the following tasks:

• Run DCDiag on a new DC with the following command line: dcdiag /e /c and review the output for any issues. 
• Review the event log on each domain controller. Refer to the "Ensure Replication is Working Correctly" section above for the log locations. 

That's it! You've successfully removed a DC and made sure nothing will come back to bite you. If you have questions/comments please feel free to leave the below. 

Additional Reading

 

Technet: Demote a Domain Controller
Technet: DCDiag

Windows 2012 NAP (NPS) with DHCP

After my last article on DHCP I decided to flesh out the Microsoft offerings and expand into Network Policy Server. NPS is actually a set of different features from Microsoft including RADIUS and what the rest of the industry knows as NAC, (Network Access Control) which Microsoft calls NAP (Network Access Protection). This allows you to grant or deny network access to clients based on criteria such as:

• Domain/Group Membership
• Client source network
• Time of day
• Client health (as determined by an SHV, see below for more)

3 out of 4 Doctors are very serious about Microsoft NAP.

These rules must be introduced to the clients at an insertion point (think of it as an entry gate). The supported insertion points include:

• IPSec
• 802.1X
• VPN
• DHCP
• RDGateway

I'll be covering integration with DHCP since it is by far the most cost effective method considering the required role of Microsoft based services in the environment. For information on the others, see Technet: NAP Enforcement Points.First, let's set up NAP:

Assumptions

• At least one Windows 2012 server ready to go. Note that NPS is not supported on server core. Most of these instructions are applicable to 2008/r2 as well.
• DHCP installed and ready to go.
• Sufficient Privileges (Domain Admin generally)
• I assume you'll be installing NAP on the DHCP server itself. You can have these roles on separate servers should you desire, but you'll need to install the NAP piece on the DHCP server as a RADIUS proxy. I won't be covering that piece here. For more on that, see this Technet article.

Installing NAP

1. Using server manager, select "Manage"->"Add Roles or Features"
2. Navigate through the Add Roles and Features Wizard, selecting the target server and "Network Policy and Access Services".





3. When prompted to select role services, you need keep only the default "Network Policy Server" selected and continue through the wizard.

Repeat this step for the DHCP server as well if it's a different server than the one you installed NAP on.

Configuring NAP

The NAC portion of NAP is actually a collection of several different elements. The following elements make up a NAP policy that can be used by DHCP; numbers in front represent the default number of that item for one overall policy:

• Connection Request Policy (Created by Wizard)
• (3) Network Policies (Created by Wizard)
• (2) Health Policies (Created by Wizard)
• Windows Security Health Validator
• At least one Remediation Server Group

There is a (seemingly hidden) wizard to guide you through the process of creating most of these elements, but we're going to create the unguided ones first and then circle back and use the Wizard since we'll want to point to those during the Wizard portion. For all these sections save the DHCP and client sections we'll be working in the Network Policy Server management tool.

Windows Security Health Validator

The Security Health Validator is the policy for defining what a healthy windows client is. The built-in Windows SHV gives the NAP server the capability to interact with the Network Access Protection Agent service on Windows clients (XP SP3 and newer) to determine the health status of that given client. The Security Health Validator determines what criteria the client must pass for the client to be considered healthy. There is a default configuration that can be utilized but for the sake of experience we'll configure the Windows SHV. To do so:

1. First, open the NAP management tool by selecting "Tools"->"Network Policy Server" from Server Manager.
2. Expand "Network Access Protection"->"System Health Validators"->"Windows Security Health Validator"->"Settings"
3. Right click "Settings"->"New"
4. Give the SHV a name; make it something meaningful to describe the target client base, i.e. "Standard Client Set"
5. Set your settings as desired; they're split into settings for Windows XP and Vista or higher. The settings are quite straightforward though it should be noted that if you would like to utilize WSUS to update out of compliance clients (assuming you select being out of date makes for a failed check) you'll need to check the box on the very bottom of the options dialog. Click "OK" when complete.





6. Click on the "Error Codes" under "Settings" and take note of the Error Code Configurations. Generally the default state of "Noncompliant" for each setting is desirable, but depending on your clients and equipment there may be situations where you would want to change some of these SHV check failures to compliant.

Note that third parties can also create SHVs to plugin to the NAP architecture for use with other products. (Old list of some others here)

Remediation Server Groups

The remediation server groups are the servers that will be made accessible to non-compliant clients if you choose to do so. These servers can be used to patch clients to a compliant status. Note that any services needed to contact the remediation services (WSUS, Antivir FTP, etc.) need to be available for the clients to update properly. (DNS, etc.) Under the DHCP enforcement model, these servers are made available via static routes. Also note that you can provide a help URL to a website instructing access-limited clients on how to repair their machines. The server that hosts this URL and all resources required to resolve it must be part of the remediation server group as well. For more information about RSGs, see Technet: Planning the Placement of a NAP Remediation Server. Assuming you have determined which servers need be part of the remediation group, let's set them up:

1. Expand "Network Access Protection"->"Remediation Server Groups"
2. Right Click->"New" (Note that these can be stored as templates for use elsewhere as well)
3. Give the group a meaningful name; note that each health policy can use only one group, so everything for that client base will need be in that group. Something like "Sitename Remediation Group"
4. Click the "Add" button to add a server.
5. Enter a friendly name for this server, i.e. "Minneapolis WSUS 01" or just the server name and then enter the DNS name and click "Resolve". Hit "OK" when complete.
6. Repeat steps 4 and 5 for each server in this group. Then hit "OK" on the "New Remediation Server Group" page.

Setup Remaining Items with The Whiz

1. Click "NPS" on the top of the Network Policy Server management tool.
2. Click "Configure NAP" to launch the secret hidden Wizard.





3. Select "Dynamic Host Configuration Protocol (DHCP)" under "Network Connection Method". Note this is where you would select a different option should you want to use a different insertion point. If you wish to enable this policy on only specific scopes then give the policy a name, I.E. "Minneapolis NAP DHCP".  If you wish this policy to be effective on all scopes do not change it from the default name. Click "Next".
4. On the next screen, click add if the DHCP server is not on the same server as the main NAP server. If that is the case, enter a Friendly Name, Address, and Shared Secret and click "OK". If not, no action is necessary. After this is complete, click "Next".
5. On the next screen, "Specify DHCP Scopes", we need only add scopes if we want this policy to apply to a specific set of scopes. If we do not specify any scopes it will apply to all NAP enabled scopes. Either add the name of all specific scopes to which this policy will apply or just click "Next" with the scopes empty to have it apply to all.





6. The "Configure Machine Groups" screen, like the previous DHCP Scopes screen, is only for limiting access to a set of computers. Should you choose, specify the group(s) of computers you would like to receive IP addresses. In most cases you should leave this blank, but in the event you would restrict via group click "Add" and enter the groups you would like to have access. Click "Next" when you are done with this screen.





7. On the next "Specify a NAP Remediation Server Group and URL" screen select the server remediation group we created earlier. If you would like the clients to have access to a web site describing how to re-mediate their machines enter that under "Troubleshooting URL". Note you must setup this site, it is not included with NAP since the instructions will be different depending on your software selection. After entering the required information, click "Next".





8.  "Define NAP Health Policy" is the final screen. You should see the "Windows Security Health Validator" selected and you can go ahead and enable auto-remediation of client computers with the applicable check box. I'll address this a bit more in closing.
9. As for "Network access restrictions for NAP-ineligible client computers", select whichever you prefer. In most cases if you've bothered to come this far you'll be selecting to deny full network access since that's usually the point. Click "Next".





10. You will be presented with a summary screen; review the information and click "Finish".

Configure Clients

We need to configure two important elements to make the clients functional: Enable the service and configure it to enforce via DHCP. To accomplish that, we'll use group policy. You will need to ensure your group policy objects are targeted appropriately via something like OU linking or security filtering. If you need more information on how to target group policy objects, see this link. Also note that according to some Microsoft documentation, the Wired and/or Wireless Autoconfig services need to be set to automatic, but in my testing they worked when set to manual. Keep that in mind if you have issues.

1. Edit the group policy object you plan to use for NAP client enforcement using the Group Policy Management tool.
2. Navigate to Computer Configuration->Policies->Windows Settings->Security Settings->System Services
3. Double click "Network Access Protection Agent" and check "Define this policy setting" and select "Automatic". Click "OK" to save the setting.





4. Navigate to Computer Configuration->Policies->Windows Settings->Security Settings->Network Access Protection->NAP Client Configuration->Enforcement Clients
5. Double click "DHCP Quarantine Enforcement Client", check "Enable this enforcement client" and click "OK" to save the setting.





6. Provided you're ready, take any steps necessary to apply the GPO to the desired clients. (Link, etc.)

Enable in DHCP

You can either enable globally or on a per-scope basis. I will give the instructions on how to enable globally. If you want to enable on a per-scope basis substitute right-clicking on the "IPv4" below with the scope(s) you desire instead. In that case, you'll need to specify the custom name of the policy you created in step 3 above.

1. Open the DHCP Manager and point it at the server in question.
2. Right click "IPv4" and click "Properties"
3. Click "Network Access Protection"
4. Select what you would like to happen should the NAP server become unavailable, then click "Enable on all scopes".





5. You will be notified that your NAP settings will be overridden on all scopes. Assuming this is what you would like to do, click "Yes".
6. Click "OK".

If you're using a 2012 failover/loadbalance configuration the NAP settings will replicate for each scope, but make sure communication to the NAP server is configured correctly for each DHCP server. That connection information is not replicated. Here is additional info should you need it: Technet: Configure a DHCP Server for NAP. No restart of anything is needed; you should be using NAP now!

Usage Notes/Troubleshooting

• If using a custom profile name for a specific scope, you'll need to provide the custom profile name. This name is the name of the connection request policy you would like to use. For more info see: Configuring Custom NPS Policies Per DHCP scope





• The event log location relevant to NPS authentication is in the security log with the task category of "Network Policy Server". A filtered view of this can be found under "Custom Views->Server Roles->Network Policy and Access Server".
• Know that this solution won't keep out anyone trying to infiltrate your network; anyone with a moderate amount of savvy can take steps to determine what IP to assign themselves should they have physical access to your network.
• What references what?
• * Compliant and * Non-Compliant Network Policies reference Health Policies
• Health Policies reference Security Health Validators
• DHCP scopes reference Connection Request Policies
• You can add several other types of criteria to your policy should you desire. Take a look at your policies under NPS->Policies->Network Policies for more options.
  





• If you enabled auto-remediation, clients will try to repair themselves for simple issues. For example: if a client's firewall is off and the policy requires it, the client will re-enable the firewall and attempt to pass the health check again. Should this fail manual intervention will be necessary. For more information on how to troubleshoot the client, see "Configure NAP Tracing" on Technet.
• Make sure you monitor the load on your NPS servers; the last thing you want is for them to get overwhelmed and prevent proper servicing of DHCP requests. There are some performance counters that can help you with this task; for more info see Technet: Load Balancing with NPS Proxy. Also, be sure to read up on Technet: Best Practices for NPS which covers performance as well as other important info.
• For some information regarding how to use Powershell to configure/manipulate NPS, see my post here.

Prologue

Obviously we're just scratching the surface here; there is quite a bit more that we could dig into but I'm going to stop here in the interest of time. NAC solutions aren't particularly popular right now in a regular office scenario, but as issues continue to arise with malware, etc. more companies may determine these sorts of solutions are necessary. If you already have a Microsoft based infrastructure you probably have nearly everything you need to implement this solution. If you have questions/concerns/comments please feel free to comment. Thanks!

References

Network Policy Server

Network Policy and Access Services Overview

Network Access Protection in NPS

Checklist: Implementing a DHCP Enforcement Design

Networking and Access Technologies: Network Policy and Access Services

Planning the Placement of a NAP Health Requirement Server

NAP Client and Server-side Component Communication