HyperV Live Migration Changes in Windows Server 2016

 

After upgrading my lab servers to Windows Server 2016, I had an “interesting” (ask a Minnesotan what that means) weekend troubleshooting Hyper-V Live Migration, finally finding that there has been a major change in the way virtual machine migration works, and a couple gotchas. In an effort to save others the same trouble, I’ll discuss them here.

Image From Polarstein on Flickr

Kerberos Constrained Delegation, 0x8009030E, and You(r Network Service Account)

“No credentials are available in the security package”, Event ID 20306. Under previous circumstances, this would have indicated that you didn’t have constrained delegation set up correctly as outlined in numerous other articles on the internet, but due to an underlying change the correct configuration is now different.

Previously, failover would be set up as outlined in articles such as this, with each HyperV host set up to allow constrained delegation over the Kerberos protocol only.

Starting in server 2016, the delegation must be set up to allow delegation over any protocol as displayed here:

The reason for this is that 2016 has changed the WMI provider used to a new version, which relies on WinRM to execute remote procedures rather than DCOM. WinRM, running as the Network Service, cannot access the Kerberos service ticket obtained to perform the action. By allowing any protocol, a “S4U” logon is sufficient to authenticate the request. While this setting is somewhat less secure, the point is made by the Team PM (published a few days ago, link below) that sensitive (privileged) accounts in any domain should have the “Account is sensitive and cannot be delegated” flag enabled to mitigate delegation risk.

NIC Teaming, 0x8007274C/0x80072741, and You(r Service Startup Problem)

This may impact 2012/R2 as well, though for some reason it only bit me on 2016. If using NIC teaming on your host for your failover network, the interface may not be available when the Virtual Machine Management Service (VMMS) attempts to start on bootup. This condition will result in the service not opening the port (6600) on the server, which makes it impossible to failover virtual machines. To fix this, change the service startup type from “Automatic” to “Automatic(Delayed Start)”. With PowerShell as our weapon of choice (hey nano server!) this is a two-step process:

Set-Service –Name vmms –StartupType "Automatic"
Set-ItemProperty -Path "Registry::HKLM\System\CurrentControlSet\Services\vmms" -Name "DelayedAutoStart" -Value 1 -Type DWORD

The service type should already be automatic, but we’ll re-assert that here to be sure. This will only delay service (and thus VM) startup by a small bit, but ensure that the adapter is available when it does.

EventID 21024, Failed at Migration Source, and You(r Crazy, Still Unexplained Error)

This is an odd one I can’t fully explain, but I’m including it in the hopes it may save others some time. On 2 of the 3 hosts, I had the following error preventing live migration after full 2016 setup:

Virtual machine migration operation for 'VMNAME' failed at migration source 'VMHOST'. (Virtual machine ID GUID-GOES-HERE)

This error message was not accompanied by any supporting information whatsoever. After numerous network captures and log combing, I found evidence of something slightly off with domain membership. In both cases the host was able to process group policy for the computer object, but never for any logged on users. This led me to attempt leaving and re-joining the domain, which in all cases remediated the problem. Note that when doing so you will need to delete the computer account prior to re-joining, then set up the constrained delegation as outlined above for each host again.

I wish I had more information about the root cause of this issue, but with it fixed I’m moving on.

In Closing

The upgrades to my lab didn’t go as smoothly as I would like, but I’m glad to have these issues out of the way to make for smoother efforts with production efforts. Hopefully this information will help you as well!

Additional References

Microsoft Virtualization Blog: Live Migration via Constrained Delegation with Kerberos in Windows Server 2016

Microsoft GTCS Romania EPS: Shared Nothing Migration Fails

Canberra PFE Team Blog: Kerberos Troubleshooting

Nyan Cat: 10 Hours 4k UHD For Endless Kerberos Packet Caps and Analysis!

What is RemoteFX on Windows 2012 Hyper-V and Deploying a Win 8 Virtual Desktop

Updated 2/3/2017 for Windows Server 2016 and Windows 10!

RemoteFX is Microsoft's advanced desktop RDP solution based on technology acquired from Calista Technologies. It allows for the following functionality above and beyond standard RDP:

This Article Written in RemoteFX. It's... amazing?

Server 2008r2 Based:

• Virtualized 3d GPU: you can split a single GPU into multiple virtual GPUs for VDI VMs. This allows for 3d accelerated apps to run in the RDP session.
• RemoteFX Codec: A more advanced version of the RDP compression scheme that allows for more efficient streaming of both video and audio. While it works fine for text, you can always get groceries in a Yugo. 
• USB Device redirection: Now you can finally use your USB Scan Toaster on your remote desktop!

Server 2012 Additions: 

• Multi-Touch: Multi-Touch support through RDP. Needs a compatible client! (RDP 8.0, etc.)
• Adaptive Graphics: Different compression codecs depending on the content (dynamically on a given screen) among other enhancements. 
• Media Redirection API: Used primarily to facilitate rendering audio and video of VOIP clients local to the client. (Only Lync right now) 
• WAN Enhancements: Support UDP; tweaks to lower bandwidth requirements for both remote and WiFi scenarios.  
• GPU Changes: Support either a software-emulated GPU or a real hardware GPU. DX11 support added. 

Server 2016 Additions: 

• OpenGL/CL Support: Now supports hardware-accelerated OpenGL (^4.4) and CL (^1.1)... CAD/Lightwave/Photoshop users rejoice!
• vRAM Assignment Enhancements: Decouple vRAM amount from monitors and resolution, increased dedicated vRAM per host capability to 1GB from 256MB. 
• Better Performance: More FPS=smoother performance. Noticeable in my case, videos now watchable through RemoteFX without the audio issues that sometimes accompanied the prior version. 
• Generation 2 VM Support: Allows for use of the new VM platform and all the associated features. 
• ... and more, see the reference links at the bottom of this article.

Hardware Requirements:

Full MSFT Article , RDS Blog reqs link

• The standard HyperV Requirements
• SLAT Enabled CPU; EPT on Intel, NPT/RVI on AMD
• DX11 (WDDM 1.2) Compatible GPU; All made for purpose DX11 GPUs from NVIDIA and AMD work, and for testing I've been using a consumer grade NVIDIA 650 in my lab without issue.

Software Requirements:

• Server: Server 2012/2016 (core preferred, see next section)
• Client: Windows 8/10 Enterprise, and Enterprise only. Pro won't work. Additionally, I found that re-keying a Win8 Pro to Enterprise install will not work either. While it will report that all is well, and it will install the RemoteFX graphics adapter, when you try to connect it will report "error during licensing protocol" in the final stage of connection negotiation.

Setup

Assumptions

• You have a fully functional HyperV Host that meets the hardware requirements listed above.
• You're using server core. You can do this on standard, but you should be using core for performance and patching (or lack thereof) reasons. The commands I relay assume core. 

Prepare the Host

1. Patch it up; make sure you're fully up to date. 
2. Install the graphics drivers for your GPU on the host. This may pose some issues on server core, but fortunately I've got a guide for NVIDIA GPUs to help. The procedure is most likely similar for AMD GPUs. Bounce the host after driver install regardless of if it tells you to do so or not. 
3. Install the Remote Desktop Virtualization Host feature. Either use the server manager remotely to do so, or execute the following directly on the Hyper-V host: "Install-WindowsFeature –name RDS-Virtualization -source j:\sources\sxs" where J: is the DVD-ROM or image with the 2012 install bits. You can use other sources if desired, for more information see this post.
4. Reboot the server. 
5. Enable the Adapter for use with the Hyper-V host by opening the Hyper-V management console, right click the server->Hyper-V Settings....
6. Select "Physical GPUs"
7. If configured correctly, you should see your adapter listed. Check the "Use this GPU with RemoteFX" box and click "OK". 

Alternatively for steps 5 through 7 you can use the following powershell cmdlets: 

• Get-VMRemoteFXPhysicalVideoAdapter
• Enable-VMRemoteFXPhysicalVideoAdapter

Setup The VM

1. As noted above, only Win 8 (7 if you want, update + 10) Enterprise will work. Use a template or build from scratch. There are no special needs at build time. 
2. Enable RDP on the Win8 VM
3. Install the VM Integration services & shut down the VM
4. Add a 3D Adapter to the VM by opening the Hyper-V management console, right click the VM->Settings
5. Under "Add Hardware" select "RemoteFX 3D Video Adapter", click "Add", select the max resolution and # of monitors you plan on using via RDP, and select "OK"
6. Start up the VM. You now will need to logon via RDP, as the Hyper-V remoting will be locked out. The machine should notify of new hardware being installed that requires a reboot. If it does not, re-install the integration services. Reboot the VM. 
7. To ensure the VM is operating with the adapter open device manager and look under "Display Adapters". You should see "Microsoft RemoteFX Graphics Device - WDDM"

Alternative Powershell Command:

• Add-VMRemoteFx3dVideoAdapter

Configuration

Note that unlike the previous version of RDP, you don't need to select your connection speed on RDS 8.0. So that said, the only configuration you really need in 8.0 can be done via GPO.

Adjust Settings via GPO

Primer: Modifying GPO settings.

Microsoft makes some RemoteFX settings "Tweakable" via GPO. These settings can be found here:

Computer Configuration->Policies->Administrative Templates->Windows Components->Remote Desktop Services->
  Remote Desktop Connection Client->RemoteFX USB Device Redirection
     -Allow RDP redirection of other supported RemoteFX USB devices from this computer
  Remote Desktop Session Host->Remote Session Environment
     -Configure compression for RemoteFX data
     -Configure image quality for RemoteFX Adaptive Graphics
     -Enable RemoteFX encoding for RemoteFX clients designed for WindowServer 2008r2 SP1
     -Configure RemoteFX Adaptive Graphics (Tweak if using only on a LAN)
     RemoteFX for Windows Server 2008 R2
        -Configure RemoteFX (this just allows for disabling RemoteFX via GPO)
        -Optimize visual experience when using RemoteFX (Tweak this one if using only on a LAN)
        -Optimize visual experience for Remote Desktop Service Sessions (Change to "Text" if you setup RemoteFX so Bobby can do his spreadsheets. Why did you setup RemoteFX again?)

Troubleshooting

 

Sound isn't working on the youtubes! : Seems like flash doesn't work correctly with audio redirection. HTML5 works though! Switch to the HTML 5 beta by going here.

Seems Slow: Make sure your color depth is set to 32bit. Based on my testing it seems that the codec favors full 32 bit color.

FAQ (By me of myself @ least, and by frequently I mean once until I answered them)

• Q: Should I want to, what can I use to benchmark this thing?  A: PassMark works great; I must admit that watching DX11 accelerated 3d benchmarks through an RDP session is kind of cool. :) 
• Q: Does RemoteFX work through an RDP gateway? What versions? A: Yup, it works through a gateway running on 2008, 2008R2, or 2012.
• Q: Is GPU performance good enough to run intensive applications or GPU computation tasks? A: In most cases, no. The RemoteFX solution virtualizes the GPU and exposes it as a Microsoft "GPU", essentially translating all requests through a proxy driver. While that solution allows for splitting the GPU across multiple VMs, it also makes for reduced functionality/performance of the card since the OS isn't exposed directly to the native driver. Additionally, framerate is limited by the RDP protocol itself.
• Q: Does RemoteFX support accelerated OpenGL? A: No, it is done in software.
• Q: Can RemoteFX do GPU direct passthrough? A: No, only Xenserver Enterprise or higher can do that right now. 
• Q: How do I know how much memory on my GPU is being used by my VMs? A: You can find the answer on the Hyper-V/Physical GPUs page for each server. The memory stats are listed right under GPU Details->Summary. 
• Q: Can I deploy RemoteFX in a VDI environment? A: Yeah, though note your max simultaneous RemoteFX sessions will be limited by available GPU memory. (See above post) Fortunately for you, I have a guide on how to setup Hyper-V based VDI!
• Q: Will this VMWare tweak help up the framerate on Hyper-V/RemoteFX? A: In my testing, it did not. I suspect this is because the HyperV NICs probably don't support interrupt coalescing. It looks like one could enable interrupt coalescing on the host with the following instructions and perhaps that would change the answer depending on how the packets are handled from the RemoteFX machine. 

References

Microsoft: What is RemoteFX?
RDS Blog: RemoteFX Features for Windows 8 and 2012
Technet: Frequently Asked Questions and Troubleshooting Tips
RDS Blog: Your desktop will be a rich DX11-based experience, and your virtual GPU should be too
My Crazy Adventures with RemoteFX, Part 1

2016 Updates

Enterprise Mobility and Security Blog: RemoteFX vGPU Updates in Windows Server Next
Technet: Experience guide for Enabling OpenGL Support for vGPU in Server 2016

Questions/Concerns? Contact me or leave a comment!

Installing nVidia Consumer Drivers on Server 2012 Core for RemoteFX

I'm wanting to establish Microsoft RemoteFX in my lab, and to do so, one needs a dedicated 3d accelerator. Obviously, nVidia didn't make the drivers with Server 2012 (now 2016 as well, see below) Core in mind. To accomplish this, we need do the following:

1> Download the newest drivers from nVidia. (Note the later half of these points will probably work on ATI cards as well, you just need to unpack the drivers)
2> Execute the driver setup directly on core server the and select a temporary directory. 
3> Hit "OK"; the installer will crash because it's uncomfortable in the lovely world of server core.
4> Navigate to the display.driver directory underneath the extracted files in the temporary directory you selected earlier. You should find the .inf files in this directory. For nVidia it is nv_disp.if. Update 5/3/2015: the .inf file is now nv_dispi.inf, thanks bearkiter.
5> From cmd.exe, execute "pnputil -i -a nv_disp.inf"
6> The screen will blank! Don't be afraid. After completion, you should see a screen that looks like the shot below. (yes the borders are gone)
7> Assuming it's OK to do so execute "Shutdown /r /t 0" to reboot the machine.

I'm working on another article to cover VDI/RemoteFX. Stay tuned. Update: Article here.

Update 2/3/2017: Confirmed working on Windows 2016! Consumer grade hardware works in the lab as well.