Showing posts with label Hyper-V. Show all posts
Showing posts with label Hyper-V. Show all posts

Sunday, February 5, 2017

HyperV Live Migration Changes in Windows Server 2016

 

After upgrading my lab servers to Windows Server 2016, I had an “interesting” (ask a Minnesotan what that means) weekend troubleshooting Hyper-V Live Migration, finally finding that there has been a major change in the way virtual machine migration works, and a couple gotchas. In an effort to save others the same trouble, I’ll discuss them here.

Image From Polarstein on Flickr

Kerberos Constrained Delegation, 0x8009030E, and You(r Network Service Account)

“No credentials are available in the security package”, Event ID 20306. Under previous circumstances, this would have indicated that you didn’t have constrained delegation set up correctly as outlined in numerous other articles on the internet, but due to an underlying change the correct configuration is now different.

Previously, failover would be set up as outlined in articles such as this, with each HyperV host set up to allow constrained delegation over the Kerberos protocol only.


Starting in server 2016, the delegation must be set up to allow delegation over any protocol as displayed here:


The reason for this is that 2016 has changed the WMI provider used to a new version, which relies on WinRM to execute remote procedures rather than DCOM. WinRM, running as the Network Service, cannot access the Kerberos service ticket obtained to perform the action. By allowing any protocol, a “S4U” logon is sufficient to authenticate the request. While this setting is somewhat less secure, the point is made by the Team PM (published a few days ago, link below) that sensitive (privileged) accounts in any domain should have the “Account is sensitive and cannot be delegated” flag enabled to mitigate delegation risk.

NIC Teaming, 0x8007274C/0x80072741, and You(r Service Startup Problem)

This may impact 2012/R2 as well, though for some reason it only bit me on 2016. If using NIC teaming on your host for your failover network, the interface may not be available when the Virtual Machine Management Service (VMMS) attempts to start on bootup. This condition will result in the service not opening the port (6600) on the server, which makes it impossible to failover virtual machines. To fix this, change the service startup type from “Automatic” to “Automatic(Delayed Start)”. With PowerShell as our weapon of choice (hey nano server!) this is a two-step process:

Set-Service –Name vmms –StartupType "Automatic"
Set-ItemProperty -Path "Registry::HKLM\System\CurrentControlSet\Services\vmms" -Name "DelayedAutoStart" -Value 1 -Type DWORD

The service type should already be automatic, but we’ll re-assert that here to be sure. This will only delay service (and thus VM) startup by a small bit, but ensure that the adapter is available when it does.

EventID 21024, Failed at Migration Source, and You(r Crazy, Still Unexplained Error)

This is an odd one I can’t fully explain, but I’m including it in the hopes it may save others some time. On 2 of the 3 hosts, I had the following error preventing live migration after full 2016 setup:

Virtual machine migration operation for 'VMNAME' failed at migration source 'VMHOST'. (Virtual machine ID GUID-GOES-HERE)

This error message was not accompanied by any supporting information whatsoever. After numerous network captures and log combing, I found evidence of something slightly off with domain membership. In both cases the host was able to process group policy for the computer object, but never for any logged on users. This led me to attempt leaving and re-joining the domain, which in all cases remediated the problem. Note that when doing so you will need to delete the computer account prior to re-joining, then set up the constrained delegation as outlined above for each host again.

I wish I had more information about the root cause of this issue, but with it fixed I’m moving on.

In Closing

The upgrades to my lab didn’t go as smoothly as I would like, but I’m glad to have these issues out of the way to make for smoother efforts with production efforts. Hopefully this information will help you as well!

Additional References

Microsoft Virtualization Blog: Live Migration via Constrained Delegation with Kerberos in Windows Server 2016

Microsoft GTCS Romania EPS: Shared Nothing Migration Fails

Canberra PFE Team Blog: Kerberos Troubleshooting

Nyan Cat: 10 Hours 4k UHD For Endless Kerberos Packet Caps and Analysis!

Tuesday, November 11, 2014

Software Review: Altaro Hyper-V Backup

Backups can be a tough topic to get excited about. In a forward-looking technology field even needing to utilize your backup platform is generally a sign that something went wrong. As a result I haven't delved into the topic of backups for awhile. I'm taking a break from that dry spell to examine a Hyper-V backup solution, Altaro Hyper-V Backup.


Being that my local lab is Hyper-V based, I was able to take Altaro through its paces without impacting a production environment; something affording me the freedom to explore most of the features. Speaking of features (that's a $2 segue) let's examine the high level features offered by Altaro:

  • Support for backup of Hyper-V virtual machines
  • Fully Hypervisor/VSS-based, meaning no worries regarding open files or system state
  • Straightforward GUI-Driven backup schedule and retention policies
  • Offsite replication to another Altaro target (no additional license required) on a defined schedule, honoring retention policies
  • Seeding to offsite location using physical disk if desired
  • Agentless
  • Granular Restore:
    • File System Level
    • Exchange Items
  • "Sandbox" Restore for testing backup images
    • Automated sandbox scheduling
  • Cluster Shared Volume support
  • Boot directly from backup for testing, etc.
  • E-Mail based alerting

Taking the target market (more on that below) into consideration I will set my priorities as follows (in order of importance):

  • Reliability/General Quality
  • Labor investment to set up and maintain
  • Alerting Effectiveness (Not too noisy, alerting when necessary) 
  • Cost
  • Basic Features (Granular Restore, etc.)
  • Advanced Features (VM Sandbox Restore, etc.)

Having spent over half my career in consulting I've been fortunate to see all sorts of environments. In my experience the main constraint for companies targeted by Altaro is usually time (labor) and potentially cost. That coupled with a predominant and understandable lack of enthusiasm for backup solutions makes the ease of setup and management crucial to ensure successful backups. 

Target Market

Altaro Hyper-V backup supports Hyper-V servers and is likely most applicable at small to medium size businesses. Traditional backups of physical servers are not supported, which makes sense given the market gap Altaro aims to fill. The pricing is very competitive (more below) and comes in three versions: Free, Standard, and Unlimited. 

This isn't to say the product may not be applicable in a larger enterprise, but the tools aren't geared to manage a huge number of servers and most enterprises are not exclusively utilizing Hyper-V for their virtualization needs. This review is written with this in mind. As with all things in IT you're looking for the right tool for the job.

Test Systems

I've tested this product in my lab on two Hyper-V hosts, one 2012 R2 Storage Server (with Hyper-V) and one 2012 Core Hyper-V host. While storage server isn't on the Altaro list of supported platforms I experienced no problems with it. Collectively this environment has 16 threads, 64GB RAM, and 12TB of various storage. There are 14 VMs eligible for backup including different Windows versions and two flavors of Linux. For test purposes my offsite backup target was an IaaS VM hosted in Windows Azure (Note: due to cost I do not recommend using Azure IaaS as a backup target in production).

Disclosures/Scoring

When reviewing I feel that full disclosure is critical to maintaining journalistic integrity. Altaro contacted me and asked if I would review their software; in the market myself for lab backup software I agreed to assess it. They provided me with a license to perform the appropriate testing. I did not accept any payment for this article and as always my opinions are my own.

I use a two-axes scoring square similar in design and function to the Gartner "Magic Quadrant". I use this when the testing is subjective and the target market of the product is not universal. This allows me to attempt to assess quality in two ways:

  • Implementation Quality: The quality of the features discussed in this section, but not relevant to the quantity or specific functionality of the features.
  • Feature Set: The quantity and specific functionality of the features discussed in this section, but not relevant to the quality.

Higher on the scale indicates better fit and finish, stability, and general quality. Farther right on the scale indicates a more extensive and useful feature set. The best position would be the top right and the worst the bottom left. It is possible that a given piece of software may have great features but is very buggy, few features but very stable, etc. I believe this scoring methodology allows me to communicate the overall software quality more effectively.

In some cases I will eliminate an axis from scoring if the category warrants it. In this review, "cost" is the only single axis score.

Installation/Basic Configuration

Installation couldn't be much more straightforward; the installer even works on server core. There are no special service account requirements; since the agent runs on each Hyper-V host it utilizes the built-in "SYSTEM" principal. Normally I wouldn't care for the use of the admin-equivalent SYSTEM principal, but since this is backup software there isn't much wiggle room access wise.

After installation on each Hyper-V host you will want to create a centralized management console if you have more than one host. This can be done on a management workstation if you desire; just install the free version on your workstation. After installation, connect to the local agent and select "Central Management Console" from the left pane and click "yes" when asked if you would like to create a shortcut.


From henceforth you can use the central console if desired.

Assuming your firewall rules were created and enabled you should be able to click "Add Agent to Group" to add each host. If you have multiple sites make sure you create appropriate groups for each and then add agents as appropriate. 

Now you can connect to and manage each host in your environment from a central location. Note that you will need to launch a console for each host, but in a small/med environment this should be acceptable as setup is a one-time occurrence per host. For each host you can now configure the following:

  • VMs to backup
  • Primary backup location
  • Backup schedules and retention
  • Offsite backup target (see install note below)
  • Notification
  • Encryption key

Offsite install note: Altaro also provides a WAN target only client that does not count towards your license. Installation is simple; just click through, create a user account for your servers, and open firewall ports as necessary. The software is available here.

When configuring the encryption key, make sure you do so right away as doing so later will invalidate your previous backups; there is no support for rotating and retaining encryption keys, but based on the design that is a good thing (see below as to why). After that you'll want to finish up by specifying your base schedules and retention as well as notifications and offsite details.

Now let's address the review categories:

Reliability/General Quality

As stated, this is the most critical point for backup software. If it were buggy and didn't work consistently it wouldn't be worth using. I used the software for just over a month, performing onsite and offsite backups regularly, restoring full VMs and more granular items, and changing settings to exercise the software. Backup targets ranged from local disk through UNC paths and on to attached, dedicated storage. During that month I did not encounter any issues that concerned me. There were a couple very minor issues such as backups failing if the administrative console had a lock on previous backup files associated with the VM in question; these were to be expected and were addressed by support right away.

Performing a basic restore is almost *too* easy (not that I'm docking points). You can select which VM to restore, if you want it restored to a clone or to the original location, backup sources other than the main, as well as which day you want it restored to. As an added bonus, Altaro can automatically disable the NIC when restoring to avoid IP address conflicts. After a non-overwrite restore Altaro even adds the clone to the console of the Hyper-V host to which you restored! When utilizing any of these options I found the operation quick and encountered no problems whatsoever.

As for software updates, the Altaro console will check for updates and notify when they are released. The GUI provides an upgrade link and takes you directly to the download page. After downloading and running the setup, the installer handled the upgrade process flawlessly, automatically closing open executables and retaining settings. While it would be nice if the console facilitated the download & install directly, the execution of the existing system was flawless and in-line with the best its competitors have to offer.


Offsite backups can be a tricky thing to coordinate and pay for; fortunately Altaro has you covered there as well. Both over the network (internet) and disk seeding is supported in the standard and unlimited versions. Disk seeding allows you to push backups to disk and then send that disk to your offsite location rather than seed the initial full backups, which can be very useful when sending out many, many GBs of data for the first go around. After that you can rely on WAN/internet based uploads for deltas. There is a special software component you use as the backup target that does not require an additional license. This component can be installed on a Windows machine; keep in mind you'll need a network port opened and forwarded to the machine for it to work. To enable offsite backups Altaro requires that you set an encryption key, which is a welcome restriction since we don't want to pump unencrypted machine images over the internet. Once enabled you can schedule automatic offsite backups anywhere between once a month and every day. Note that if selecting the weekly/daily backup option in your schedule for normal backups you're locked to once a week at a minimum for offsite backups. It would be nice to be able to schedule the offsite backups less often in that case, but that's not too big a deal. Retention can be set independently between on and off-site. The only other thing I'd like to see here that isn't in the software at the time of review is offsite upload bandwidth throttling... perhaps in a future version.


Altaro also supports "reverse delta" which ensures that when performing incremental backups the latest backup taken contains the full image rather than the traditional concept of a full backup followed by deltas until the next full backup. This is similar to a feature in Veeam to accomplish the same thing, and one of my favorite things about this product. This means that the net size for cross site replication is limited to the delta size and the concept of regular "full" backups is obsolete. Note you still maintain the ability to restore to an older state; Altaro just down-revs the image as necessary when you request a restore.

Unfortunately there is no backup level de-duplication across multiple VM backups, nor is there support for OS level de-duplication at this time. The lack of de-duplication for regular offsite backups (bandwidth) is mitigated by the aforementioned reverse delta methodology however.

Hyper-V share nothing live migration does pose a couple interesting challenges; since the schedules and previous backups for a particular VM are tied to a host they don't follow a migration to a different host. They do remain on the originating host and work fine after migrating the VM back to the original host. The implications of this are:

  • The originating host will throw errors until the backup schedule for the missing VM is disabled.
  • Restores on the new host will not be available.
  • If you backup the VM on the new host, the initial backup will be full and offsite replication will be impacted as such.

While these issues are theoretically fixable it would be a mess to do so and I suspect it wouldn't impact customers much. The most common use case for share nothing live migration is to maintain uptime across host reboots, transferring the VMs back thereafter. In that case this is not an issue; the impact in your environment will be up to you to judge. 

A more robust Hyper-V environment would likely utilize Hyper-V Cluster Shared Volumes, which Altaro Ultimate edition has native support for.  While I did not test this feature extensively, the support and features is in line with what I would expect. This support is likely more important than the share nothing live migration issues I noted above.

The built in reporting is easy to consume and effective. It is not geared for notification of eternal parties, but barring that use case I don't see much of anything deficiency wise. The dashboard has impressively effective visualizations of backup times and volumes on a per VM or overall basis. The porting section has records of backups, restores, and errors. 

Reliability and General Quality score:


Labor Investment to Set Up and Maintain

This is a category where Altaro really shines. Installation couldn't be much more straightforward, and usage/maintenance is impressively easy. This fact has many positive ramifications on a labor constrained environment and is why I have it ranked so highly. Altaro has an impressive collection of online documentation including user guides, a knowledgebase/FAQ, and they maintain a detailed change log. 

As a demonstration of the ease of use, take for example the setup and association of a backup schedule:

  1. Under the schedules tab, click to create a new schedule.
  2. Define schedule frequency, time, and offsite schedule.
  3. Drag the desired VMs to the schedule.

Most of the software is as intuitive as what you see here. I know I can reasonably delegate responsibility for this tool without having to provided dedicated training or even significant time in most target environments.

Ease of Set Up/Maintenance score:


Alerting Effectiveness



It is critical that backup software alert effectively; a silent failure in this space is a much larger problem waiting to happen. Altaro natively supports individually configuring the following categories for backup notifications:

  • Successful Backups/Offsite Copies
  • Backups/Offsite Copies with Skipped Files
  • Failed Backups/Offsite Copies
  • Completed Restore Operations

The following notification methodologies are supported:

  • E-Mail
  • Windows Event Log

Additionally, anyone connected to the console will receive alerts as well. While e-mail notification will generally be sufficient in most cases, large installations may want to log to the event log and use another tool to consolidate job status and deliver one consolidated report. I found all available notification methodologies to be reliable; the only minor nitpick I have is that the retry interval and thus the failure notification timing cannot be configured.

Alerting Effectiveness score:


Cost

Altaro pricing is very competitive. In my research I've found it cheaper than most of its main competitors, and the beauty of the licensing is that it is very straightforward. As of this writing the standard version is $395 per Hyper-V host (5 guests per maximum, no core/socket limit) and $585 for unlimited (unlimited VMs per guest, no core/socket limit). That price includes one year of upgrades and support, and you can choose to purchase additional years of upgrades and support for 25% of the purchase price ($99/$146). The free version that can backup two VMs per host is, oddly enough, free. The free version does not support some of the more advanced features which is to be expected. For the sake of comparison Veeam Hyper-V essentials 2CPU Socket licenses with a year of maintenance and support cost the following: "Standard" $820, "Enterprise" $1500, and "Enterprise Plus" $2300.

Cost score:


Basic Features

Altaro supports granular file level restore of any drive in a backup image; after selecting the VM you want a file from it loads up a file browser where you can select what files or directories you would like and where to restore them to (Standard and Unlimited). It also supports Exchange message level restore (Unlimited version). This option scans a VM backup image for Exchange databases in the default install locations. If your DB is in a non-standard location you can browse to and then mount it. You are then presented with a list of mailboxes; upon selecting one you can restore messages by their from, time, and/or subject criteria.

Score:


Advanced Features

In addition to the file level and Exchange object level restores, Altaro Standard and Unlimited support restoring to a sandbox.The Sanbox restore is a great start; it allows you to restore & boot selected VMs on a schedule to ensure the backups being taken are successful from a high level. This sort of thing can be a great feature to help business lines gain confidence in your ability to restore mission-critical virtual machines. It would be nice if this feature supported custom notifications (for non-IT interested parties) and eventually restoring multiple VMs to a shared sandbox and then running automated test scripts, but that's likely a big ask at this point in time. Point is they've already got great functionality and are moving in the right direction.

Altaro also supports booting directly from a backup drive (image), but this feature is hamstrung by the fact that it requires the backup image to be neither compressed or encrypted. I can't imagine a scenario where you wouldn't at least want to compress backups, so while this feature looks good on paper it isn't practical in most situations. That said, restoring a VM to a clone is so easy and quick that wouldn't fret too much about it. Frankly if I were Altaro I would consider removing this feature; not because it's not interesting but because assuming compression is the norm it isn't practical and could serve to disappoint a customer who thought too much of this feature on paper.

Score:


Bonus Super Feature: Encryption

The concept of "Trust No One" relies on software allowing one to specify their own key (or seed) to protect data. If implemented correctly then only the individual(s) with the key would be able to decrypt the data. Adherence to this concept can be critical for some businesses trusted with protecting not only their data but the data of their customers. I'm pleased to say that Altaro, at least in theory, supports "Trust No One" style AES encryption for both local (ultimate only) and offsite (standard & ultimate).

First a note: while the theory of trust no one results in unbreakable encryption, the actual security relies on the details of the implementation. While the key is specified by you, you still are actually trusting that the software doesn't intentionally or unintentionally store the encryption key in a way that could reveal it to anyone else. That can be difficult and I have no way to examine the implementation specifics at a code level.

That said, I'm very impressed that Altaro takes this seriously. One should be able to rest much easier knowing that if their backups are compromised their data is safe. To inquire about the implementation of this encryption, I asked the following question of Altaro support:

"If one enables onsite and remote encryption and provides their own key, but then somehow loses that key, is there any way to recover the backups? "

I received this response within hours:

"Hi Toby,
With regards to encrypted backups I’m afraid that the key cannot be recovered in any way and must thereby be remembered. This is designed this way in order to ensure that the sensitive information contained within the backups doesn’t fall into the wrong hands.
"

Perfect answer! This illustrates to me that not only is the software designed to be secure, but the support department is aware of the fact that it is and also aware of why. This makes me much more comfortable with offsite backups in particular. This to me is one of the most important aspects of Altaro; the fact that they get this right gives me a huge confidence in their overall design.

Note: Remember that as stated this encryption is optional and not enabled by default. If you don't want to encrypt you don't need to worry about storing a key. 

Conclusion

When I set out to do this review I was afraid I would be one of two things: disappointed or bored. The quality of the software prevented me from being disappointed and the ease of use prevented me from being bored. This tool is a powerful backup solution for a small to medium size business implementation of a Hyper-V based private cloud, and the pricing is right in line, if not below that level. Additionally, Altaro's support department was impressive and their documentation is thorough. There are a few missing features, but taking the full picture into account I consider those minor. I have no reservations recommending Altaro for the Hyper-V based small to medium sized business.

Sunday, February 9, 2014

Windows Server Failover Clustering Quorum Behavior Guide

“A Republic Quorum, if you can keep it.”  -  Ben Franklin.

Your WSFC Quorum is like a Republic, or more accurately, a Democracy. There are many articles out there regarding Quorum voting logic but most are somewhat lengthy. I decided to set out to see how few words I could effectively explain Quorum rules in, so here we go. Don't count this part. Or this. Wait... er... start counting.... NOW.

Windows 2008 and higher:



  • A Quorum is the act of n nodes agreeing on a majority. 
  • A node is a cluster member, shared disk, or fileshare witness. 
  • A cluster can have a shared disk or fileshare, but not both. 
  • A majority is defined by greater than 50% consensus. (a tie is not majority)
  • Fractions (only) are rounded up to the nearest integer. (2.5=3)
  • There is a legacy quorum method called "disk only" wherein one (defined quorum) disk is the only vote. This is considered obsolete because it creates a single point of failure. 

Windows 2008/r2 with Hotfix 2494036 or Higher:

"Nodeweight" was added to revoke a node of its voting privileges (NodeWeight=0). You can use this for nodes in a different site or to ensure that shared disk/fileshare casts the deciding vote. This is generally used in cross-site clusters.

Windows 2012 or Higher:


Dynamic Clustering
"Dynamic Clustering" changes the nodeweight of downed cluster member and effectively reduces the number of participating nodes by one. This works under the following circumstances:
  • Prior to the outage, the cluster has achieved quorum (normal under most circumstances)
  • Nodes must go down one at a time so the remaining nodes can agree to removed the downed member. If multiple nodes go down simultaneously the dynamic removal will not take place. 

Examples/Illustration:



Closing


Windows Server Failover Clustering is an excellent option for SQL Server, Hyper-V, and other services. Hopefully this understanding of cluster failover behavior enables you to design solutions that better meet the needs of your clients.

Note: It is important to consider how a Quorum is formed when considering patching strategy. 

References


TechNet: Understanding Quorum Configurations in a Failover Cluster
Aeval Shah's Blog: Windows Server 2012 Failover Clustering Dynamic Quorum
Configure Cluster Quorum NodeWeight Settings
Microsoft Support: Cluster NodeWeight hotfix for 2008/r2

Sunday, August 18, 2013

Hyper-V Port Mirroring and Network Capture

Introduction

Hyper-V port mirroring, introduced in Windows Server 2012, allows you to easily monitor traffic on virtual machines without having to capture the traffic directly on that VM. It's exactly the same thing as standard port mirroring, only we're doing it using a virtual switch rather than a physical one. You can use this in your Hyper-V environment for all sorts of network troubleshooting. It only takes a couple of minutes to setup, so let's do it and then capture some packets!

Assumptions

  • Administrative Access to 2012 Hyper-V Host/Cluster
  • At least one vSwitch already configured (though we'll cover multiple)
  • At least two VMs configured; one to monitor and one to collect the data

Set Up Port Mirroring (GUI)


Configure Machine to be Mirrored

  1. Open the Hyper-V Manager
  2. Right click the machine that you would like to capture from and select "Settings".


  3. Expand the properties of the NIC (Network Interface Card) you would like to mirror by clicking the plus sign to its left and click "Advanced Features". 
  4. Under "Port mirroring"->"Mirroring Mode" click the drop down and select "Source". This sets this NIC as the source of mirroring on the Hyper-V switch it is connected to.


  5. Make note of the vSwitch the NIC is connected to (right below "Network Adapter") and click "OK".

Configure the Mirror Target


It makes setting up a network capture substantially easier if you add a dedicated NIC for each source machine. This NIC must be on the same virtual switch. A dedicated NIC allows unbinding all services/protocols in the guest OS, which will facilitate an entirely clean capture. More on that below...

  1. Shut down the virtual machine you intend on being the target of the network capture.
  2. After shutdown, right click that machine and select "Settings".
  3. Under "Add Hardware" in the right hand plane select "Network Adapter" and click "Add".
  4. The network adapter properties page for the new NIC will come up. Under "Virtual switch:" select the same switch that the source machine/NIC is connected to.
  5. Expand the properties of the new NIC and select "Advanced Features".
  6. Under "Port mirroring"->"Mirroring Mode" click the drop down and select "Destination". This sets this NIC as the source of mirroring on the Hyper-V switch it is connected to. Click "OK".


  7. Start the VM back up.

Configure the Mirror NIC in Capturing VM OS


Note: The instructions for this portion are somewhat generic because your guest OS on the capturing VM may differ from mine.

  1. After the capturing VM starts back up, log on via RDP or otherwise.
  2. Open your network connections and determine which NIC is the added one for mirroring. Rename it something like "{VSwitchName} Port Mirror" for easy identification.
  3. Open the properties of that NIC.
  4. Un-bind all protocols and services from that NIC and click "OK". By removing all bindings we'll be able to ensure a clean capture without interfering with the existing network connection. None of the standard protocols or services are used in the mirror process; Hyper-V takes care of everything for you. If you already have your network sniffing software installed, you may need to reboot the capture machine in order to see the NIC.


Install and Use Packet Capture Software


I'll be using Wireshark for Windows; if desired you could substitute something like Microsoft Network Monitor or Microsoft Message Analyzer on Windows; Wireshark or tcpdump on Linux. 

  1. Download and install Wireshark. The portable version works just as well if you prefer.
  2. Open Wireshark and click the "Interface List" button on the upper left hand corner.


  3. Select the dedicated capture NIC (which we renamed earlier), ensure it is the only selected, and click "Start".


  4. Enjoy all your packets.

Powershell Commands for NIC Setup/Mirroring

Caveats

  • You won't be able to decrypt encrypted packets unless you get the private key from the target server for decryption, which obviously may be a security issue given we're not on that machine.
  • Make sure you de-config (new word!) the port mirroring in HyperV when you're done as the packet replication continues even if you're not capturing.
  • After unbinding all services/protocols in Windows the adapter won't appear in the "Network and Sharing Center" anymore. You'll have to click "Change adapter settings" to get to the NIC.

More Reading



That's it, easy eh? Questions/Comments, leave 'em below!

Friday, May 17, 2013

OpenVPN on CentOS Running on Hyper-V Using Windows PKI

I have a knack for titles eh? (Just smile and nod) I've been very impressed by the scalability demonstrated by OpenVPN in applications such as proXPN. Because of this I decided to implement OpenVPN on my favorite Hyper-V friendly distro, CentOS (6.4) and utilize a Windows PKI infrastructure for issuing certificates. Wanna do that? Let's:


Assumptions

  • You're familiar with the basics of PKI and how to process certificate requests. For more information on how to setup Windows PKI, see my article here.
  • You're familiar with the basics of Linux administration including editing config files, etc. 
  • You'll use sudo where applicable. I don't put it on command references to keep them cleaner. 

VM Creation

Create a VM with the following minimal virtual hardware: 
  • 384MB RAM
  • 1 Virtual Proc
  • 10GB HD
  • 2 NICs, (native) one either public/DMZ and one on your internal network (or inside the same DMZ if you desire)
It's possible to setup OpenVPN with 1 NIC, but in most configurations it's less secure and I won't be covering it here.

CentOS Setup

  1. Download from here; minimal is all you'll need.
  2. Insert the ISO to the virtual optical drive on the VM and start it up.
  3. Install CentOS to the VM with desired settings; no special configuration is necessary.
  4. Setup the NICs with static IP addresses. ("/etc/sysconfig/network-scripts/ifcfg-eth0" and "eth1", more instructions here) The following combinations for connectivity are possible: External/Internal, DMZ/Internal, External/DMZ, DMZ/DMZ. While different options have different pros and cons I'm not going to say what you should use because your circumstances will dictate your direction. I am using DMZ/Internal.
  5. Enable ipv4 forwarding by editing "/etc/sysctl.conf" and setting the following value
    net.ipv4.ip_forward = 1
    (It's there and 0 by default)
  6. Enable the EPEL so we can grab OpenVPN from it. Ensure you get the correct version, it was 6.8 as of this writing.
    rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
  7. Make the directory for the keys:(We'll copy keys into this dir before install)
    mkdir /etc/openvpn/keys  
  8. Make sure SSH is installed; if not:
    yum -y install openssh-server
  9. Perform any other tasks you should find applicable to a OpenVPN server such as configuring auto-update (yum install vixie-cron -y needed on CentOS 5 minimal install) or installing any other supporting apps you should want. Just remember that any application you add is a potential security liability.

Create Certificate Templates


I won't be covering the basics of setting up certificate templates; for that see my article.We'll just cover the specifics of this implementation here.

Server Template

  1. Open the certificate template manager for you enterprise PKI, and duplicate the Computer template. 
  2. When prompted, select the "Windows 2003 Server, Enterprise Edition" and hit "OK". (The version of the OS determines the template version) 
  3. Change the template display and name to something sufficiently descriptive such as "Computer (CompanyName OpenVPN Server)"
  4. Up the Validity period should you desire. 
  5. On the "Request Handling" tab set the private key to be exportable. (We're putting this on a Linux server afterall)


  6. On the "Subject Name" tab change the option to "Supply in the request". 
  7. On the "Issuance Requirements" tab check that the template requires CA certificate manager approval should you desire. 
  8. Change to the "Extensions" tab, select "Application Policies" and click "Edit". Change it so that the "Server Authentication" is the only listed policy. Do not mark the extension critical. 
  9. Everything else can be left the same unless you should want to change the permissions under security. 

Client Template

Repeat the steps for the Server template except change the name in 3: to a client name and 8: to "Client Authentication".

After creating your templates publish them to your issuing CA. Again, see my article for instructions.

Enroll For Server Certificate

Since this step is virtually the same for the client I'll be redirecting you here for that step as well. There are a few modifications listed to these steps to accommodate that; don't pay any attention to "For Clients:" on the first go-around.

  1. Open the MMC (mmc.exe) and add the certificates snap-in for the local computer.  This can be done on any machine since we'll be exporting it.
  2. Enroll in the server certificate template we created earlier. We need to specify a few options, so click "More information is required to enroll for this certificate.  Click here to configure settings." For Clients: use the client template
  3. On the subject tab under Subject name, set the "common name" to the FQDN of the OpenVPN server. If you're using split DNS, make sure you set this to the external FQDN. For Clients: use your domain FQDN and set the OU to something consistent for all certs so that you can use the tls-verify option down the road should you choose. I use "vpn.company.com ClientCert"


  4. On the "General" tab set the friendly name and the description to whatever you desire. I copy the common name and append "OpenVPN server"
  5. On the "Extensions" tab change "Key Usage" to include only "Key agreement" and "Key encipherment". "Make these key usages critical" can be checked. Based on the template it should already have "Extended Key Usage" set to "Server Authentication". For Clients: Use "Client Authentication" rather than server authentication.


  6. Under "Basic Constraints" click "Enable this extension" and "Make the basic constraints extension critical"
  7. Under the "Private Key" tab select "Allow the private key to be archived" should you desire.
  8. Click "OK" and then "Enroll". After the enrollment finishes you will need to click "Finish"
  9. Go to your CA and complete the enrollment process, then export the public key and import it back on the requesting computer to complete the certificate request process. 

Export and Split the Certificate

Like the section above, this will be used for the client as well. See For Clients: for that.
  1. Using the MMC or command line/powershell, export the newly created certificate WITH the private key to a pkcs12 format. Do not include all certificates in the certification path. Make sure you remember the password.
  2. Use OpenSSL (you'll need to download/install from here) to convert the pfx file to standard text, then cut and split the file into a cert file and a key file. I'd walk you through this, but how about I give you a Powershell script to do it instead. :)  Make sure to set the variables at the top of the file to what suits you. If you don't want to use powershell, just follow the comments and the commands to convert and split the file. For Clients: You may want to change the variable definitions at the top of the file to reference client file names.

#set the output files, working file, and password for the PFX
$inputFile="c:\temp\server.pfx" 
$outputPublicKey="c:\temp\serverCert.cer"
$outputPrivateKey="c:\temp\serverCert.key"
$tempFile="c:\temp\temp.txt"
$password="password"


#Set the location to the Openssl.exe file
$openSSLLoc="C:\Program Files\OpenSSL-Win32\bin"


#Execute OpenSSL to convert the PFX file to standard text
& $openSSLLoc\OpenSSL.exe pkcs12 -in $inputFile -nodes -out $tempFile -passin pass:$password


#Now we need to find where the key and cert start and end. 
#For non-script users the point here is that we chop this one text file into two files,
# one the public key and one the private key. The two of those are denoted by the patterns listed below. 
$beginKey=get-content $tempFile |select-string -Pattern "-----BEGIN PRIVATE KEY-----"
$beginKey=$beginKey.LineNumber
$endKey=get-content $tempFile |select-string -Pattern "-----END PRIVATE KEY-----"
$endKey=$endKey.LineNumber
$beginCert=get-content $tempFile |select-string -Pattern "-----BEGIN CERTIFICATE-----"
$beginCert=$beginCert.LineNumber
$endCert=get-content $tempFile |select-string -Pattern "-----END CERTIFICATE-----"
$endCert=$endCert.LineNumber
$length=Get-Content $tempFile |Measure-Object
$length=$length.Count


$key=Get-Content $tempFile |Select-String "-----BEGIN PRIVATE KEY-----" -Context 0,($endKey-$beginKey)
$cert=Get-Content $tempFile |Select-String "-----BEGIN CERTIFICATE-----" -Context 0,($endCert-$beginCert)


#Write out the files. Make sure they're ASCII encoded!
$key.line| Out-File $outputPrivateKey -Encoding ASCII
$key.Context.DisplayPostContext | Out-File $outputPrivateKey -Encoding ASCII -Append
$cert.line| Out-File $outputPublicKey -Encoding ASCII
$cert.Context.DisplayPostContext | Out-File $outputPublicKey -Encoding ASCII -Append


#Get rid of the temporary working file. Not needed for those not using the script. 
Remove-Item $tempFile -Force

Even if you can't run that hopefully you can decipher what's going on. Let me know if you would like any help.



Export The CA Certs and Copy to Server

For Clients: You can use the CA cert you created earlier for the server on the client as well. Just include it in the keys directory with the other certs. 
  1. Using the MMC or cmdline/powershell, export the root and (if applicable) intermediate certs to base-64 encoded x.509. For specifics, see here.
  2. If using 2 or more tier PKI: You can only specify one CA cert so if you have more than one CA in your chain you need to "stack" them. To do so, pipe both into one file with the simple command from a command prompt: "type RootCA.cer IntermediateCA.cer > CAs.cer" where rootca.cer and intermediateca.cer are the exported certificates. Substitute cat for type if you're doing this on a Linux platform. This will join any listed certs into one chain for use with OpenVPN. 
  3. (For Clients: This step is not necessary.) Generate Diffe-Helman by executing
    openssl genpkey -genparam -algorithm DH -out dh.pem 1024
    This can be done on either the Windows machine you are on or directly on the OpenVPN server. Either way, just make sure you copy the dh.pem to /etc/openvpn/keys/ on the OpenVPN server. (see step below)
  4. Copy the server private key, public key (both created earlier) and (stacked) ca public key to "/etc/openvpn/keys" on the OpenVPN server. These keys will be referred to as server.cer, server.key, and ca.cer in the future. In order to do security correctly, make sure permissions are cranked down on these files as much as possible, especially the server.key file. (private key) On Windows I use pscp.exe which ships with Putty. For Clients: Just copy the certs/keys into a temp directory for now, we'll move them to the final location later.

Install/Configure OpenVPN on Server

  1. Make sure you enabled EPEL above and then execute the following to to install OpenVPN and LZO:
    yum install openvpn lzo -y
  2. Copy the sample configs as a starting point:
    cp /usr/share/doc/openvpn-2.2.2/server.conf /etc/openvpn/
    (Make sure the version number in the /usr/share/doc path hasn't changed since I wrote this) 

  3. Edit /etc/openvpn/server.conf and pay attention to the following lines: 
    1. Set the ipaddress to the address of the NIC you intend to bind to. This should be the NIC closest to the internet, be it an external or DMZ NIC. 
      local 10.1.0.1
    2. Set the port to have OpenVPN listen on. 1194 is default and a firewall rule below assumes that, so be aware if you change it.
      port 1194
    3. Protocol, UDP vs. TCP. UDP is faster and the client machine will already be managing TCP retrans should they be needed. 
      proto UDP
    4. A bit on the the difference between tun and tap: tun is a layer 3 IP routing based solution, whereas tap is a level 2 based solution. Tap can shoot non-routable protocols over the tunnel, so if you're going to play Descent II this is the one you want. The problem (or not) with using tap for standard client connectivity is besides being more complex to setup it also routes broadcast traffic. This won't scale well. Generally tap is used for site-to-site connections. If you want to use tap, you'll need to setup Ethernet bridging. This guide assumes dev tun.
      dev tun
    5. Specify the keys we copied over earlier:
      ca /etc/openvpn/keys/ca.cer
      cert /etc/openvpn/keys/server.cer
      key /etc/openvpn/keys/server.key
    6. Specify Diffe-Helman copied over earlier:
      dh /etc/openvpn/keys/dh.pem
    7. Set the subnet you want to route clients to. This makes that network route through the OpenVPN connection. Note that if your IP range is the same on the close end you'll have problems. Feel free to add multiple lines for multiple subnets.
      push "route 192.168.1.0 255.255.255.0"
    8. Set the internal DNS servers you want to push to the clients to facilitate looking up servers on the other side of the VPN. Add one line for each DNS server you would like in the search order of the client. 
      push "dhcp-option DNS 192.168.1.30"
    9. If you uncomment client-to-client OpenVPN will allow clients to talk to each other. You'll also need to config firewall rules appropriately should you wish to do this.
      client-to-client
    10. Set the maximum number of simultaneous clients.
      max-clients 100
    11. Uncomment the following lines to increase security on 'nix systems by lowering privilege level after program launch.
      user nobody
      group nobody
  4. Set OpenVPN to start on boot:
    chkconfig openvpn on
  5. Configure the IPTables firewall; assuming eth0 is your internal NIC and eth1 is your external NIC. Change the listed NICs in each rule if that assumption is not correct!
    1. Edit the IPTables rule by editing /etc/sysconfig/iptables . There are other ways to accomplish IPTables editing, but since I make few changes I would rather just work directly on the config.
    2. First, change your SSH rule to allow port 22 on the internal NIC only; you should already have a rule listing --dport 22. Find it and change it to:
      -A INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT"
    3. Allow VPN connections on the external interface. If you have changed from the default port make sure to change the port 1194 reference listed here: " -A INPUT -i eth1 -m state --state NEW -m udp -p udp --dport 1194 -j ACCEPT"
    4. Setup forwarding rules for this network:
      -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT"
      -A FORWARD -s 10.8.0.0/24 -j ACCEPT
    5. Make sure the rules from 2 and 3 are listed before the -A INPUT -j REJECT and the rules from 4 are listed before the -A FORWARD -j REJECT default rejection rules or they'll be ignored. 
  6. Reboot the box and ensure that OpenVPN starts successfully. Logs will be written to /var/log/messages. If you have problems and need to troubleshoot you can up the "verb" value in /etc/openvpn/server.conf.
Alternatively, you may see rules listed such as iptables -A INPUT -i tun0 -j ACCEPT and iptables -A FORWARD -i tun0 -j ACCEPT listed elsewhere; these would work in lieu of the 10.8.0.0 rule, but there are security tradeoffs. Feel free to try either if it suits you. For more information about iptables, see this.

Enroll for Client Certificate


For this section you can follow the same instructions as the section "Enroll for Server Ceritificate" with the modifications listed within those points. (Mods in point 2, 3, and 5)


After doing this refer to the Export and Split the Certificate section and use that logic to do the same task to the client cert as well.

Export the CA Certs and Copy to Client

You can use the CA cert you created earlier for the server on the client as well. Just include it in the keys directory with the other certs.

Install/Configure OpenVPN on Windows Clients

  1. Download and install from here. 
  2. In the install directory, copy the client.ovpn file from the sample-config folder to the config folder for a starting point.
  3. Copy the keys we created earlier to a folder on the client; I use (OpenVPN Install Directory)\Keys.
  4. Edit ".\config\client.ovpn" and pay attention to the following lines: 
    1. Again, we're going to cover tun here. More work will be necessary if you want to do tap. See the server section for an explanation of what these options are. 
      dev tun
    2. Enter the external FQDN and port for your OpenVPN server.
      remote vpn.company.com 1194
    3. List your certs copied over earlier. Make sure you set the directory correctly and protect that directory relative to the fact that a private key resides within it. On windows you need to use double backslashes because "\" is an escape character. 
      ca C:\\InstallDirectory\\Keys\\ca.cer
      cert C:\\InstallDirectory\\Keys\\clientcert.cer
      key C:\\InstallDirectory\\Keys\\clientcert.key
    4. Remark out ns-cert-type server and add remote-cert-tls server in its place. ns-cert-type is deprecated and the template wasn't updated. See here for more info.
      # ns-cert-type server
      remote-cert-tls server
  5. After saving the file, launch OpenVPN GUI as Administrator.


  6. Connect and enjoy!

Closing

Obviously given the scope of this article there is substantially more depth one could go into; if you have any questions please hit me up in the comments. I'm sure you can see the possibilities of this combination as there is much more automation that could be done. Thus far I've been very impressed by the performance of OpenVPN and I look forward to scaling it out in the future.

References/Additional Reading

Monday, April 1, 2013

CentOS ready for Hyper-V 2012 NICs

CentOS 6.4 was released on March 13th, and per the release notes it now ships with the Hyper-V components. Now the native network adapters work right from the base install, even on the minimal distribution. All that need be done is enable the adapters after install. To do so, do the following:


  • Log in as root
  • Navigate to /etc/sysconfig/network-scripts
  • Each NIC config is located in this directory. The configs are named ifcfg-ethX where X is a number starting with 0 and incrementing by 1 for each adapter, i.e. ifcfg-eth0, ifcfg-eth1. Edit each by typing "vi ifcfg-eth0"
  • At the least, you will need to edit the file to change ONBOOT=no to ONBOOT=yes by using the following keys:
    • INSERT
    • (make your modifications)
    • ESC
    • :w
    • :q
  • Reboot
If you want to make further modifications such as static IP, etc, then refer to this guide. Feel free to post questions!

Update 12/8/2013: 

When updating/rebuilding my Hyper-V hosts from 2012 to 2012 r2 the MAC addresses seem to have changed. (Perhaps a good reason to set static MAC addresses on these VMs) You can see if your install exhibits this behavior by examining the following:
  •  cat /var/log/dmesg | grep -I "net"
and looking for:

hv_netvsc vmbus_0_13: Device MAC XX:XX:XX
and
udev: renamed network interface ethX to ethX

To fix it, edit the ifcfg-ethX file as noted above and change the HWADDR entry to match the new MAC, and then remove the /etc/udev/rules.d/70-persistent-net.rules file and reboot.