itToby: November 2014

Saturday, November 15, 2014

Microsoft KB 2992611 "Winshock": More Thank You Bargained For

(updated 11/18 for re-issue, see below)

Microsoft released one of the most important patches in many years on Tuesday, and while I would advise you install it right away I also want to make you aware of some odd behavior I found that could lead to problems. First, a primer:

Here are the release notes and here is more information. This is the worst kind of exploit there can be; a remote code execution with no workarounds. If one knew the details, they could easily exploit any Microsoft based internet facing server supporting TLS and then turn around and use it to infect unpatched Windows based clients. Obviously you should patch immediately.

That said, you will notice that they mention the addition of four new cipher suites but there is one other change that may impact you that is not mentioned. I've found that this patch also re-orders the cipher suites. Historically Microsoft has notified customers when re-ordering cipher suites; see KB2919355 for example.

This is important to understand for two reasons, one theoretical and one practical.

Theoretical is that changing cipher suites impacts your security posture, and one should always know these things going into a patch. Fortunately most of the re-order does seem in line with a tighter security policy.
Practical is that this can break connectivity with some applications. Specifically, one of my peers found that Java 6 based applications attempting purposely or otherwise to use the ECDH key agreement protocol will fail to connect. This happens when Windows based services present ECDH before the older RSA. Side note: Oddly the Microsoft JDBC driver tries to negotiate SSL even if it isn't being used for a connection to SQL.

Here are the cipher suite details, first 2008 R2:

2008 R2 Default Before KB299261	2008 R2 Default After KB299261
TLS_RSA_WITH_AES_128_CBC_SHA256	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256
TLS_RSA_WITH_AES_128_CBC_SHA	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384
TLS_RSA_WITH_AES_256_CBC_SHA256	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P521
TLS_RSA_WITH_AES_256_CBC_SHA	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P521
TLS_RSA_WITH_RC4_128_SHA	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P521
TLS_RSA_WITH_3DES_EDE_CBC_SHA	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P521
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256	TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384	TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256	TLS_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384	TLS_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256	TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P521
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384	TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P384
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256	TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P521
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256	TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P521
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P384
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P521
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256	TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P521
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P521
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256	TLS_RSA_WITH_NULL_MD5
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384
TLS_DHE_DSS_WITH_AES_128_CBC_SHA	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384
TLS_DHE_DSS_WITH_AES_256_CBC_SHA	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384
TLS_RSA_WITH_RC4_128_MD5	TLS_RSA_WITH_AES_256_CBC_SHA256
SSL_CK_RC4_128_WITH_MD5	TLS_RSA_WITH_AES_128_CBC_SHA256
SSL_CK_DES_192_EDE3_CBC_WITH_MD5	TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_NULL_SHA256	TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_NULL_SHA	TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384
	TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256
	TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384
	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256
	TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256
	TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384
	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256
	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384
	TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
	TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
	TLS_DHE_DSS_WITH_AES_256_CBC_SHA
	TLS_DHE_DSS_WITH_AES_128_CBC_SHA
	TLS_RSA_WITH_3DES_EDE_CBC_SHA
	TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
	TLS_RSA_WITH_RC4_128_SHA
	TLS_RSA_WITH_RC4_128_MD5
	TLS_RSA_WITH_NULL_SHA256
	TLS_RSA_WITH_NULL_SHA
	SSL_CK_RC4_128_WITH_MD5
	SSL_CK_DES_192_EDE3_CBC_WITH_MD5

And 2012 (not R2):

2012 Default Before KB299261	2012 Default After KB299261
TLS_RSA_WITH_AES_128_CBC_SHA256	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256
TLS_RSA_WITH_AES_128_CBC_SHA	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384
TLS_RSA_WITH_AES_256_CBC_SHA256	TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
TLS_RSA_WITH_AES_256_CBC_SHA	TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
TLS_RSA_WITH_RC4_128_SHA	TLS_RSA_WITH_AES_256_GCM_SHA384
TLS_RSA_WITH_3DES_EDE_CBC_SHA	TLS_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256	TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384	TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384	TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256	TLS_RSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384	TLS_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256	TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384	TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256	TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384
TLS_DHE_DSS_WITH_AES_128_CBC_SHA	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256	TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256
TLS_DHE_DSS_WITH_AES_256_CBC_SHA	TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256
TLS_RSA_WITH_RC4_128_MD5	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384
SSL_CK_RC4_128_WITH_MD5	TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
SSL_CK_DES_192_EDE3_CBC_WITH_MD5	TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
TLS_RSA_WITH_NULL_SHA256	TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_NULL_SHA	TLS_DHE_DSS_WITH_AES_128_CBC_SHA
	TLS_RSA_WITH_3DES_EDE_CBC_SHA
	TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
	TLS_RSA_WITH_RC4_128_SHA
	TLS_RSA_WITH_RC4_128_MD5
	TLS_RSA_WITH_NULL_SHA256
	TLS_RSA_WITH_NULL_SHA
	SSL_CK_RC4_128_WITH_MD5
	SSL_CK_DES_192_EDE3_CBC_WITH_MD5

2012 R2 is unchanged since the aforementioned April patch.

The point is that you should ensure your applications & clients don't have an issue with the cipher suite re-order. It's unlikely that your apps will have a problem but worthwhile to do a quick connectivity check in a test environment to be sure. If you do have issues you can re-order your suites after the patch by manipulating the registry keys listed here (not necessarily deleting the keys they list); use the before/after information above for reference. For more information about prioritizing cipher suites, see this.

It's unfortunate that they didn't communicate this change as it may have unanticipated impacts. Here's hoping they return to their generally good communication in the next cycle.

Update 11/18/2014:

As I'm sure you've heard, Microsoft has released three patches today, one of which is a re-issue of the patch mentioned in this post. The re-issue removes the four newly added cipher suites as there have been multiple problems reported with them. Microsoft also updated their article to include the following statement:

"Customers who customized their cipher suite priority list should review their list after they apply this update to make sure that the sequence meets their expectations.

Removing these cipher suites does not affect the security updates that are part of this release. On November 18, 2014, a new secondary package was added to the release for Windows Server 2008 R2 and Windows Server 2012 to achieve this. This new package is update 3018238, and it will install automatically and transparently together with security update 2992611. It will appear separately in the list of installed updates. If you already have security update 2992611 installed, you will notice that security update 2992611 will be reoffered (for Windows Server 2008 R2 or Windows Server 2012 only) by Windows Update or by Windows Server Update Services (WSUS) to make sure that update 3018238 is also installed.

The cipher suites may be re-added to the default priority list in a future release after the community has had an opportunity to make sure of correct execution in all customer scenarios."

I've just re-applied the newly released patches using WSUS to evaluate them. Note that you must re-appy using the same method you originally applied with, meaning that if you downloaded manually you would need to repeat that, and if you applied via WSUS you would need to use that methodology. Upon reviewing the "new" cipher suite order I was both surprised and happy with what I found:

Server 2008 R2 2992611 Patch 1 (11/14)	Server 2008 R2 2992611 Patch 2 (11/18)
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256	TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384	TLS_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P521	TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P521	TLS_RSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P521	TLS_RSA_WITH_RC4_128_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P521	TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384
TLS_RSA_WITH_AES_256_GCM_SHA384	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256
TLS_RSA_WITH_AES_128_GCM_SHA256	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P521	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P384	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P521	TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P521	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P384	TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P521	TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P521	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P521	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384
TLS_RSA_WITH_NULL_MD5	TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256	TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384	TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256	TLS_DHE_DSS_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384	TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256	TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384	TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA256	TLS_RSA_WITH_RC4_128_MD5
TLS_RSA_WITH_AES_128_CBC_SHA256	SSL_CK_RC4_128_WITH_MD5
TLS_RSA_WITH_AES_256_CBC_SHA	SSL_CK_DES_192_EDE3_CBC_WITH_MD5
TLS_RSA_WITH_AES_128_CBC_SHA	TLS_RSA_WITH_NULL_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384	TLS_RSA_WITH_NULL_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_DHE_DSS_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_RC4_128_MD5
TLS_RSA_WITH_NULL_SHA256
TLS_RSA_WITH_NULL_SHA
SSL_CK_RC4_128_WITH_MD5
SSL_CK_DES_192_EDE3_CBC_WITH_MD5

And now 2012:

Server 2012 2992611 Patch 1 (11/14)	Server 2012 2992611 Patch 2 (11/18)
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256	TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384	TLS_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384	TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256	TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_256_GCM_SHA384	TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_AES_128_GCM_SHA256	TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P384	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P384	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256	TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256
TLS_RSA_WITH_AES_256_CBC_SHA256	TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384
TLS_RSA_WITH_AES_128_CBC_SHA256	TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384
TLS_RSA_WITH_AES_256_CBC_SHA	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256
TLS_RSA_WITH_AES_128_CBC_SHA	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384	TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256	TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384	TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256	TLS_DHE_DSS_WITH_AES_128_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256	TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384	TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256	TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384	TLS_RSA_WITH_RC4_128_MD5
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256	SSL_CK_RC4_128_WITH_MD5
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256	SSL_CK_DES_192_EDE3_CBC_WITH_MD5
TLS_DHE_DSS_WITH_AES_256_CBC_SHA	TLS_RSA_WITH_NULL_SHA256
TLS_DHE_DSS_WITH_AES_128_CBC_SHA	TLS_RSA_WITH_NULL_SHA
TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_RC4_128_MD5
TLS_RSA_WITH_NULL_SHA256
TLS_RSA_WITH_NULL_SHA
SSL_CK_RC4_128_WITH_MD5
SSL_CK_DES_192_EDE3_CBC_WITH_MD5

Again, no changes for 2012 R2. If the above looks familiar, good eye. They're the same as pre-patch:

Server 2008 "patch 2" vs. no patch:

Server 2012 "patch 2" vs. no patch:

So as it pertains to the cipher suite order, we're right back where we started. I'm sure this will alleviate some of the issues some customers encountered. Keep in mind that there is more to this patch (binaries) than the cipher suite re-order, so to echo the previous assesment it should be installed.

Tuesday, November 11, 2014

Software Review: Altaro Hyper-V Backup

Backups can be a tough topic to get excited about. In a forward-looking technology field even needing to utilize your backup platform is generally a sign that something went wrong. As a result I haven't delved into the topic of backups for awhile. I'm taking a break from that dry spell to examine a Hyper-V backup solution, Altaro Hyper-V Backup.

Being that my local lab is Hyper-V based, I was able to take Altaro through its paces without impacting a production environment; something affording me the freedom to explore most of the features. Speaking of features (that's a $2 segue) let's examine the high level features offered by Altaro:

Support for backup of Hyper-V virtual machines
Fully Hypervisor/VSS-based, meaning no worries regarding open files or system state
Straightforward GUI-Driven backup schedule and retention policies
Offsite replication to another Altaro target (no additional license required) on a defined schedule, honoring retention policies
Seeding to offsite location using physical disk if desired
Agentless
Granular Restore:

File System Level
Exchange Items

"Sandbox" Restore for testing backup images

Automated sandbox scheduling

Cluster Shared Volume support
Boot directly from backup for testing, etc.
E-Mail based alerting

Taking the target market (more on that below) into consideration I will set my priorities as follows (in order of importance):

Reliability/General Quality
Labor investment to set up and maintain
Alerting Effectiveness (Not too noisy, alerting when necessary)
Cost
Basic Features (Granular Restore, etc.)
Advanced Features (VM Sandbox Restore, etc.)

Having spent over half my career in consulting I've been fortunate to see all sorts of environments. In my experience the main constraint for companies targeted by Altaro is usually time (labor) and potentially cost. That coupled with a predominant and understandable lack of enthusiasm for backup solutions makes the ease of setup and management crucial to ensure successful backups.

Target Market

Altaro Hyper-V backup supports Hyper-V servers and is likely most applicable at small to medium size businesses. Traditional backups of physical servers are not supported, which makes sense given the market gap Altaro aims to fill. The pricing is very competitive (more below) and comes in three versions: Free, Standard, and Unlimited.

This isn't to say the product may not be applicable in a larger enterprise, but the tools aren't geared to manage a huge number of servers and most enterprises are not exclusively utilizing Hyper-V for their virtualization needs. This review is written with this in mind. As with all things in IT you're looking for the right tool for the job.

Test Systems

I've tested this product in my lab on two Hyper-V hosts, one 2012 R2 Storage Server (with Hyper-V) and one 2012 Core Hyper-V host. While storage server isn't on the Altaro list of supported platforms I experienced no problems with it. Collectively this environment has 16 threads, 64GB RAM, and 12TB of various storage. There are 14 VMs eligible for backup including different Windows versions and two flavors of Linux. For test purposes my offsite backup target was an IaaS VM hosted in Windows Azure (Note: due to cost I do not recommend using Azure IaaS as a backup target in production).

Disclosures/Scoring

When reviewing I feel that full disclosure is critical to maintaining journalistic integrity. Altaro contacted me and asked if I would review their software; in the market myself for lab backup software I agreed to assess it. They provided me with a license to perform the appropriate testing. I did not accept any payment for this article and as always my opinions are my own.

I use a two-axes scoring square similar in design and function to the Gartner "Magic Quadrant". I use this when the testing is subjective and the target market of the product is not universal. This allows me to attempt to assess quality in two ways:

Implementation Quality: The quality of the features discussed in this section, but not relevant to the quantity or specific functionality of the features.
Feature Set: The quantity and specific functionality of the features discussed in this section, but not relevant to the quality.

Higher on the scale indicates better fit and finish, stability, and general quality. Farther right on the scale indicates a more extensive and useful feature set. The best position would be the top right and the worst the bottom left. It is possible that a given piece of software may have great features but is very buggy, few features but very stable, etc. I believe this scoring methodology allows me to communicate the overall software quality more effectively.

In some cases I will eliminate an axis from scoring if the category warrants it. In this review, "cost" is the only single axis score.

Installation/Basic Configuration

Installation couldn't be much more straightforward; the installer even works on server core. There are no special service account requirements; since the agent runs on each Hyper-V host it utilizes the built-in "SYSTEM" principal. Normally I wouldn't care for the use of the admin-equivalent SYSTEM principal, but since this is backup software there isn't much wiggle room access wise.

After installation on each Hyper-V host you will want to create a centralized management console if you have more than one host. This can be done on a management workstation if you desire; just install the free version on your workstation. After installation, connect to the local agent and select "Central Management Console" from the left pane and click "yes" when asked if you would like to create a shortcut.

From henceforth you can use the central console if desired.

Assuming your firewall rules were created and enabled you should be able to click "Add Agent to Group" to add each host. If you have multiple sites make sure you create appropriate groups for each and then add agents as appropriate.

Now you can connect to and manage each host in your environment from a central location. Note that you will need to launch a console for each host, but in a small/med environment this should be acceptable as setup is a one-time occurrence per host. For each host you can now configure the following:

VMs to backup
Primary backup location
Backup schedules and retention
Offsite backup target (see install note below)
Notification
Encryption key

Offsite install note: Altaro also provides a WAN target only client that does not count towards your license. Installation is simple; just click through, create a user account for your servers, and open firewall ports as necessary. The software is available here.

When configuring the encryption key, make sure you do so right away as doing so later will invalidate your previous backups; there is no support for rotating and retaining encryption keys, but based on the design that is a good thing (see below as to why). After that you'll want to finish up by specifying your base schedules and retention as well as notifications and offsite details.

Now let's address the review categories:

Reliability/General Quality

As stated, this is the most critical point for backup software. If it were buggy and didn't work consistently it wouldn't be worth using. I used the software for just over a month, performing onsite and offsite backups regularly, restoring full VMs and more granular items, and changing settings to exercise the software. Backup targets ranged from local disk through UNC paths and on to attached, dedicated storage. During that month I did not encounter any issues that concerned me. There were a couple very minor issues such as backups failing if the administrative console had a lock on previous backup files associated with the VM in question; these were to be expected and were addressed by support right away.

Performing a basic restore is almost *too* easy (not that I'm docking points). You can select which VM to restore, if you want it restored to a clone or to the original location, backup sources other than the main, as well as which day you want it restored to. As an added bonus, Altaro can automatically disable the NIC when restoring to avoid IP address conflicts. After a non-overwrite restore Altaro even adds the clone to the console of the Hyper-V host to which you restored! When utilizing any of these options I found the operation quick and encountered no problems whatsoever.

As for software updates, the Altaro console will check for updates and notify when they are released. The GUI provides an upgrade link and takes you directly to the download page. After downloading and running the setup, the installer handled the upgrade process flawlessly, automatically closing open executables and retaining settings. While it would be nice if the console facilitated the download & install directly, the execution of the existing system was flawless and in-line with the best its competitors have to offer.

Offsite backups can be a tricky thing to coordinate and pay for; fortunately Altaro has you covered there as well. Both over the network (internet) and disk seeding is supported in the standard and unlimited versions. Disk seeding allows you to push backups to disk and then send that disk to your offsite location rather than seed the initial full backups, which can be very useful when sending out many, many GBs of data for the first go around. After that you can rely on WAN/internet based uploads for deltas. There is a special software component you use as the backup target that does not require an additional license. This component can be installed on a Windows machine; keep in mind you'll need a network port opened and forwarded to the machine for it to work. To enable offsite backups Altaro requires that you set an encryption key, which is a welcome restriction since we don't want to pump unencrypted machine images over the internet. Once enabled you can schedule automatic offsite backups anywhere between once a month and every day. Note that if selecting the weekly/daily backup option in your schedule for normal backups you're locked to once a week at a minimum for offsite backups. It would be nice to be able to schedule the offsite backups less often in that case, but that's not too big a deal. Retention can be set independently between on and off-site. The only other thing I'd like to see here that isn't in the software at the time of review is offsite upload bandwidth throttling... perhaps in a future version.

Altaro also supports "reverse delta" which ensures that when performing incremental backups the latest backup taken contains the full image rather than the traditional concept of a full backup followed by deltas until the next full backup. This is similar to a feature in Veeam to accomplish the same thing, and one of my favorite things about this product. This means that the net size for cross site replication is limited to the delta size and the concept of regular "full" backups is obsolete. Note you still maintain the ability to restore to an older state; Altaro just down-revs the image as necessary when you request a restore.

Unfortunately there is no backup level de-duplication across multiple VM backups, nor is there support for OS level de-duplication at this time. The lack of de-duplication for regular offsite backups (bandwidth) is mitigated by the aforementioned reverse delta methodology however.

Hyper-V share nothing live migration does pose a couple interesting challenges; since the schedules and previous backups for a particular VM are tied to a host they don't follow a migration to a different host. They do remain on the originating host and work fine after migrating the VM back to the original host. The implications of this are:

The originating host will throw errors until the backup schedule for the missing VM is disabled.
Restores on the new host will not be available.
If you backup the VM on the new host, the initial backup will be full and offsite replication will be impacted as such.

While these issues are theoretically fixable it would be a mess to do so and I suspect it wouldn't impact customers much. The most common use case for share nothing live migration is to maintain uptime across host reboots, transferring the VMs back thereafter. In that case this is not an issue; the impact in your environment will be up to you to judge.

A more robust Hyper-V environment would likely utilize Hyper-V Cluster Shared Volumes, which Altaro Ultimate edition has native support for. While I did not test this feature extensively, the support and features is in line with what I would expect. This support is likely more important than the share nothing live migration issues I noted above.

The built in reporting is easy to consume and effective. It is not geared for notification of eternal parties, but barring that use case I don't see much of anything deficiency wise. The dashboard has impressively effective visualizations of backup times and volumes on a per VM or overall basis. The porting section has records of backups, restores, and errors.

Reliability and General Quality score:

Labor Investment to Set Up and Maintain

This is a category where Altaro really shines. Installation couldn't be much more straightforward, and usage/maintenance is impressively easy. This fact has many positive ramifications on a labor constrained environment and is why I have it ranked so highly. Altaro has an impressive collection of online documentation including user guides, a knowledgebase/FAQ, and they maintain a detailed change log.

As a demonstration of the ease of use, take for example the setup and association of a backup schedule:

Under the schedules tab, click to create a new schedule.
Define schedule frequency, time, and offsite schedule.
Drag the desired VMs to the schedule.

Most of the software is as intuitive as what you see here. I know I can reasonably delegate responsibility for this tool without having to provided dedicated training or even significant time in most target environments.

Ease of Set Up/Maintenance score:

Alerting Effectiveness

It is critical that backup software alert effectively; a silent failure in this space is a much larger problem waiting to happen. Altaro natively supports individually configuring the following categories for backup notifications:

Successful Backups/Offsite Copies
Backups/Offsite Copies with Skipped Files
Failed Backups/Offsite Copies
Completed Restore Operations

The following notification methodologies are supported:

E-Mail
Windows Event Log

Additionally, anyone connected to the console will receive alerts as well. While e-mail notification will generally be sufficient in most cases, large installations may want to log to the event log and use another tool to consolidate job status and deliver one consolidated report. I found all available notification methodologies to be reliable; the only minor nitpick I have is that the retry interval and thus the failure notification timing cannot be configured.

Alerting Effectiveness score:

Cost

Altaro pricing is very competitive. In my research I've found it cheaper than most of its main competitors, and the beauty of the licensing is that it is very straightforward. As of this writing the standard version is $395 per Hyper-V host (5 guests per maximum, no core/socket limit) and $585 for unlimited (unlimited VMs per guest, no core/socket limit). That price includes one year of upgrades and support, and you can choose to purchase additional years of upgrades and support for 25% of the purchase price ($99/$146). The free version that can backup two VMs per host is, oddly enough, free. The free version does not support some of the more advanced features which is to be expected. For the sake of comparison Veeam Hyper-V essentials 2CPU Socket licenses with a year of maintenance and support cost the following: "Standard" $820, "Enterprise" $1500, and "Enterprise Plus" $2300.

Cost score:

Basic Features

Altaro supports granular file level restore of any drive in a backup image; after selecting the VM you want a file from it loads up a file browser where you can select what files or directories you would like and where to restore them to (Standard and Unlimited). It also supports Exchange message level restore (Unlimited version). This option scans a VM backup image for Exchange databases in the default install locations. If your DB is in a non-standard location you can browse to and then mount it. You are then presented with a list of mailboxes; upon selecting one you can restore messages by their from, time, and/or subject criteria.

Score:

Advanced Features

In addition to the file level and Exchange object level restores, Altaro Standard and Unlimited support restoring to a sandbox.The Sanbox restore is a great start; it allows you to restore & boot selected VMs on a schedule to ensure the backups being taken are successful from a high level. This sort of thing can be a great feature to help business lines gain confidence in your ability to restore mission-critical virtual machines. It would be nice if this feature supported custom notifications (for non-IT interested parties) and eventually restoring multiple VMs to a shared sandbox and then running automated test scripts, but that's likely a big ask at this point in time. Point is they've already got great functionality and are moving in the right direction.

Altaro also supports booting directly from a backup drive (image), but this feature is hamstrung by the fact that it requires the backup image to be neither compressed or encrypted. I can't imagine a scenario where you wouldn't at least want to compress backups, so while this feature looks good on paper it isn't practical in most situations. That said, restoring a VM to a clone is so easy and quick that wouldn't fret too much about it. Frankly if I were Altaro I would consider removing this feature; not because it's not interesting but because assuming compression is the norm it isn't practical and could serve to disappoint a customer who thought too much of this feature on paper.

Score:

Bonus Super Feature: Encryption

The concept of "Trust No One" relies on software allowing one to specify their own key (or seed) to protect data. If implemented correctly then only the individual(s) with the key would be able to decrypt the data. Adherence to this concept can be critical for some businesses trusted with protecting not only their data but the data of their customers. I'm pleased to say that Altaro, at least in theory, supports "Trust No One" style AES encryption for both local (ultimate only) and offsite (standard & ultimate).

First a note: while the theory of trust no one results in unbreakable encryption, the actual security relies on the details of the implementation. While the key is specified by you, you still are actually trusting that the software doesn't intentionally or unintentionally store the encryption key in a way that could reveal it to anyone else. That can be difficult and I have no way to examine the implementation specifics at a code level.

That said, I'm very impressed that Altaro takes this seriously. One should be able to rest much easier knowing that if their backups are compromised their data is safe. To inquire about the implementation of this encryption, I asked the following question of Altaro support:

"If one enables onsite and remote encryption and provides their own key, but then somehow loses that key, is there any way to recover the backups? "

I received this response within hours:

"Hi Toby,
With regards to encrypted backups I’m afraid that the key cannot be recovered in any way and must thereby be remembered. This is designed this way in order to ensure that the sensitive information contained within the backups doesn’t fall into the wrong hands."

Perfect answer! This illustrates to me that not only is the software designed to be secure, but the support department is aware of the fact that it is and also aware of why. This makes me much more comfortable with offsite backups in particular. This to me is one of the most important aspects of Altaro; the fact that they get this right gives me a huge confidence in their overall design.

Note: Remember that as stated this encryption is optional and not enabled by default. If you don't want to encrypt you don't need to worry about storing a key.

Conclusion

When I set out to do this review I was afraid I would be one of two things: disappointed or bored. The quality of the software prevented me from being disappointed and the ease of use prevented me from being bored. This tool is a powerful backup solution for a small to medium size business implementation of a Hyper-V based private cloud, and the pricing is right in line, if not below that level. Additionally, Altaro's support department was impressive and their documentation is thorough. There are a few missing features, but taking the full picture into account I consider those minor. I have no reservations recommending Altaro for the Hyper-V based small to medium sized business.

itToby