Monday, December 22, 2014

Mumble + Splunk

Who's on the Server? Splunk it!


Mumble is a great VOIP solution for latency sensitive situations; I run several servers for different applications, and monitoring those has always been a bit of a challenge since it only generates text logs and doesn't do any historic usage tracking. Fortunately we've got a tool to solve that problem in both real time and historic situations: Splunk.

In this article we'll walk through a simple example of data ingestion, parsing, and dashboard creation in Splunk. When we're finished we'll be able to tell who is online at any given time and how popular your server has been in the recent past.


Step 1: Data Ingestion

This section assumes the following:

  • Mumble (or another app if using this as a general guideline) installed and logging to a consistent location.
  • Logging is not set to rename logfiles as part of the log rollover process; doing so would complicate our source setup. (note this is achievable) 
  • The Splunk forwarder is installed on the target machine(s) and is already configured to output to the indexer(s). (install location referred to henceforth as $SPLUNK_HOME) What's that? You've got thousands of boxes and no time to install? We can fix that!

I'm using Windows boxes in this example, but there's no reason all of this won't work on Linux as well with some minor tweaks.


To get the data into Splunk we'll first need to identify where the data is and how to ensure it gets into the target indexer(s). In my case I'll be targeting the following (4 instances) data sources from one server:
  • C:\Apps\Murmur1LowQual\murmur.log
  • C:\Apps\Murmur2LowQual\murmur.log
  • C:\Apps\Murmur1HighQual\murmur.log
  • C:\Apps\Murmur2HighQual\murmur.log
  • Performance data: Network interface, CPU Usage

If the application to be monitored is distributed densely and consistently in your enterprise you will want to make the input changes in an "app" for deployment assuming you use "Forwarder Management". In this case it is a one-off configuration so I will be specifying these inputs manually.


For our one-off case, open the $SPLUNK_HOME/etc/system/local/inputs.conf file for editing (for background, make sure you understand "About Configuration Files") and add the following entries:

    [monitor://C:\Apps\MurmurLowQual\murmur.log]
    disabled = false
    sourcetype = murmur
    
    [monitor://C:\Apps\MurmurHighQual\murmur.log]
    disabled = false
    sourcetype = murmur

    [monitor://C:\Apps\Murmur2HighQual\murmur.log]
    disabled = false
    sourcetype = murmur
    
    [monitor://C:\Apps\Murmur2LowQual\murmur.log]
    disabled = false
    sourcetype = murmur
    
    [perfmon://CPU Load]
    counters = % Processor Time
    instances = _Total
    interval = 20
    object = Processor
    
    [perfmon://Network Interface]
    counters = Bytes Received/sec;Bytes Sent/sec
    instances = *
    interval = 15
    object = Network Interface
    
Note these entries are Windows specific; the paths and the perfmon data would need be changed on a Linux host.

Where:
  • monitor://<logfilePath> specifies the path to the logfile to be ingested (Murmur is Mumble's server component)
  • sourcetype = murmur specifies the sourcetype to store this information under in Splunk. This is critical to properly sorting data.
  • perfmon://<counter> , counters = <counter1>;<counter2> , instances = *, and object = <object in question> specify the performance information we want to bring in. This line is Windows specific and needs to be changed for *nix.
  • interval = <interval in seconds> is the interval to bring that performance data in at. Lower interval = more granular performance information but more data to store.  

This may or may not be sufficient in your case. For more information, see "Edit inputs.conf" on the Splunk site


Step 2: Pre-Format Data


Now we need to take the steps necessary to easily create searches against this data. At a minimum any Splunk Admin should take the time to do proper field extractions from your new data source. Creating custom fields ensure there will be meaningful information for users to search on. This part of Splunk operation is often the make-or-break point in many organizations, as proper field extraction can be the difference between an end user figuring out how to create meaningful searches vs. giving up and going to the original log files.

In this example I'll use one regex to extract three fields from the Murmur logfile under the "search" app context. To set up this field extraction:

  1. Log in to your Splunk web interface
  2. Change to the "search" app context
  3. Navigate to "Settings -> Fields


  4. Click on "Field extractions"


  5. Click "New"
  6. Leave the "destination app"as "search" (We'll work in search for now but you could make this into an app)
  7. Name it per your enterprise standard. In my case that is <AppName_fields extracted>, so "Murmur_sessionID_UserName_AdminStatus".
  8. Change "Apply to" to "sourcetype" named "murmur"
  9. Keep "Type" to "Inline"
  10. For the Extraction/Transform insert your regex statement. To extract the Session ID (as session_ID), UserName (as u_name), and AdminStatus (as userIsAdmin) from a Murmur logfile use this: "=> <(?<session_ID>[^:]+):(?<u_name>[^\(]+)\((?<userIsAdmin>[^\)]+)"
  11. Click "Save"


  12. To make this usable for everyone, click "Permissions" under "Sharing" to the right of the name on the "Field extractions" screen. 


  13. Select "This app only(search)" (change later if you use a different app) and check "Read" under "Everyone" Click "Save". 

We have now configured field extractions for Mumble. While there is much more one should do (data sizing, business use cases, etc.) to on-board an application this will be enough for now to develop a basic dashboard.


Step 3: Searches and a Dashboard!


Now we'll make use of this data. This is the beauty of Splunk; you can format almost all the data in a meaningful way, and even create new data points inferred from the other available data. To illustrate what I mean, one of these searches will determine who is online right now from the logon/logoff information in the log file. Let's tackle that and a few others:


Search 1: Who is online right now?

sourcetype=murmur |transaction session_ID,u_name maxspan=24h|search authenticated NOT closed|eval AdminRange = case(userIsAdmin < 0, "False", userIsAdmin >= 1, "True")|table u_name,session_ID,AdminRange,_time | rename u_name as "User Name", session_ID as "Session ID", AdminRange as "User is Administrator"

Will generate a table like this:


Where:

sourcetype=murmur : only search the appropriate sourcetype. Note you may want to further limit by host, index, or other locations.

transaction session_ID,u_name maxspan=24h : uses the powerful "transaction" command to string events into a transaction by the listed fields. Note: This command can be computationally expensive so be careful when using it!

search authenticated NOT closed : look for sessions illustrating a user connecting but not yet closed Note: again, since we're using a "NOT" clause this search could be expensive depending on your data volume from the previous parts of the search. Fortunately we should be very limited data wise in this search string by now.

eval AdminRange = case(userIsAdmin < 0, "False", userIsAdmin >= 1, "True") : use the eval command to determine admin status as Boolean

table u_name,session_ID,AdminRange,_time : Render as a table

rename u_name as "User Name", session_ID as "Session ID", AdminRange as "User is Administrator" : Rename table fields to be more meaningful to the end user

Since we're limiting a session span to 24hours in the transaction search, you may as well reduce the time for this search to 24 hours as well. For a slight performance tweak on searches that display a table or graphic of specific fields you can change the search mode from "smart" to "fast".  Now let's add this to a new dashboard:
  1. After the results come up, click "Save As" the "Dashboard Panel"  on the upper right hand side.
  2. Select "New"
  3. Insert an appropriate title, i.e. "Mumble Statistics" following enterprise standards if applicable. Generally it is OK to let the Dashboard ID auto-populate with this name.
  4. Write a description if desired and change the Dashboard Permissions to "Shared in App" so we can share this information with others. 
  5. Type an appropriate Panel Tile such as "Users Online Now!". Accept the defaults for the remaining and click "Save".
For fun you can make a little tachometer displaying this information by opening the saved search and saving it to the existing dashboard under a different name, then changing the display to "Radial Gauge". End users generally like these "at a glance" graphics for important information, especially at the top of a dashboard.

Search 2: How many people have logged on per day for the last 30 days?

sourcetype=murmur u_name=* authenticated |timechart count

Will generate:


Where:

sourcetype=murmur u_name=* authenticated
: only search the appropriate sourcetype for events where u_name is populated and the word "authenticated" is present.

timechart count : charts all results using the very easy timechart command. Make sure you limit the search scope (below).

Set the search scope for 30 days, execute, then save to our existing Mumble Statistics dashboard as a panel named "Logins Per Day - Last 30 Days". I prefer this as a bar chart, which you can select at the dashboard level.

Search 3: How much bandwidth did the server use in the last day?

host=hostname_here sourcetype="Perfmon:Network Interface"|eval DataSrc=case((instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Received/sec"),"ETH0bitsSecIN",(instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Sent/sec"),"ETH0bitsSecOUT")|eval Kbits_Sec=Value*.008| timechart span=5m avg(Kbits_Sec) by DataSrc| rename LANbitsSecIN as "LAN Kbits/sec IN" LANbitsSecOUT as "LAN Kbits/sec OUT" WANbitsSecIN as "WAN Kbits/sec IN" WANbitsSecOUT as "WAN Kbits/sec OUT"

Will generate:


Where:

host=hostname_here sourcetype="Perfmon:Network Interface"
:  Specify events to search. You will need to change the hostname and potentially the sourcetype depending on your host platform, etc.

eval DataSrc=case((instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Received/sec"),"ETH0bytesSecIN",(instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Sent/sec"),"ETH0bytesSecOUT") : Here is where we use eval to map interfaces to directions. If you have multiple interfaces you'll need to address them on this line and also note you will need to change the "names" of the adapters to match your data.

eval Kbits_Sec=Value*.008 : Convert Bytes/Sec to Kbits/Sec

timechart span=5m avg(Kbits_Sec) by DataSrc : Chart the data by interface direction on a 5 minute average. Note if you made a longer-term chart you'll need to change the average to calculate on a wider basis to keep your datapoints low enough to chart.

rename ETH0bytesSecIN as "ETH0 Kbits/sec IN" ETH0bytesSecOUT as "ETH0 Kbits/sec OUT" : Rename the datapoints relative to our eval statement above.

Set the search scope for 24 hours, execute, then save to our existing Mumble Statistics dashboard as a panel named "Network Traffic 5m Average Last 24 Hours"

How many users on the sever? Plenty.

Those three should get you started; clearly there is substantially more one could do with all the data available to us. After you decide what else to add make sure you go through your dashboard and reposition/edit each panel as necessary. Keep in mind that you can rename x/y axis as well as change the way data is rendered. Hopefully this tutorial has demonstrated that even with a simple application you can use a tool like Splunk to make the day-to-day use and impact on an organization much more transparent. This methodology could easily be turned into an app and distributed throughout your enterprise and/or the Splunk community.

Saturday, November 15, 2014

Microsoft KB 2992611 "Winshock": More Thank You Bargained For

(updated 11/18 for re-issue, see below)

Microsoft released one of the most important patches in many years on Tuesday, and while I would advise you install it right away I also want to make you aware of some odd behavior I found that could lead to problems. First, a primer:

Here are the release notes and here is more information. This is the worst kind of exploit there can be; a remote code execution with no workarounds. If one knew the details, they could easily exploit any Microsoft based internet facing server supporting TLS and then turn around and use it to infect unpatched Windows based clients. Obviously you should patch immediately.

That said, you will notice that they mention the addition of four new cipher suites but there is one other change that may impact you that is not mentioned. I've found that this patch also re-orders the cipher suites. Historically Microsoft has notified customers when re-ordering cipher suites; see KB2919355 for example.

This is important to understand for two reasons, one theoretical and one practical.

  • Theoretical is that changing cipher suites impacts your security posture, and one should always know these things going into a patch. Fortunately most of the re-order does seem in line with a tighter security policy.
  • Practical is that this can break connectivity with some applications. Specifically, one of my peers found that Java 6 based applications attempting purposely or otherwise to use the ECDH key agreement protocol will fail to connect. This happens when Windows based services present ECDH before the older RSA. Side note: Oddly the Microsoft JDBC driver tries to negotiate SSL even if it isn't being used for a connection to SQL.

Here are the cipher suite details, first 2008 R2:

2008 R2 Default Before KB299261 2008 R2 Default After KB299261
TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256
TLS_RSA_WITH_AES_128_CBC_SHA TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384
TLS_RSA_WITH_AES_256_CBC_SHA256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P521
TLS_RSA_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P521
TLS_RSA_WITH_RC4_128_SHA TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P521
TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P521
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384 TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256 TLS_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384 TLS_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P521
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P384
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P521
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P521
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P384
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P521
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P521
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P521
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256 TLS_RSA_WITH_NULL_MD5
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384
TLS_DHE_DSS_WITH_AES_128_CBC_SHA TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384
TLS_DHE_DSS_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384
TLS_RSA_WITH_RC4_128_MD5 TLS_RSA_WITH_AES_256_CBC_SHA256
SSL_CK_RC4_128_WITH_MD5 TLS_RSA_WITH_AES_128_CBC_SHA256
SSL_CK_DES_192_EDE3_CBC_WITH_MD5 TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_NULL_SHA256 TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_NULL_SHA TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384

TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256

TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384

TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256

TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256

TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384

TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256

TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384

TLS_DHE_DSS_WITH_AES_256_CBC_SHA256

TLS_DHE_DSS_WITH_AES_128_CBC_SHA256

TLS_DHE_DSS_WITH_AES_256_CBC_SHA

TLS_DHE_DSS_WITH_AES_128_CBC_SHA

TLS_RSA_WITH_3DES_EDE_CBC_SHA

TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA

TLS_RSA_WITH_RC4_128_SHA

TLS_RSA_WITH_RC4_128_MD5

TLS_RSA_WITH_NULL_SHA256

TLS_RSA_WITH_NULL_SHA

SSL_CK_RC4_128_WITH_MD5

SSL_CK_DES_192_EDE3_CBC_WITH_MD5

And 2012 (not R2):

2012 Default Before KB299261 2012 Default After KB299261
TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256
TLS_RSA_WITH_AES_128_CBC_SHA TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384
TLS_RSA_WITH_AES_256_CBC_SHA256 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
TLS_RSA_WITH_AES_256_CBC_SHA TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
TLS_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_AES_256_GCM_SHA384
TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384 TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384 TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256 TLS_RSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384 TLS_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384
TLS_DHE_DSS_WITH_AES_128_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256
TLS_DHE_DSS_WITH_AES_256_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256
TLS_RSA_WITH_RC4_128_MD5 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384
SSL_CK_RC4_128_WITH_MD5 TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
SSL_CK_DES_192_EDE3_CBC_WITH_MD5 TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
TLS_RSA_WITH_NULL_SHA256 TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_NULL_SHA TLS_DHE_DSS_WITH_AES_128_CBC_SHA

TLS_RSA_WITH_3DES_EDE_CBC_SHA

TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA

TLS_RSA_WITH_RC4_128_SHA

TLS_RSA_WITH_RC4_128_MD5

TLS_RSA_WITH_NULL_SHA256

TLS_RSA_WITH_NULL_SHA

SSL_CK_RC4_128_WITH_MD5

SSL_CK_DES_192_EDE3_CBC_WITH_MD5

2012 R2 is unchanged since the aforementioned April patch.

The point is that you should ensure your applications & clients don't have an issue with the cipher suite re-order. It's unlikely that your apps will have a problem but worthwhile to do a quick connectivity check in a test environment to be sure. If you do have issues you can re-order your suites after the patch by manipulating the registry keys listed here (not necessarily deleting the keys they list);  use the before/after information above for reference. For more information about prioritizing cipher suites, see this.

It's unfortunate that they didn't communicate this change as it may have unanticipated impacts. Here's hoping they return to their generally good communication in the next cycle.

Update 11/18/2014:


As I'm sure you've heard, Microsoft has released three patches today, one of which is a re-issue of the patch mentioned in this post. The re-issue removes the four newly added cipher suites as there have been multiple problems reported with them. Microsoft also updated their article to include the following statement:

"Customers who customized their cipher suite priority list should review their list after they apply this update to make sure that the sequence meets their expectations.

Removing these cipher suites does not affect the security updates that are part of this release. On November 18, 2014, a new secondary package was added to the release for Windows Server 2008 R2 and Windows Server 2012 to achieve this. This new package is update 3018238, and it will install automatically and transparently together with security update 2992611. It will appear separately in the list of installed updates. If you already have security update 2992611 installed, you will notice that security update 2992611 will be reoffered (for Windows Server 2008 R2 or Windows Server 2012 only) by Windows Update or by Windows Server Update Services (WSUS) to make sure that update 3018238 is also installed.

The cipher suites may be re-added to the default priority list in a future release after the community has had an opportunity to make sure of correct execution in all customer scenarios.
"

I've just re-applied the newly released patches using WSUS to evaluate them. Note that you must re-appy using the same method you originally applied with, meaning that if you downloaded manually you would need to repeat that, and if you applied via WSUS you would need to use that methodology. Upon reviewing the "new" cipher suite order I was both surprised and happy with what I found:

Server 2008 R2 2992611 Patch 1 (11/14) Server 2008 R2 2992611 Patch 2 (11/18)
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256 TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384 TLS_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P521 TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P521 TLS_RSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P521 TLS_RSA_WITH_RC4_128_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P521 TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384
TLS_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256
TLS_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P521 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P384 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P521 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P521 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P384 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P521 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P521 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P521 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384
TLS_RSA_WITH_NULL_MD5 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384 TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256 TLS_DHE_DSS_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384 TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256 TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384 TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA256 TLS_RSA_WITH_RC4_128_MD5
TLS_RSA_WITH_AES_128_CBC_SHA256 SSL_CK_RC4_128_WITH_MD5
TLS_RSA_WITH_AES_256_CBC_SHA SSL_CK_DES_192_EDE3_CBC_WITH_MD5
TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_NULL_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384 TLS_RSA_WITH_NULL_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_DHE_DSS_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_RC4_128_MD5
TLS_RSA_WITH_NULL_SHA256
TLS_RSA_WITH_NULL_SHA
SSL_CK_RC4_128_WITH_MD5
SSL_CK_DES_192_EDE3_CBC_WITH_MD5

And now 2012:

Server 2012 2992611 Patch 1 (11/14) Server 2012 2992611 Patch 2 (11/18)
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256 TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384 TLS_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_256_GCM_SHA384 TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_AES_128_GCM_SHA256 TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256
TLS_RSA_WITH_AES_256_CBC_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384
TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384
TLS_RSA_WITH_AES_256_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256
TLS_RSA_WITH_AES_128_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384 TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256 TLS_DHE_DSS_WITH_AES_128_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256 TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384 TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256 TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384 TLS_RSA_WITH_RC4_128_MD5
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 SSL_CK_RC4_128_WITH_MD5
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 SSL_CK_DES_192_EDE3_CBC_WITH_MD5
TLS_DHE_DSS_WITH_AES_256_CBC_SHA TLS_RSA_WITH_NULL_SHA256
TLS_DHE_DSS_WITH_AES_128_CBC_SHA TLS_RSA_WITH_NULL_SHA
TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_RC4_128_MD5
TLS_RSA_WITH_NULL_SHA256
TLS_RSA_WITH_NULL_SHA
SSL_CK_RC4_128_WITH_MD5
SSL_CK_DES_192_EDE3_CBC_WITH_MD5

Again, no changes for 2012 R2. If the above looks familiar, good eye. They're the same as pre-patch:

Server 2008 "patch 2" vs. no patch:

Server 2008 R2 2992611 Patch 2 (11/18) Server 2008 R2 Before Either 2992611 Patch
TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
TLS_DHE_DSS_WITH_AES_128_CBC_SHA TLS_DHE_DSS_WITH_AES_128_CBC_SHA
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
TLS_DHE_DSS_WITH_AES_256_CBC_SHA TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_RC4_128_MD5 TLS_RSA_WITH_RC4_128_MD5
SSL_CK_RC4_128_WITH_MD5 SSL_CK_RC4_128_WITH_MD5
SSL_CK_DES_192_EDE3_CBC_WITH_MD5 SSL_CK_DES_192_EDE3_CBC_WITH_MD5
TLS_RSA_WITH_NULL_SHA256 TLS_RSA_WITH_NULL_SHA256
TLS_RSA_WITH_NULL_SHA TLS_RSA_WITH_NULL_SHA

Server 2012 "patch 2" vs. no patch:

Server 2012 2992611 Patch 2 (11/18) Server 2012 Before Either 2992611 Patch
TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
TLS_DHE_DSS_WITH_AES_128_CBC_SHA TLS_DHE_DSS_WITH_AES_128_CBC_SHA
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
TLS_DHE_DSS_WITH_AES_256_CBC_SHA TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_RC4_128_MD5 TLS_RSA_WITH_RC4_128_MD5
SSL_CK_RC4_128_WITH_MD5 SSL_CK_RC4_128_WITH_MD5
SSL_CK_DES_192_EDE3_CBC_WITH_MD5 SSL_CK_DES_192_EDE3_CBC_WITH_MD5
TLS_RSA_WITH_NULL_SHA256 TLS_RSA_WITH_NULL_SHA256
TLS_RSA_WITH_NULL_SHA TLS_RSA_WITH_NULL_SHA

So as it pertains to the cipher suite order, we're right back where we started. I'm sure this will alleviate some of the issues some customers encountered. Keep in mind that there is more to this patch (binaries) than the cipher suite re-order, so to echo the previous assesment it should be installed.

Tuesday, November 11, 2014

Software Review: Altaro Hyper-V Backup

Backups can be a tough topic to get excited about. In a forward-looking technology field even needing to utilize your backup platform is generally a sign that something went wrong. As a result I haven't delved into the topic of backups for awhile. I'm taking a break from that dry spell to examine a Hyper-V backup solution, Altaro Hyper-V Backup.


Being that my local lab is Hyper-V based, I was able to take Altaro through its paces without impacting a production environment; something affording me the freedom to explore most of the features. Speaking of features (that's a $2 segue) let's examine the high level features offered by Altaro:

  • Support for backup of Hyper-V virtual machines
  • Fully Hypervisor/VSS-based, meaning no worries regarding open files or system state
  • Straightforward GUI-Driven backup schedule and retention policies
  • Offsite replication to another Altaro target (no additional license required) on a defined schedule, honoring retention policies
  • Seeding to offsite location using physical disk if desired
  • Agentless
  • Granular Restore:
    • File System Level
    • Exchange Items
  • "Sandbox" Restore for testing backup images
    • Automated sandbox scheduling
  • Cluster Shared Volume support
  • Boot directly from backup for testing, etc.
  • E-Mail based alerting

Taking the target market (more on that below) into consideration I will set my priorities as follows (in order of importance):

  • Reliability/General Quality
  • Labor investment to set up and maintain
  • Alerting Effectiveness (Not too noisy, alerting when necessary) 
  • Cost
  • Basic Features (Granular Restore, etc.)
  • Advanced Features (VM Sandbox Restore, etc.)

Having spent over half my career in consulting I've been fortunate to see all sorts of environments. In my experience the main constraint for companies targeted by Altaro is usually time (labor) and potentially cost. That coupled with a predominant and understandable lack of enthusiasm for backup solutions makes the ease of setup and management crucial to ensure successful backups. 

Target Market

Altaro Hyper-V backup supports Hyper-V servers and is likely most applicable at small to medium size businesses. Traditional backups of physical servers are not supported, which makes sense given the market gap Altaro aims to fill. The pricing is very competitive (more below) and comes in three versions: Free, Standard, and Unlimited. 

This isn't to say the product may not be applicable in a larger enterprise, but the tools aren't geared to manage a huge number of servers and most enterprises are not exclusively utilizing Hyper-V for their virtualization needs. This review is written with this in mind. As with all things in IT you're looking for the right tool for the job.

Test Systems

I've tested this product in my lab on two Hyper-V hosts, one 2012 R2 Storage Server (with Hyper-V) and one 2012 Core Hyper-V host. While storage server isn't on the Altaro list of supported platforms I experienced no problems with it. Collectively this environment has 16 threads, 64GB RAM, and 12TB of various storage. There are 14 VMs eligible for backup including different Windows versions and two flavors of Linux. For test purposes my offsite backup target was an IaaS VM hosted in Windows Azure (Note: due to cost I do not recommend using Azure IaaS as a backup target in production).

Disclosures/Scoring

When reviewing I feel that full disclosure is critical to maintaining journalistic integrity. Altaro contacted me and asked if I would review their software; in the market myself for lab backup software I agreed to assess it. They provided me with a license to perform the appropriate testing. I did not accept any payment for this article and as always my opinions are my own.

I use a two-axes scoring square similar in design and function to the Gartner "Magic Quadrant". I use this when the testing is subjective and the target market of the product is not universal. This allows me to attempt to assess quality in two ways:

  • Implementation Quality: The quality of the features discussed in this section, but not relevant to the quantity or specific functionality of the features.
  • Feature Set: The quantity and specific functionality of the features discussed in this section, but not relevant to the quality.

Higher on the scale indicates better fit and finish, stability, and general quality. Farther right on the scale indicates a more extensive and useful feature set. The best position would be the top right and the worst the bottom left. It is possible that a given piece of software may have great features but is very buggy, few features but very stable, etc. I believe this scoring methodology allows me to communicate the overall software quality more effectively.

In some cases I will eliminate an axis from scoring if the category warrants it. In this review, "cost" is the only single axis score.

Installation/Basic Configuration

Installation couldn't be much more straightforward; the installer even works on server core. There are no special service account requirements; since the agent runs on each Hyper-V host it utilizes the built-in "SYSTEM" principal. Normally I wouldn't care for the use of the admin-equivalent SYSTEM principal, but since this is backup software there isn't much wiggle room access wise.

After installation on each Hyper-V host you will want to create a centralized management console if you have more than one host. This can be done on a management workstation if you desire; just install the free version on your workstation. After installation, connect to the local agent and select "Central Management Console" from the left pane and click "yes" when asked if you would like to create a shortcut.


From henceforth you can use the central console if desired.

Assuming your firewall rules were created and enabled you should be able to click "Add Agent to Group" to add each host. If you have multiple sites make sure you create appropriate groups for each and then add agents as appropriate. 

Now you can connect to and manage each host in your environment from a central location. Note that you will need to launch a console for each host, but in a small/med environment this should be acceptable as setup is a one-time occurrence per host. For each host you can now configure the following:

  • VMs to backup
  • Primary backup location
  • Backup schedules and retention
  • Offsite backup target (see install note below)
  • Notification
  • Encryption key

Offsite install note: Altaro also provides a WAN target only client that does not count towards your license. Installation is simple; just click through, create a user account for your servers, and open firewall ports as necessary. The software is available here.

When configuring the encryption key, make sure you do so right away as doing so later will invalidate your previous backups; there is no support for rotating and retaining encryption keys, but based on the design that is a good thing (see below as to why). After that you'll want to finish up by specifying your base schedules and retention as well as notifications and offsite details.

Now let's address the review categories:

Reliability/General Quality

As stated, this is the most critical point for backup software. If it were buggy and didn't work consistently it wouldn't be worth using. I used the software for just over a month, performing onsite and offsite backups regularly, restoring full VMs and more granular items, and changing settings to exercise the software. Backup targets ranged from local disk through UNC paths and on to attached, dedicated storage. During that month I did not encounter any issues that concerned me. There were a couple very minor issues such as backups failing if the administrative console had a lock on previous backup files associated with the VM in question; these were to be expected and were addressed by support right away.

Performing a basic restore is almost *too* easy (not that I'm docking points). You can select which VM to restore, if you want it restored to a clone or to the original location, backup sources other than the main, as well as which day you want it restored to. As an added bonus, Altaro can automatically disable the NIC when restoring to avoid IP address conflicts. After a non-overwrite restore Altaro even adds the clone to the console of the Hyper-V host to which you restored! When utilizing any of these options I found the operation quick and encountered no problems whatsoever.

As for software updates, the Altaro console will check for updates and notify when they are released. The GUI provides an upgrade link and takes you directly to the download page. After downloading and running the setup, the installer handled the upgrade process flawlessly, automatically closing open executables and retaining settings. While it would be nice if the console facilitated the download & install directly, the execution of the existing system was flawless and in-line with the best its competitors have to offer.


Offsite backups can be a tricky thing to coordinate and pay for; fortunately Altaro has you covered there as well. Both over the network (internet) and disk seeding is supported in the standard and unlimited versions. Disk seeding allows you to push backups to disk and then send that disk to your offsite location rather than seed the initial full backups, which can be very useful when sending out many, many GBs of data for the first go around. After that you can rely on WAN/internet based uploads for deltas. There is a special software component you use as the backup target that does not require an additional license. This component can be installed on a Windows machine; keep in mind you'll need a network port opened and forwarded to the machine for it to work. To enable offsite backups Altaro requires that you set an encryption key, which is a welcome restriction since we don't want to pump unencrypted machine images over the internet. Once enabled you can schedule automatic offsite backups anywhere between once a month and every day. Note that if selecting the weekly/daily backup option in your schedule for normal backups you're locked to once a week at a minimum for offsite backups. It would be nice to be able to schedule the offsite backups less often in that case, but that's not too big a deal. Retention can be set independently between on and off-site. The only other thing I'd like to see here that isn't in the software at the time of review is offsite upload bandwidth throttling... perhaps in a future version.


Altaro also supports "reverse delta" which ensures that when performing incremental backups the latest backup taken contains the full image rather than the traditional concept of a full backup followed by deltas until the next full backup. This is similar to a feature in Veeam to accomplish the same thing, and one of my favorite things about this product. This means that the net size for cross site replication is limited to the delta size and the concept of regular "full" backups is obsolete. Note you still maintain the ability to restore to an older state; Altaro just down-revs the image as necessary when you request a restore.

Unfortunately there is no backup level de-duplication across multiple VM backups, nor is there support for OS level de-duplication at this time. The lack of de-duplication for regular offsite backups (bandwidth) is mitigated by the aforementioned reverse delta methodology however.

Hyper-V share nothing live migration does pose a couple interesting challenges; since the schedules and previous backups for a particular VM are tied to a host they don't follow a migration to a different host. They do remain on the originating host and work fine after migrating the VM back to the original host. The implications of this are:

  • The originating host will throw errors until the backup schedule for the missing VM is disabled.
  • Restores on the new host will not be available.
  • If you backup the VM on the new host, the initial backup will be full and offsite replication will be impacted as such.

While these issues are theoretically fixable it would be a mess to do so and I suspect it wouldn't impact customers much. The most common use case for share nothing live migration is to maintain uptime across host reboots, transferring the VMs back thereafter. In that case this is not an issue; the impact in your environment will be up to you to judge. 

A more robust Hyper-V environment would likely utilize Hyper-V Cluster Shared Volumes, which Altaro Ultimate edition has native support for.  While I did not test this feature extensively, the support and features is in line with what I would expect. This support is likely more important than the share nothing live migration issues I noted above.

The built in reporting is easy to consume and effective. It is not geared for notification of eternal parties, but barring that use case I don't see much of anything deficiency wise. The dashboard has impressively effective visualizations of backup times and volumes on a per VM or overall basis. The porting section has records of backups, restores, and errors. 

Reliability and General Quality score:


Labor Investment to Set Up and Maintain

This is a category where Altaro really shines. Installation couldn't be much more straightforward, and usage/maintenance is impressively easy. This fact has many positive ramifications on a labor constrained environment and is why I have it ranked so highly. Altaro has an impressive collection of online documentation including user guides, a knowledgebase/FAQ, and they maintain a detailed change log

As a demonstration of the ease of use, take for example the setup and association of a backup schedule:

  1. Under the schedules tab, click to create a new schedule.
  2. Define schedule frequency, time, and offsite schedule.
  3. Drag the desired VMs to the schedule.

Most of the software is as intuitive as what you see here. I know I can reasonably delegate responsibility for this tool without having to provided dedicated training or even significant time in most target environments.

Ease of Set Up/Maintenance score:


Alerting Effectiveness



It is critical that backup software alert effectively; a silent failure in this space is a much larger problem waiting to happen. Altaro natively supports individually configuring the following categories for backup notifications:

  • Successful Backups/Offsite Copies
  • Backups/Offsite Copies with Skipped Files
  • Failed Backups/Offsite Copies
  • Completed Restore Operations

The following notification methodologies are supported:

  • E-Mail
  • Windows Event Log

Additionally, anyone connected to the console will receive alerts as well. While e-mail notification will generally be sufficient in most cases, large installations may want to log to the event log and use another tool to consolidate job status and deliver one consolidated report. I found all available notification methodologies to be reliable; the only minor nitpick I have is that the retry interval and thus the failure notification timing cannot be configured.

Alerting Effectiveness score:


Cost

Altaro pricing is very competitive. In my research I've found it cheaper than most of its main competitors, and the beauty of the licensing is that it is very straightforward. As of this writing the standard version is $395 per Hyper-V host (5 guests per maximum, no core/socket limit) and $585 for unlimited (unlimited VMs per guest, no core/socket limit). That price includes one year of upgrades and support, and you can choose to purchase additional years of upgrades and support for 25% of the purchase price ($99/$146). The free version that can backup two VMs per host is, oddly enough, free. The free version does not support some of the more advanced features which is to be expected. For the sake of comparison Veeam Hyper-V essentials 2CPU Socket licenses with a year of maintenance and support cost the following: "Standard" $820, "Enterprise" $1500, and "Enterprise Plus" $2300.

Cost score:


Basic Features

Altaro supports granular file level restore of any drive in a backup image; after selecting the VM you want a file from it loads up a file browser where you can select what files or directories you would like and where to restore them to (Standard and Unlimited). It also supports Exchange message level restore (Unlimited version). This option scans a VM backup image for Exchange databases in the default install locations. If your DB is in a non-standard location you can browse to and then mount it. You are then presented with a list of mailboxes; upon selecting one you can restore messages by their from, time, and/or subject criteria.

Score:


Advanced Features

In addition to the file level and Exchange object level restores, Altaro Standard and Unlimited support restoring to a sandbox.The Sanbox restore is a great start; it allows you to restore & boot selected VMs on a schedule to ensure the backups being taken are successful from a high level. This sort of thing can be a great feature to help business lines gain confidence in your ability to restore mission-critical virtual machines. It would be nice if this feature supported custom notifications (for non-IT interested parties) and eventually restoring multiple VMs to a shared sandbox and then running automated test scripts, but that's likely a big ask at this point in time. Point is they've already got great functionality and are moving in the right direction.

Altaro also supports booting directly from a backup drive (image), but this feature is hamstrung by the fact that it requires the backup image to be neither compressed or encrypted. I can't imagine a scenario where you wouldn't at least want to compress backups, so while this feature looks good on paper it isn't practical in most situations. That said, restoring a VM to a clone is so easy and quick that wouldn't fret too much about it. Frankly if I were Altaro I would consider removing this feature; not because it's not interesting but because assuming compression is the norm it isn't practical and could serve to disappoint a customer who thought too much of this feature on paper.

Score:


Bonus Super Feature: Encryption

The concept of "Trust No One" relies on software allowing one to specify their own key (or seed) to protect data. If implemented correctly then only the individual(s) with the key would be able to decrypt the data. Adherence to this concept can be critical for some businesses trusted with protecting not only their data but the data of their customers. I'm pleased to say that Altaro, at least in theory, supports "Trust No One" style AES encryption for both local (ultimate only) and offsite (standard & ultimate).

First a note: while the theory of trust no one results in unbreakable encryption, the actual security relies on the details of the implementation. While the key is specified by you, you still are actually trusting that the software doesn't intentionally or unintentionally store the encryption key in a way that could reveal it to anyone else. That can be difficult and I have no way to examine the implementation specifics at a code level.

That said, I'm very impressed that Altaro takes this seriously. One should be able to rest much easier knowing that if their backups are compromised their data is safe. To inquire about the implementation of this encryption, I asked the following question of Altaro support:

"If one enables onsite and remote encryption and provides their own key, but then somehow loses that key, is there any way to recover the backups? "

I received this response within hours:

"Hi Toby,
With regards to encrypted backups I’m afraid that the key cannot be recovered in any way and must thereby be remembered. This is designed this way in order to ensure that the sensitive information contained within the backups doesn’t fall into the wrong hands.
"

Perfect answer! This illustrates to me that not only is the software designed to be secure, but the support department is aware of the fact that it is and also aware of why. This makes me much more comfortable with offsite backups in particular. This to me is one of the most important aspects of Altaro; the fact that they get this right gives me a huge confidence in their overall design.

Note: Remember that as stated this encryption is optional and not enabled by default. If you don't want to encrypt you don't need to worry about storing a key. 

Conclusion

When I set out to do this review I was afraid I would be one of two things: disappointed or bored. The quality of the software prevented me from being disappointed and the ease of use prevented me from being bored. This tool is a powerful backup solution for a small to medium size business implementation of a Hyper-V based private cloud, and the pricing is right in line, if not below that level. Additionally, Altaro's support department was impressive and their documentation is thorough. There are a few missing features, but taking the full picture into account I consider those minor. I have no reservations recommending Altaro for the Hyper-V based small to medium sized business.