Tunnel to the Cloud: Azure Site to Site IPsec Connection

Do you have multi-continent datacenters with gobs of bandwidth, IOps, and processing capability? No? I can help get you part of the way there... a network presence in one.

A tunnel.

A Site to Site Connection?

It's easier to think of this as an extension to your network into another datacenter over the internet. Using IPsec we can provide a relatively (comments at the end) secure, direct connection between on on-premises datacenter and Azure hosted resources by encrypting the traffic that flows between the two.  What do I mean by:

• Secure: IPsec tunnels all your traffic so it is encrypted over the internet; in reality, this is really "more secure" rather than definitively secure, as the effective security depends highly on implementation specifics.
• Direct: Your router (played by pfSense in this case) will recognize the Azure site as another routable network within the boundaries of your own, enabling you to talk to Azure resources as if they were in your own datacenter.

While I'll be using pfSense for the initiator side as it exposes the options in the most clear way I've found, this article will also be useful for non-pfSense devices since we discuss the details of the IPsec tunnel; the information here should be applicable to any IPsec solution. Update 1/2017: I've personally tested on various Cisco, Sonicwall, and pFsense equipment, and Microsoft has added some great documentation about overall device support here.

Note: this works for Amazon Web Services (AWS) as well but is slightly more complex. Fortunately pfSense includes a wizard that works, but takes a lot of the fun out of it as it strips you of understanding how it works. In addition the wizard is necessary because of how Amazon does VPC routing, whereas Azure is a bit more straightforward.

With that, let's get to it!

Pre-Requisites

• pfSense firewall(s): The steps in this article were performed on a pair of  HA SG-4860 firewalls running pfSense 2.32p1.
• Microsoft Azure account with adequate permissions: We'll be performing our actions using the "new" portal based on Azure Resource Manager (ARM or AzureRM).
• AzureRM PowerShell Cmdlets installed: On Win10/Server 2016 this can be accomplished with Install-Module AzureRM; for more info see this post.

Configure Azure IPSec Endpoint

Before we set up and initiate the connection from pfSense, we need to set up our endpoint in Azure. To do so, we'll create the following objects:

• A Resource Group
• A Public IP Address
• A Virtual Network
• A Gateway Subnet
• A Virtual Network Gateway
• A Local Network Gateway with a Connection

Resource Group

A Resource Group is a logical grouping of Azure Resources. This logical group allows for easy organization and clearer billing reports. We won't get too much into concepts and naming standards here other than to say groups should be logically tied with similar lifecycle expectancies and you should be consistent. For more information, see this Azure article.

Note: We'll be doing most of our steps in the web portal, but this whole process is much more efficient with PowerShell.

Let's go!

1. Open and log into the ARM Azure Portal at portal.azure.com; ensure you're working with the subscription you intend to use.
2. Navigate to "Resource Groups".
3. Click "Add" and type a name for your resource group, select the subscription, and resource group location. Note: The resource group location has no bearing on where you'll be connecting to as it's just a location the metadata is stored.
4. Click "Save".

Public IP Address

While we could do the IP at the time we make the Virtual Network Gateway, we'll take care of it now to ensure it's provisioned prior to getting to that step and to discuss the IP details.

1. Navigate to "Public IP addresses".
2. Click "Add" and populate the following:
• Name: Select a name leveraging consistent naming standards.
• IP address assignment: Select "Dynamic". I know what you're saying.. you're saying "but Toby, I'm not saying anything", and what I'm saying is it seems this should be static. Unfortunately if we make a static IP we'll be greeted later with the following:
  
  

  

  Why Azure, WHY?!

  
  Now I've been running a tunnel straight for almost a month thus far and my dynamic IP has not shifted on me; I suspect it will behave the same as IPs for other resources and stay static so long as it is used. If this does change, you'll need to change the info in the Phase 1 and 2 setup of the tunnel on the pfSense side as outlined below. For the record, as of the writing of this article the pricing of IPs in Azure is a bit odd; dynamic IPs and static IPs beyond the first 5 in any region are charged the same (pretty trivial), while the first 5 static in a region are free. See here for more info.
• Idle Timeout: The default of 4 minutes should be fine here.
• DNS Name Label: Optionally, specify a DNS alias here, though we will not reference it again in this guide as I'm not addressing DNS issues associated with IPsec at this time.
• Subscription: Select your subscription.
• Resource Group: Click the resource group we created in the last step.
• Location: The IP is our IPsec target, so select a location close to your local network connection. The Azure Speed Test comes in quite handy here.





3. Click "Create". This provisioning will take a few minutes minutes as Azure re-arranges its SDN infrastructure to give you an IP.





What a successful deployment looks like!

Virtual Network/Subnet/Gateway Subnet

A "Virtual Network" is a network space within Azure that you can carve up and protect (firewall) to suit your needs. We're required to make one subnet, and we'll create our "gateway subnet" (landing point) as well. If this is your first foray into virtual networks on Azure you may want to take a step back and consider your design before proceeding. Oh, you're back already? Let's go.

1. Navigate to "Virtual Networks".





2. Click "Add" and supply the following:
• Name: Type a name for your Virtual Network; you should follow the naming standards as discussed above.
• Address Space: This is the overall space for your logical network within Azure. You can create more granular subnets within this space at any time, so erring on the side of a large subnet would be wise. If you're unsure, use 10.1.0.0/16.
• Subnet Name: You're required to create one subnet within your virtual network off the bat. You need to name it here and ensure you use a consistent and meaningful naming standard.
• Subnet Address Range: Specify a subnet range within your virtual network. This won't be used by our IPsec connection directly, but we will use it later as a target for testing. If unsure, use 10.1.10.0/24.
• Subscription: Select your desired subscription.
• Resource Group: Select "Use Existing" and select the resource group we created earlier.
• Location: Select the same location used for the IP above. 





3. After the Virtual Network has been created (use the refresh key if necessary), click it to navigate to the next pane, and then click "Subnets".





4. On the next pane, click "+ Gateway Subnet". And specify a subnet in "Address Range". This subnet needs to be different than the one we created earlier and should not be used for non-network resources, but rather as an ingress point to your Virtual Network. If unsure, use 10.1.0.0/24.

Virtual Network Gateway

The "Virtual Network Gateway" is our configuration element that facilitates the IPsec tunnel. Microsoft refers to this as a "VPN" gateway (as opposed to Express Route). There are three different VPN gateway SKUs; we'll be doing the "Standard" offering (of Basic, Standard, High-Performance). It's worth having a read about the differences here.

1. Navigate to "Virtual Network Gateways".





2. Click "Add" and supply the following:
• Name: Again, follow consistent naming standards.
• Gateway Type: Select "VPN".
• VPN type: Select "Route-Based" (packets routed by routing table) in this case; it would be advisable to familiarize yourself with the difference between route and policy here. Note that policy requires IKEv1, so if you need to use it note the settings will be quite a bit different.
• SKU: Update 1/2018: The SKU selection is now VpnGw<x> or Basic. Note you cannot change a basic VNG to the higher tier (VpnGwX) or vice versa at a later time. For more information see this article.
• Virtual Network: Select the virtual network we created in the last step.
• Public IP address: Select the IP address we created earlier.





3. Click "Create".

Note: This step may take up to 45 minutes to complete provisioning. I've tracked 8 of these and it's averaging almost 40 minutes per regardless of if you pre-provision the IP or not. You may want to consider skipping ahead to the pfSense section for a bit and coming back here.

Local Network Gateway/Connection

A Local Network Gateway is the specification of our local IP and networks you would like to route over the tunnel. 

This actually works fine with a dynamic IP if that is your scenario, but we'll cover the details of that later.

1. Navigate to "Local Network Gateways".





2. Click "Add" and supply the following:
• Name: Naming? Standards? Consistent? Yeah!
• IP Address: Enter the public IP address of your device that will instantiate the tunnel.
• Address Space: This is where you enter the CIDR notation of the local networks you would like to route over the tunnel... for example, if you would like to route 192.168.1.x over the tunnel, then enter "192.168.1.0/24"
• Subscription: Enter your desired subscription.
• Resource Group: Select our resource group we created above.
• Location: For consistency, select the same location as you have selected above.
• Update 1/2018: You can configure BGP settings here now as well, cool eh?





3. Click "Create".
4. After provisioning, (you may need to hit "refresh) click your newly created Local Network Gateway and click "Connections".





5. On the newly expanded pane, click "Add" and supply the following:
• Name: You know the drill by now.
• Connection type: This should be fixed to "Site to Site (IPsec)"
• Virtual Network Gateway: Enter the Virtual Network Gateway we entered in the step above.
• Shared Key: Specify a unique, randomly generated passphrase comprised of alphanumeric characters. Some devices have issues with special characters, hence the omission of. I recommend using at least 30 characters; since it has no impact on tunnel performance I personally use at least 60 characters for each key. You'll need to specify this key on your local side as well.
• Subscription: This should be hard coded to the same subscription as the LNG.
• Resource Group: This also should be locked to the same resource group as the LNG.
• Location: Locked to that of the LNG.





6. Click "OK".

Configure pfSense

Now we'll set up the IPsec initiator connection on your pfSense firewall(s).

Phase 1 Setup

1. Login to the firewall and navigate to "VPN->IPsec"
2. Click "Add" and specify the following:
• Key Exchange Version: Auto
• Internet Protocol: IPv4
• Interface: Select the WAN interface from which you would like to instantiate the connection
• Remote Gateway: Enter the Azure public IP address created in the "Public IP Address" section above
• Description: Whatever you would like; maybe troll your firewall team with a message here for fun times.
• Authentication Method: Mutual PSK
• Negotiation Mode: Main Note: Do not use "Aggressive" mode as the hash of the PSK is sent over the internet in clear text.
• My Identifier: If the WAN interface selected above holds your public IP address, you can select "My IP Address". If that interface lies behind another edge device that holds the public IP, you'll need to select "IP Address" and specify your external IP.
• Peer Identifier: Peer IP Address
• Pre-Shared Key: Enter the same Pre-Shared Key used in the Azure connection specification above.
• Encryption Algorithm: The strongest available in Azure is AES 256 bit, so preferably specify that. For more information on supported features in Azure, see the References section below.
• Hashing Algorithm: The best we can do here is SHA256, so let's go with that.
• DH Group: 2(1024 bit)
• Lifetime (seconds): 10800
• Disable Rekey: Unchecked
• Responder Only: Unchecked
• NAT Traversal: Auto (Even in NAT scenarios Auto usually works)
• Dead Peer Detection: Checked
• Delay: 10
• Max Failures: 5





3. Click "Save" to return to the "VPN->IPsec" menu.
4. Since we don't want to use this yet, click "Disable" in front of the new tunnel definition and then "Apply Changes".

Phase 2 Setup

1. Under our newly created tunnel definition, click "Show Phase 2 Entries"
2. Click "Add P2" and supply the following information:
• Disabled: Unchecked
• Mode: Tunnel IPv4
• Local Network: Select "Network" and specify the same network range(s) that you specified during the set up of the local network gateway on Azure using CIDR notation, i.e. 192.168.1.0/24. This specifies which local network(s) you would like to route through the tunnel.
• NAT/BINAT translation: None  ; Note: even in scenarios where your pfSense device is using NAT behind an upstream router, this should not be necessary. NAT-T will take care of that scenario.
• Remote Network: Select "Network" and specify the same network range(s) that you specified during the set up of the target virtual network in Azure using CIDR notation, i.e. 10.1.0.0/16. This specifies the remote network(s) present in Azure.
• Description: Put something here to help you remember what all this fun stuff is about.
• Protocol: ESP
• Encryption Algorithms: Check only "AES" and "256 bits".
• Hash Algorithms: Unfortunately Azure only supports "SHA1" at this time. Update 1/2017: SHA256 supported now, use that!
• PFS Key Group: Azure documentation states that PFS groups are only supported when Azure acts as responder, and in this case it is being set up as the initiator. Oddly, I've actually had luck specifying DH Group 14, but there is no guarantee that will work. I'm going to stick with it but for this by the book exercise you'll need to select "off". Note: Because of this setting and the prior Hash Algorithm setting, I do not consider this tunnel secure against state-level or similarly equipped actors. If that is a concern you may wish to investigate alternatives. In less extreme cases, however, this can be considered relatively secure. Update 1/2017: The compatibility has been improved here as well; match your Encryption/Auth with the right group using the table here.
• Lifetime: 3600
• Automatically ping host: blank





3. Click "Save".  

Note: Depending on your configuration it may be necessary to navigate to "VPN->IPsec->Advanced Settings" and check "Enable Maximum MSS", then specify 1350. If you get packet loss with large packets this setting may be needed.

Firewall Rules

Now that our tunnel is set up we have to create local firewall rules that allow for traffic to pass. First we'll create a network alias for the Azure side network and then we'll make a rule to allow out Azure based traffic to pass here.

1. Navigate to "Firewall->Aliases"
2. Click "Add" and supply the following:
• Name: Supply something that explains this network is to represent the Azure side of the tunnel; only alphanumeric and "_" are allowed.
• Description: Enter full description here; there are no special character limitations.
• Type: "Network(s)"
• Network(s): Enter the CIDR notation of the network you created for your Virtual Network in Azure. If you followed the example addresses in this article, that would be 10.1.0.0/16.
3. Click "Save" and then "Apply Changes".
4. Navigate to "Firewall->Rules->IPsec".
5. Click "Add -^" and supply the following:
• Action: "Pass"
• Disable this rule: Unchecked
• Interface: "IPsec"
• Address Family: "IPv4"
• Protocol: "Any"   Discussion: Feel free to limit the traffic that goes through the tunnel if you like. In this example I'm allowing all traffic through.
• Source: "Single host or alias" and then specify the Azure network alias you created in step 2.
• Destination: This needs to be the local network(s) to which you would like to allow traffic. You can either do "network" with a CIDR notation or specify the entire network represented by an interface on the firewall. Note: if you have multiple networks you'll need a rule for each, so repeat the last couple steps for each.
• Log: Unchecked; keep in mind that should you need to troubleshoot temporarily logging traffic using this rule can be very useful.
• Description: It's a description, so let's do that!
• No advanced options necessary unless you would like to do so.
6. Click "Save" and then "Apply Changes".

As long as you have a blanket egress traffic rule we should now be able to route traffic over the tunnel. If you do not I expect you are aware of how to make a more specific rule to suit your needs.

A Note on NAT-T and Upstream Routers

If your pfSense device is behind another upstream router, you may need some changes to facilitate the port switchover after initialization. If this matches your configuration, consider that you may need the following on the upstream router:

• A firewall rule that allows UDP port 4500 into your pfSense device(s).
• A NAT port mapping rule that forwards UDP port 4500 to your pfSense device(s).

Try without first; some devices are aware enough of the switch to 4500 to perform the transition without rules, but if it does not work consult the documentation for the device in question.

Enable and Test

There are several ways to test our connection; in this case I'll be pinging a VM host in Azure assigned to the same virtual network that this tunnel is connecting to. We won't go through the provisioning of that; should you need to refer to this basic guide and ensure you place the VM in your target virtual network and initially created subnet (10.1.10.x in the example above).

We're about to go into a tunnel... a long one.

Preparing Your Target

• Ensure your VM is up and provisioned in the correct target virtual network.
• Since you can't put anything in the gateway subnet (correctly) this would be a good opportunity to put the VM in the subnet you were forced to create when creating the virtual network in the first place. Check/change VM->Network interfaces->Details->Settings->IP Configurations
• Get the private IP address from VM->Network interfaces. It should be 10.1.10.x if you're following the example addresses in this article.
• Make sure your VM is pingable! If you have instituted Network Security Groups that would inhibit access you'll need to modify them, though this should work by default since we're tunneled in. Make sure the firewall on the VM allows for incoming ICMP requests as well; on Win 2012 and higher set-netfirewallrule -DisplayName "File and Printer Sharing (Echo Request - ICMPv4-In)" -Enabled True will take care of you.

Bring Up The Tunnel

1. In the pfSense interface, navigate to VPN->IPsec.
2. In front of our new tunnel, click "Enable" then "Apply" toward the top.
3. Check tunnel status under Status->IPsec. The tunnel should come up automatically in about a minute. If there is trouble you can check the Status->System Logs->IPsec section for more details.

Check Tunnel Status in Azure & Ping Dat VM!

For this portion we'll use PowerShell; ensure you have the Azure ARM cmdlets installed. If not, give install-module AzureRM a shot from an elevated PowerShell prompt.

1. Login to your account: Login-AzureRmAccount
2. Look at your subscriptions and grab the name of the target sub: Get-AzureRmSubscription
3. Change to your correct subscription: Select-AzureRmSubscription -SubscriptionName <subscription name>
4. Check the status: Get-AzureRmVirtualNetworkGatewayConnection -name <Local Gateway Connection> -ResourceGroupName <Name of Resourcegroup to which it belongs>
5. On the output pane, check the "ConnectionStatus" property. It should be "Connected".

The Get-AzureRmVirtualNetworkGatewayConnection has a series of other interesting properties as well, including EgressBytesTransferred and IngressBytesTransferred.

Now proceed to ping your VM by the private IP listed in Azure. As long as everything is configured correctly you should receive a response!

Cost

VPN Tunnels are subject to a costs from a few different categories:

• VPN Gateway Pricing: This is an hourly cost incurred while the tunnel is available, not necessarily used. This means once it's provisioned you will incur charges at the hourly rate. As of the writing the standard performance level that we'll be using is billed at $0.19/hr in the US. If you have multiple Virtual Networks you will also be subject to a fee for outgoing traffic destined for another VNet. This rate depends on the zone and varies between $.035 and $.16 per GB. Data outbound to your site is charged at the standard data transfer rates (below) and inbound data is free.  Update 1/2017: Updated pricing for the new tiers can be found here.
  
• Data Transfer Rates: This depends on your level of utilization. The first 5GB of outgoing/month is free and the prices are set on a curve thereafter.
  
• Virtual Network Pricing: VNets are free; you can have up to 50 VNets per subscription across all regions.
  
• IP Addresses: You'll be using at least one IP address. The first 5 static in a given region are free, additional and dynamic are charged at a rate of $.004/hr.

Overall the cost of a "standard" class data tunnel each month for a single IP address, no additional support, and without including outgoing bandwidth, is about $140/month. 

Note: Costs as of 10/26/2016, subject to change. Up to date pricing information is available here.

Dynamic IP? Changing Your IP Address

There is no reason that this IPsec tunnel will not work without a dnymic IP, but each time the IP changes you'll need to take a series of steps to restore tunnel functionality. These are:

1. In Azure, the local network gateway specifies your IP, change it under "Local network gateway-><NAME>->Configuration->IP address".
2. In pfSense, the Phase 1 tunnel definition under "My identifier" needs to reflect your current external IP address.
3. If there are any implications on upstream  routers you'll need to handle that as well.

After taking care of these you will need to restart the tunnel. This could all be automated with PowerShell and SSH if you like, but I won't be covering that here.

Update 1/2017: FWIW, over the last year none of my clients have had their IP rotate for an active tunnel.

A Note on Effective Security

As mentioned earlier, this set up does have a couple security issues; the impact of which I would like to discuss briefly. Without an optimal security configuration, including the support of Perfect Forward Secrecy, this tunnel may not be strong enough to stand up to attacks of a state-sponsored actor over a long period of time. Because of that I cannot recommend this solution if your traffic may be subject to that level of attack, for example traffic facilitating substantial financial transaction activity. Update 1/2017: As noted above, this situation has improved with the support of PFS and SHA256 for authentication.

With that said, this tunnel is still (for better or worse) more secure than the configuration I have seen at many clients, and should be suitable for most traffic. It also performs very well; added latency between my modestly equipped pfSense devices and Azure is trivial.

For science!

References

Azure

Microsoft: Azure Virtual Network Overview
Microsoft: Azure Resource Manager Cmdlets (PowerShell)
Microsoft: Create a VNet with a Site-to-Site connection using the Azure portal
Microsoft: Create a VNet with a Site-to-Site connection using PowerShell
Steve Buchanan: Azure & RRAS Site to Site VPN Setup (Azure Resource Manager)
Microsoft: What is a Network Security Group?
Microsoft: VPN Gateway FAQ
Microsoft: About VPN Gateway
Microsoft: How to install and configure Azure PowerShell
Microsoft: About VPN Devices for Site-to-Site VPN Gateway Connections (Critical article; contains IPsec specs of Azure side)

pfSense

pfSense Doco: Routing internet traffic through a site-to-site IPsec tunnel
pfSense Doco: VPN Capability IPsec

IPsec

PFS RFC
strongSwan Wiki: Security recommendations
Johannes Webber: Considerations About IPsec Pre-Shared Keys
Cisco Support: How Does NAT-T Work With IPsec (Only needed if behind NAT)

Cloudy I/O Performance - Increasing Azure IOPS (Part 2 of 2)

Notes: This is part 2 of a 2 part post. You can find part 1 here. Since publication of this article Microsoft has introduced new machine types with higher performing storage. More information can be found here.

Foreword

In the last article we discussed a repeatable testing methodology to quantify storage performance in the cloud, and in this article we'll put that methodology into practice. I've done substantial testing in Azure and aim to illustrate what your options are for scaling performance at this point in time.

Scope

I undertook this project to see what can be done to increase disk I/O in Windows Azure IaaS. Upon researching the topic I found several interesting articles. Among those are:

• Timothy Khouri: Windows Azure IaaS Performance – SQL and IOps (Link removed, site was exploited)
  
• Timothy Khouri: Configuring Disks in Azure for More Performance (Link removed, site was exploited)
  
• MSDN: Performance Guidance for SQL Server in Windows Azure Virtual Machines
• MSDN: Getting Started with SQL Server in Windows Azure Virtual Machines
• Azure: Storage Pricing Details
• Azure: VM Pricing Details
• Virtual Machine and Cloud Service Sizes for Windows Azure
• Azure Blogs: Windows Azure’s Flat Network Storage and 2012 Scalability Targets
  

There seems to be little consensus regarding disk striping in Windows Azure IaaS. Some blogs recommend this while some of Microsoft's own writing seems to discourage it. After combing through the options the following points stand out:

• Disk Striping (Software RAID 0) may or may not increase performance based on your workload.
• Striping will increase I/O capacity to a degree (which we'll test here).
• What software striping solution works better: legacy (Windows software RAID from 2000 to present) or Storage Spaces (new software "RAID" in Windows 2012 and up)?
• How does NTFS cluster size impact performance?
• If striping, disable geo-replication as Microsoft explicitly warns against the use of geo-replication with this solution.
• If possible, use native application load distribution rather that disk striping to split I/O.  (For example, split DB files in SQL across disks)
• Some articles reference needing to use multiple storage accounts to get maximum performance. This is not true; as of 6/7/2012 storage account targets are 20,000 IOPS per account. Unless you will exceed the 20,000 keep all your disks on one account for the sake of simplicity. We will prove that does not have an impact on performance.

With that said, I want to quantify the solution for my given scaling problem with the notion that if the tests are simple enough to run, this approach can be used for any future scaling problem as well.

Putting it All Together

We'll use the testing methodology outlined in part 1 of this article to collect the results. In this case we need to first add disks and set up stripes in Azure Windows VMs.
Note: To jump straight to Azure disk performance tests, scroll to the bottom of this article.

Create New Disks and Attach to Designated VM

In order to run all the tests listed below, you need to know how to create new disks and attach them to your virtual machine. My favorite solution to this is to use a locally created dynamic VHD and upload it to the location you would like using PowerShell. Let's go through the process of attaching one disk as a primer:

1. Decide which storage account you will use for these disks. If you plan on doing striping of any kind, ensure the storage account is set to "Locally Redundant" replication (Storage->Desired Storage Account->Configure), as "Geo Redundant" is not supported. Since the replication setting applies to all blobs (Azure's terminology; disks) in that account you may want to have a dedicated account for these disks to keep your others Geo Redundant.


 


2. Determine what container you would like to store your Azure disk blob by opening the Azure management portal and navigating to Storage->Desired Storage Account->Containers and copy the URL to your clipboard. To keep things simple you may want to create a new storage container, so do so now and use that URL if desired.
3. Using Hyper-V (On Windows 2008 or higher including Windows 8) create an empty dynamically expanding VHD disk of your desired size. For my testing I have been using 10GB disks. Note 1: Do not create a VHDX; Azure uses the older VHD format. Note 2: You'll need to re-create the VHD for each disk if you intend on using Storage Spaces as each disk must have a unique ID. 

#create a dynamically expanding 10GB VHD; change size as appropriate
New-VHD –Path $sourceVHD –SizeBytes 10GB -Dynamic

4. This disk will be uploaded to the container we selected in step 1. Determine the name you want the disk to be referenced by in Azure and execute the following script:

#import Azure cmdlets
import-module azure.psd1
#specify your subscription so PS knows what account to upload the data to
select-azuresubscription "mysubscriptionname"
#$sourceVHD should be the location of your empty vhd file
$sourceVHD = "D:\Skydrive\Projects\Azure\AzureEmpty10G_Disk.vhd"
#$destinationVHD should be the URL of the container and the name of the vhd you want created in your account. Obviously for subsequent disks you need to change the VHD name. 
$destinationVHD = "http://yourstorageacctname.blob.core.windows.net/vhds/data02.vhd"
#now upload it. 
Add-AzureVhd -LocalFilePath $sourceVHD -Destination $destinationVHD






5. Add this new disk as available to VMs by navigating to Virtual Machines->Disks->Create





6. Enter the desired management name for this disk and input or browse to the URL of the VHD you just uploaded and click the check box.
7. Attach the disk to your VM by navigating to Virtual Machines->Ensure your desired VM is highlighted->Attach->Attach Disk





8. Select the disk we just added. Your cache preference will depend on the application. In my case this is off but you will want to use the methodology outlined in the first part of this article to test caching impact for your application. Note a change of cache status requires a VM reboot.

Now for a brief tutorial on how to set up our two types of striped disks; you'll likely only be using one of the two but I'll cover both just in case. Performance results of each are outlined later in this article.

Set Up a Traditional Software Stripe in Windows

Setting up a traditional software stripe is easy. I've tested this on Windows 2003 and higher.

1. Logon to your VM as an admin and open the Disk Management tool.
2. If prompted, allow the initialization of the disks.
3. Right-click on one of the newly created empty volumes and select New Striped Volume.





4. Select the desired disks and continue.





5. Create and format a new NTFS disk using your striped volume. Make sure to pay attention to the cluster size (results below).

Setup a Storage Spaces Software Stripe in Windows 2012 or Higher

Microsoft introduced a new approach to disk pooling in Windows Server 2012 and Windows 8 called Storage Spaces. This interesting new tech allows for a myriad of different configuration options including disk tiering which can be useful for on-premise servers. In this case we'll be using the "simple" pool type which is similar to disk striping.

1. Open Server Manager and navigate to File and Storage Services -> Volumes -> Storage Pools
2. Under Storage Pools you should see "Primordial". (As opposed to "Unused Disks". I'm guessing someone was pretty proud of that.) Right click it and select "New Storage Pool".





3. Walk through the Wizard selecting each disk you would like to be part of the pool.





4. On the results page, ensure "Create a virtual disk when this wizard closes" is selected and click "Close".





5. Walk through the Virtual Disk Wizard, specifying a meaningful name and selecting simple storage layout and fixed provisioning.





6. On the results page, ensure "Create a volume when this wizard closes" is selected and click "Close".
7. Complete the New Volume Wizard specifying your desired drive letter and desired NTFS cluster size.

Run Tests/Collect Results!

Now that we have our disks configured, we need to run our tests. For instructions how how to do so, see part 1 of this topic here.
When analyzing the IOMeter output you will want to pay special attention to the following metrics:

• IOPS (Cumlative, Read, Write, Higher is better)
• MBps (Cumlative, Read, Write, Higher is better)
• Response Time (Avg, Avg Read, Avg Write, lower is better) 

If putting the data together for a report, Excel works nicely as I'll display below.

Results

Now for the most important part, the findings. Tests performed:

Sector Size Tests:

• 1 Disk, 4k Sector Size (default)
• 1 Disk, 8k Sector Size
• 1 Disk, 16k Sector Size
• 1 Disk, 32k Sector Size
• 1 Disk, 64k Sector Size
• 3 Disks, 4k Sector Size (results confirmation test)
• 3 Disks, 32k Sector Size (results confirmation test)

Table 1-Cluster Size Tests

Table 2-Cluster Size Verification

Sector size tests echo what others have observed with Azure; since IOPS are capped at 500 (or 300 for basic VMs) larger sector sizes can result in higher throughput. In my case 32k was the sweet spot; depending on your workload your results will vary slightly. I have seen consistently (albeit slightly) higher performance with larger sector sizes in Azure.

Legacy Disk Striping Tests:

• 1 Disk, 32k Sector Size
• 2 Disks, Striped Volume, 32k Sector Size
• 2 Disks in 2 Storage Accounts, Striped Volume, 32k Sector Size (Multiple Storage Account Test)
• 3 Disks, Striped Volume, 32k Sector Size
• 4 Disks, Striped Volume, 32k Sector Size

<See Bar Charts Below Under Disk Striping Methodology>

Table 3-Legacy Striping and Storage Account Tests

You can see with one disk we get 500 IOPS as expected. From there we can see a scaling trend that is most definitely not linear. Two disks result in 33% higher performance, while three disks add an additional 23% (64% from one disk). Adding the fourth disk actually results in a drop from three disks, coming it at 5% lower than three and 56% higher than one.
Additionally, we also see that splitting disks across storage accounts makes no appreciable difference.  Note: Bar charts for this results section have been combined into the graphs below.

Disk Striping Methodology Tests:

• 2 Disks, Striped Volume 32k Sector Size
• 2 Disks, Storage Spaces Simple, 32k Sector Size
• 3 Disks, Striped Volume, 32k Sector Size
• 3 Disks, Storage Spaces Simple, 32k Sector Size
• 4 Disks, Striped Volume, 32k Sector Size
• 4 Disks, Storage Spaces Simple, 32k Sector Size

Table 4-Legacy Striping vs. Storage Spaces Test

Now we compare legacy striping to the newly introduced Storage Spaces. Two disk scaling is a definitive win for Storage Spaces, while beyond that legacy striping generally performs better (save max latency). In my opinion two disk Storage Spaces stripe is the sweet spot here (56% IOPS improvement!) when considering that the with more disks we add complexity that doesn't pan out on the performance side.

Conclusion

I hope you have found these results interesting; I certainly have. Even if you choose not to run these tests yourself I hope my results prove helpful when sizing your machines. Since the access pattern I used is relatively universal it should be applicable in most scenarios.
Software level disk striping works relatively well in Microsoft Azure to increase per-disk performance in lieu of a provider level solution similar to Amazon EBS provisioned IOPS. Splitting the workload across logical disks or VMs is preferred but not applicable to all workloads. When employing this solution make sure you select only locally redundant replication because Micrsoft warns that geo-redundant replication may cause data consistency issues on the replication target.
For additional information see the links near the top of this article. Thanks for reading!

Cloudy I/O Performance - Deciphering IOPS in IaaS (Part 1 of 2)

Note: This is part 1 of a 2 part post. Part 2 can be found here.

Foreword

Disk performance scaling options in the public cloud seem limited (particularly in Azure as of this writing), but there are ways to increase your IOPS in IaaS solutions. To add to the performance problem, transactional costs of running application tests can be not only time consuming but expensive. To tune your storage performance reliably you will need a fast, consistent way to test different configurations. This article will cover that methodology and lead into a results/guidance article for Azure (but applicable to others) IaaS storage performance.

We'll be doing this testing on Windows, but you could also easily do this on Linux and the results that I'll be sharing are just as applicable there. To accomplish this testing we'll be using the following tools:

• IOMeter (for Linux try iorate or iometer on Wine)
• PowerShell cmdlets for Azure
• Disk Management

Let's begin!

Execution

We will proceed in the following order:

1. Analyze Workload
2. Create Test Scenarios
3. Collect and Analyze Results (Mainly in Part 2)
4. Findings (In Part 2)

Assumptions

If you plan on emulating my tests you'll need to have access to the following:

• Microsoft Windows Azure account (note this methodology will worth with EC2 or any other platform including standard hardware/on-prem VMs)
• IaaS VM Configured. A medium size is recommended for testing 4 disks or fewer to limit the available memory. More on that below.
• Administrator access to your VM.
• Your workload is in fact disk I/O bound. If you're not sure of that you may want to start with this article.
• Awareness that you will incur additional storage transaction costs by running these tests.

Analysis/Create Workload

Note: If you're just trying to get a general sense for your VM I/O performance capability, you don't need to collect data for a custom access specification. IOMeter includes several tests you can use so skip to the "Install IOMeter..." section below.

The first thing we need to do is create our workload. By using IOMeter we can develop custom access patterns that model common workloads and have the tool and workloads installed and configured in minutes on any machine. There is nearly endless information on this topic, so I won't attempt to create a definitive source here. For details on how to configure and use IOMeter, see the following videos/articles:

• Video Tutorial from iTechStorm
• Kristau.net Short Tutorial/Intro
• IOMeter Users Guide
• VMWare Community:Storage System Performance Analysis with Iometer

 To create an accurate workload you will need a good understanding of the access pattern of your application. If you don't have that information you can use a tool like Perfmon to do analysis on a fully configured platform. The following counters will be of interest when creating your access specification:

• Physical or Logical Disk: Average Disk Bytes per Read
• Physical or Logical Disk: Average Disk Bytes per Write
• Physical or Logical Disk: Disk Read Bytes/sec
• Physical or Logical Disk: Disk Write Bytes/sec
• Physical or Logical Disk: Disk Reads/sec
• Physical or Logical Disk: Disk Writes/sec

For further information, see this excellent Technet Article.

By collecting this data during the access pattern you wish to emulate you can accurately estimate (with one caveat) the information needed to create the IOMeter access specification. That caveat is determining the sequential vs. random access pattern of the platform since Perfmon analysis will reveal the rest. To determine that, you'll need an understanding of how the platform stores and accesses/writes data. In my case I'm tuning my VM for Splunk, which uses a Map/Reduce functionality that has a highly sequential read/write pattern. If you are unsure of your access pattern then err on the side of configuring for mostly random access (90% or so) since it is generally more common and demanding of the underlying storage subsystem. 

Install IOMeter and Config Access Specification

The following actions can be done on your target testing platform or a different machine to stage settings. We'll be saving our settings for quick use later.

1. Download and install IOMeter on your server. There are a series of ways to stage files on any VM, but if you're looking for a quick way in the Microsoft ecosystem check out my Onedrive/Azure post.





2. Open IOMeter as administrator.





3. Under "Topology" configure your workers. Each worker represents one thread generating I/O. By default it will create one per CPU thread available, but in most cases you will only want one worker per process you are emulating. In my case I'm assuming one large query at a time (and we'll scale from there), so I'll be testing with one worker. If you are unsure stick to one worker and you can move up from there when you become more familiar.





4. Under "Disk Targets" select the disk you wish to test. This can change in later runs so if the disk you want to test isn't present here select a placeholder.
5. Under "Disk Targets" configure your "Maximum Disk Size". This configures the size of your test file in sectors, which are considered to be 512 bytes each. To lessen the impact of OS caching you need to ensure this value exceeds the amount of RAM present on the machine to be tested. In my case I'll be testing on a 6GB RAM machine with a (approx) 7.5GB file, so I've configured it for 15000000 sectors. (15000000 sectors * 512 bytes per sector=7,680,000,000 bytes)  To do this quickly take your total desired size (in bytes!) and divide it by 512. (If you aren't certain you got it right, check the size iobw.tst file created at the root of your target drive after the first test is complete)





Testing T: With a 4.5GB Test File



6. Under "Disk Targets" configure your maximum outstanding I/O. This varies depending on access spec and OS, but I've had consistent (with real application access) results testing with 16 maximum outstanding I/O on windows. 
7. Under "Test Setup" configure your "Ramp Up Time" and "Run Time". Ramp up need only be about 20 seconds for most scenarios and run time is best between 1 and 10 minutes. My results are based on (many per config) 5 minute tests. 
8. Under "Access Specification" select your access spec. There is far too much to get into here; either select one or many existing access specifications that suit you needs ("4k 75% read" is a good start if you don't care) or create your own based on your findings from the Analysis/Create workload section. For the purposes of my test I made a "_Splunk" access spec with the following characteristics ascertained from my earlier performance testing:
1. Transfer Request Size: 32kB (NOTE: My access spec may not reflect yours. Most won't be this large)
2. Percent Read/Write Distribution: 53% Write/47% Read (NOTE: My access spec may not reflect yours. Most specs won't be this write heavy)
3. Percent Random/Sequential Distribution: 75% Sequential/25% Random (NOTE: My access spec may not reflect yours. Most specs won't be this sequential)




9. Add your access specification to the list of queued tests if you haven't done so already (removing all others).





10. Click the disk icon to save the settings to an ICF file. This file will save all your settings including custom access specifications if applicable. Since this file is what you'll use to shortcut future testing, save it somewhere easy to transfer to other VMs such as OneDrive, Dropbox, SpiderOak, etc.

Run the Test

After setting up or loading your test settings, all you need do is click the green flag to start the test and then select where you would like to save the results. Make sure you don't overwrite any previous results and give the file a meaningful name so you remember what this test represents later, i.e. "results_3disk_1_StorAcct_Striped_32k_sectors_noCache_run1.csv" or similar. 

The test will run for the configured time and then you will be able to run additional tests or analyze results. Since the output is in CSV format, the natural place to look at this data is Excel. When IOMeter starts for the first time on a given disk it needs to create the test file. This will take quite awhile in both Amazon EC2 and Azure. (15 mins for my 7.5G for example) I believe this is due to the way space is allocated on the backend storage. Once this is created, however, you can run subsequent tests on the same volume without needing to wait for the test file to be created. Once the run is done I recommend running several more to ensure your tests aren't subject to wild performance swings. More on analysis in part 2 of this article.

How Much Will This Cost?

Since you're charged by transaction I'm sure you will be wondering how much this will cost. Let's break down your above baseline (system running) cost in Azure:

IOPS are currently capped at 500 for standard tier machines (300 for basic). Storage transactions are currently $0.01 per 100,000. (halved on 3/14/14) For every 5 minute test per disk you access you will then execute a maximum of 150,000 transactions. As a one time per configuration cost, you will need to build the test file which will be (test file size/volume sector size) transactions. For example, a 7.5 Gib test file will be approximately 1,875,000 transactions assuming a default 4kb sector size. (7,500,000,000/4000)

Test transactions + creation transactions = 2 million or so IOPS, or $0.20 @ .01 per 100,000. So... not much. The amount is generally trivial on Amazon EC2 as well. While this methodology will save you some in transaction costs, the main savings will be in time & labor. (which is usually our real cost anyhow!)

Further Optimization

Once you are comfortable with this process I would advise doing the following to further optimize this process. After doing so you may be able to automate the whole routine!

• Create standard Perfmon counter sets for disk access and save/import them as a template. 
• Script the Perfmon analysis with PowerShell. 
• Create or download IOMeter templates for common access routines and include them with your set.
• Script the installation and running of IOMeter, including multiple runs and uploading results to a common location. Easy to do with PowerShell and refer to the IOMeter manual for command line options (page 75 or so).
• Package up all your assets with a custom installer and put it in an easy to get location. (mmmm... Chocolatey)
• If you want angry followers and think digital bits are out there to be wasted, auto tweet your results! (maybe not this)

In Closing

I/O testing in the cloud is certainly feasible but requires a little extra discipline. With several access specifications in your toolkit you can conquer most performance problems quickly. What to do if your cloud platform doesn't provide your desired IOPS? Coming up in part 2!