Thursday, October 27, 2016

Tunnel to the Cloud: pfSense to Azure Site to Site IPsec Connection

Do you have multi-continent datacenters with gobs of bandwidth, IOps, and processing capability? No? I can help get you part of the way there... a network presence in one.

A tunnel.

A Site to Site Connection?


It's easier to think of this as an extension to your network into another datacenter over the internet. Using IPsec we can provide a relatively (comments at the end) secure, direct connection between on on-premises datacenter and Azure hosted resources by encrypting the traffic that flows between the two.  What do I mean by:
  • Secure: IPsec tunnels all your traffic so it is encrypted over the internet; in reality, this is really "more secure" rather than definitively secure, as the effective security depends highly on implementation specifics.
  • Direct: Your router (played by pfSense in this case) will recognize the Azure site as another routable network within the boundaries of your own, enabling you to talk to Azure resources as if they were in your own datacenter.

Note: this works for Amazon Web Services (AWS) as well but is slightly more complex. Fortunately pfSense includes a wizard that works, but takes a lot of the fun out of it as it strips you of understanding how it works. In addition the wizard is necessary because of how Amazon does VPC routing, whereas Azure is a bit more straightforward.

This article will also be useful for non-pfSense devices since we discuss the details of the IPsec tunnel; the information here should be applicable to any IPsec solution.

With that, let's get to it!


Pre-Requisites

  • pfSense firewall(s): The steps in this article were performed on a pair of  HA SG-4860 firewalls running pfSense 2.32p1.
  • Microsoft Azure account with adequate permissions: We'll be performing our actions using the "new" portal based on Azure Resource Manager (ARM or AzureRM).
  • AzureRM PowerShell Cmdlets installed: On Win10/Server 2016 this can be accomplished with Install-Module AzureRM; for more info see this post.

Configure Azure IPSec Endpoint

Before we set up and initiate the connection from pfSense, we need to set up our endpoint in Azure. To do so, we'll create the following objects:
  • A Resource Group
  • A Public IP Address
  • A Virtual Network
  • A Gateway Subnet
  • A Virtual Network Gateway
  • A Local Network Gateway with a Connection

Resource Group

A Resource Group is a logical grouping of Azure Resources. This logical group allows for easy organization and clearer billing reports. We won't get too much into concepts and naming standards here other than to say groups should be logically tied with similar lifecycle expectancies and you should be consistent. For more information, see this Azure article.

Note: We'll be doing most of our steps in the web portal, but this whole process is much more efficient with PowerShell.

Let's go!
  1. Open and log into the ARM Azure Portal at portal.azure.com; ensure you're working with the subscription you intend to use.
  2. Navigate to "Resource Groups".
  3. Click "Add" and type a name for your resource group, select the subscription, and resource group location. Note: The resource group location has no bearing on where you'll be connecting to as it's just a location the metadata is stored.
  4. Click "Save".


Public IP Address

While we could do the IP at the time we make the Virtual Network Gateway, we'll take care of it now to ensure it's provisioned prior to getting to that step and to discuss the IP details.
  1. Navigate to "Public IP addresses".
  2. Click "Add" and populate the following:
    • Name: Select a name leveraging consistent naming standards.
    • IP address assignment: Select "Dynamic". I know what you're saying.. you're saying "but Toby, I'm not saying anything", and what I'm saying is it seems this should be static. Unfortunately if we make a static IP we'll be greeted later with the following:

      Why Azure, WHY?!

      Now I've been running a tunnel straight for almost a month thus far and my dynamic IP has not shifted on me; I suspect it will behave the same as IPs for other resources and stay static so long as it is used. If this does change, you'll need to change the info in the Phase 1 and 2 setup of the tunnel on the pfSense side as outlined below. For the record, as of the writing of this article the pricing of IPs in Azure is a bit odd; dynamic IPs and static IPs beyond the first 5 in any region are charged the same (pretty trivial), while the first 5 static in a region are free. See here for more info.
    • Idle Timeout: The default of 4 minutes should be fine here.
    • DNS Name Label: Optionally, specify a DNS alias here, though we will not reference it again in this guide as I'm not addressing DNS issues associated with IPsec at this time.
    • Subscription: Select your subscription.
    • Resource Group: Click the resource group we created in the last step.
    • Location: The IP is our IPsec target, so select a location close to your local network connection. The Azure Speed Test comes in quite handy here.


  3. Click "Create". This provisioning will take a few minutes minutes as Azure re-arranges its SDN infrastructure to give you an IP.

  4. What a successful deployment looks like!

Virtual Network/Subnet/Gateway Subnet

A "Virtual Network" is a network space within Azure that you can carve up and protect (firewall) to suit your needs. We're required to make one subnet, and we'll create our "gateway subnet" (landing point) as well. If this is your first foray into virtual networks on Azure you may want to take a step back and consider your design before proceeding. Oh, you're back already? Let's go.
  1. Navigate to "Virtual Networks".


  2. Click "Add" and supply the following:
    • Name: Type a name for your Virtual Network; you should follow the naming standards as discussed above.
    • Address Space: This is the overall space for your logical network within Azure. You can create more granular subnets within this space at any time, so erring on the side of a large subnet would be wise. If you're unsure, use 10.1.0.0/16.
    • Subnet Name: You're required to create one subnet within your virtual network off the bat. You need to name it here and ensure you use a consistent and meaningful naming standard.
    • Subnet Address Range: Specify a subnet range within your virtual network. This won't be used by our IPsec connection directly, but we will use it later as a target for testing. If unsure, use 10.1.10.0/24.
    • Subscription: Select your desired subscription.
    • Resource Group: Select "Use Existing" and select the resource group we created earlier.
    • Location: Select the same location used for the IP above. 


  3. After the Virtual Network has been created (use the refresh key if necessary), click it to navigate to the next pane, and then click "Subnets".


  4. On the next pane, click "+ Gateway Subnet". And specify a subnet in "Address Range". This subnet needs to be different than the one we created earlier and should not be used for non-network resources, but rather as an ingress point to your Virtual Network. If unsure, use 10.1.0.0/24.


Virtual Network Gateway

The "Virtual Network Gateway" is our configuration element that facilitates the IPsec tunnel. Microsoft refers to this as a "VPN" gateway (as opposed to Express Route). There are three different VPN gateway SKUs; we'll be doing the "Standard" offering (of Basic, Standard, High-Performance). It's worth having a read about the differences here.
  1. Navigate to "Virtual Network Gateways".


  2. Click "Add" and supply the following:
    • Name: Again, follow consistent naming standards.
    • Gateway Type: Select "VPN".
    • VPN type: Select "Route-Based" (packets routed by routing table) in this case; it would be advisable to familiarize yourself with the difference between route and policy here.
    • SKU: For this tutorial we'll be selecting "Standard". Select a different class if it suits your needs.
    • Virtual Network: Select the virtual network we created in the last step.
    • Public IP address: Select the IP address we created earlier.


  3. Click "Create".

Note: This step may take up to 45 minutes to complete provisioning. I've tracked 8 of these and it's averaging almost 40 minutes per regardless of if you pre-provision the IP or not. You may want to consider skipping ahead to the pfSense section for a bit and coming back here.


Local Network Gateway/Connection


A Local Network Gateway is the specification of our local IP and networks you would like to route over the tunnel. 

This actually works fine with a dynamic IP if that is your scenario, but we'll cover the details of that later.
  1. Navigate to "Local Network Gateways".


  2. Click "Add" and supply the following:
    • Name: Naming? Standards? Consistent? Yeah!
    • IP Address: Enter the public IP address of your device that will instantiate the tunnel.
    • Address Space: This is where you enter the CIDR notation of the local networks you would like to route over the tunnel... for example, if you would like to route 192.168.1.x over the tunnel, then enter "192.168.1.0/24"
    • Subscription: Enter your desired subscription.
    • Resource Group: Select our resource group we created above.
    • Location: For consistency, select the same location as you have selected above.


  3. Click "Create".
  4. After provisioning, (you may need to hit "refresh) click your newly created Local Network Gateway and click "Connections".


  5. On the newly expanded pane, click "Add" and supply the following:
    • Name: You know the drill by now.
    • Connection type: This should be fixed to "Site to Site (IPsec)"
    • Virtual Network Gateway: Enter the Virtual Network Gateway we entered in the step above.
    • Shared Key: Specify a unique, randomly generated passphrase comprised of alphanumeric characters. Some devices have issues with special characters, hence the omission of. I recommend using at least 30 characters; since it has no impact on tunnel performance I personally use at least 60 characters for each key. You'll need to specify this key on your local side as well.
    • Subscription: This should be hard coded to the same subscription as the LNG.
    • Resource Group: This also should be locked to the same resource group as the LNG.
    • Location: Locked to that of the LNG.


  6. Click "OK".

Configure pfSense


Now we'll set up the IPsec initiator connection on your pfSense firewall(s).

Phase 1 Setup

  1. Login to the firewall and navigate to "VPN->IPsec"
  2. Click "Add" and specify the following:
    • Key Exchange Version: Auto
    • Internet Protocol: IPv4
    • Interface: Select the WAN interface from which you would like to instantiate the connection
    • Remote Gateway: Enter the Azure public IP address created in the "Public IP Address" section above
    • Description: Whatever you would like; maybe troll your firewall team with a message here for fun times.
    • Authentication Method: Mutual PSK
    • Negotiation Mode: Main Note: Do not use "Aggressive" mode as the hash of the PSK is sent over the internet in clear text.
    • My Identifier: If the WAN interface selected above holds your public IP address, you can select "My IP Address". If that interface lies behind another edge device that holds the public IP, you'll need to select "IP Address" and specify your external IP.
    • Peer Identifier: Peer IP Address
    • Pre-Shared Key: Enter the same Pre-Shared Key used in the Azure connection specification above.
    • Encryption Algorithm: The strongest available in Azure is AES 256 bit, so preferably specify that. For more information on supported features in Azure, see the References section below.
    • Hashing Algorithm: The best we can do here is SHA256, so let's go with that.
    • DH Group: 2(1024 bit)
    • Lifetime (seconds): 10800
    • Disable Rekey: Unchecked
    • Responder Only: Unchecked
    • NAT Traversal: Auto (Even in NAT scenarios Auto usually works)
    • Dead Peer Detection: Checked
    • Delay: 10
    • Max Failures: 5


  3. Click "Save" to return to the "VPN->IPsec" menu.
  4. Since we don't want to use this yet, click "Disable" in front of the new tunnel definition and then "Apply Changes".

Phase 2 Setup

  1. Under our newly created tunnel definition, click "Show Phase 2 Entries"
  2. Click "Add P2" and supply the following information:
    • Disabled: Unchecked
    • Mode: Tunnel IPv4
    • Local Network: Select "Network" and specify the same network range(s) that you specified during the set up of the local network gateway on Azure using CIDR notation, i.e. 192.168.1.0/24. This specifies which local network(s) you would like to route through the tunnel.
    • NAT/BINAT translation: None  ; Note: even in scenarios where your pfSense device is using NAT behind an upstream router, this should not be necessary. NAT-T will take care of that scenario.
    • Remote Network: Select "Network" and specify the same network range(s) that you specified during the set up of the target virtual network in Azure using CIDR notation, i.e. 10.1.0.0/16. This specifies the remote network(s) present in Azure.
    • Description: Put something here to help you remember what all this fun stuff is about.
    • Protocol: ESP
    • Encryption Algorithms: Check only "AES" and "256 bits".
    • Hash Algorithms: Unfortunately Azure only supports "SHA1" at this time.
    • PFS Key Group: Azure documentation states that PFS groups are only supported when Azure acts as responder, and in this case it is being set up as the initiator. Oddly, I've actually had luck specifying DH Group 14, but there is no guarantee that will work. I'm going to stick with it but for this by the book exercise you'll need to select "off". Note: Because of this setting and the prior Hash Algorithm setting, I do not consider this tunnel secure against state-level or similarly equipped actors. If that is a concern you may wish to investigate alternatives. In less extreme cases, however, this can be considered relatively secure.
    • Lifetime: 3600
    • Automatically ping host: blank


  3. Click "Save".  
Note: Depending on your configuration it may be necessary to navigate to "VPN->IPsec->Advanced Settings" and check "Enable Maximum MSS", then specify 1350. If you get packet loss with large packets this setting may be needed.

Firewall Rules

Now that our tunnel is set up we have to create local firewall rules that allow for traffic to pass. First we'll create a network alias for the Azure side network and then we'll make a rule to allow out Azure based traffic to pass here.

  1. Navigate to "Firewall->Aliases"
  2. Click "Add" and supply the following:
    • Name: Supply something that explains this network is to represent the Azure side of the tunnel; only alphanumeric and "_" are allowed.
    • Description: Enter full description here; there are no special character limitations.
    • Type: "Network(s)"
    • Network(s): Enter the CIDR notation of the network you created for your Virtual Network in Azure. If you followed the example addresses in this article, that would be 10.1.0.0/16.
  3. Click "Save" and then "Apply Changes".
  4. Navigate to "Firewall->Rules->IPsec".
  5. Click "Add -^" and supply the following:
    • Action: "Pass"
    • Disable this rule: Unchecked
    • Interface: "IPsec"
    • Address Family: "IPv4"
    • Protocol: "Any"   Discussion: Feel free to limit the traffic that goes through the tunnel if you like. In this example I'm allowing all traffic through.
    • Source: "Single host or alias" and then specify the Azure network alias you created in step 2.
    • Destination: This needs to be the local network(s) to which you would like to allow traffic. You can either do "network" with a CIDR notation or specify the entire network represented by an interface on the firewall. Note: if you have multiple networks you'll need a rule for each, so repeat the last couple steps for each.
    • Log: Unchecked; keep in mind that should you need to troubleshoot temporarily logging traffic using this rule can be very useful.
    • Description: It's a description, so let's do that!
    • No advanced options necessary unless you would like to do so.
  6. Click "Save" and then "Apply Changes".


As long as you have a blanket egress traffic rule we should now be able to route traffic over the tunnel. If you do not I expect you are aware of how to make a more specific rule to suit your needs.

A Note on NAT-T and Upstream Routers

If your pfSense device is behind another upstream router, you may need some changes to facilitate the port switchover after initialization. If this matches your configuration, consider that you may need the following on the upstream router:
  • A firewall rule that allows UDP port 4500 into your pfSense device(s).
  • A NAT port mapping rule that forwards UDP port 4500 to your pfSense device(s).

Try without first; some devices are aware enough of the switch to 4500 to perform the transition without rules, but if it does not work consult the documentation for the device in question.


Enable and Test

There are several ways to test our connection; in this case I'll be pinging a VM host in Azure assigned to the same virtual network that this tunnel is connecting to. We won't go through the provisioning of that; should you need to refer to this basic guide and ensure you place the VM in your target virtual network and initially created subnet (10.1.10.x in the example above).

We're about to go into a tunnel... a long one.

Preparing Your Target

  • Ensure your VM is up and provisioned in the correct target virtual network.
  • Since you can't put anything in the gateway subnet (correctly) this would be a good opportunity to put the VM in the subnet you were forced to create when creating the virtual network in the first place. Check/change VM->Network interfaces->Details->Settings->IP Configurations
  • Get the private IP address from VM->Network interfaces. It should be 10.1.10.x if you're following the example addresses in this article.
  • Make sure your VM is pingable! If you have instituted Network Security Groups that would inhibit access you'll need to modify them, though this should work by default since we're tunneled in. Make sure the firewall on the VM allows for incoming ICMP requests as well; on Win 2012 and higher set-netfirewallrule -DisplayName "File and Printer Sharing (Echo Request - ICMPv4-In)" -Enabled True will take care of you.

Bring Up The Tunnel

  1. In the pfSense interface, navigate to VPN->IPsec.
  2. In front of our new tunnel, click "Enable" then "Apply" toward the top.
  3. Check tunnel status under Status->IPsec. The tunnel should come up automatically in about a minute. If there is trouble you can check the Status->System Logs->IPsec section for more details.

Check Tunnel Status in Azure & Ping Dat VM!

For this portion we'll use PowerShell; ensure you have the Azure ARM cmdlets installed. If not, give install-module AzureRM a shot from an elevated PowerShell prompt.

  1. Login to your account: Login-AzureRmAccount
  2. Look at your subscriptions and grab the name of the target sub: Get-AzureRmSubscription
  3. Change to your correct subscription: Select-AzureRmSubscription -SubscriptionName <subscription name>
  4. Check the status: Get-AzureRmVirtualNetworkGatewayConnection -name <Local Gateway Connection> -ResourceGroupName <Name of Resourcegroup to which it belongs>
  5. On the output pane, check the "ConnectionStatus" property. It should be "Connected".

The Get-AzureRmVirtualNetworkGatewayConnection has a series of other interesting properties as well, including EgressBytesTransferred and IngressBytesTransferred.

Now proceed to ping your VM by the private IP listed in Azure. As long as everything is configured correctly you should receive a response!


Cost

VPN Tunnels are subject to a costs from a few different categories:
  • VPN Gateway Pricing: This is an hourly cost incurred while the tunnel is available, not necessarily used. This means once it's provisioned you will incur charges at the hourly rate. As of the writing the standard performance level that we'll be using is billed at $0.19/hr in the US. If you have multiple Virtual Networks you will also be subject to a fee for outgoing traffic destined for another VNet. This rate depends on the zone and varies between $.035 and $.16 per GB. Data outbound to your site is charged at the standard data transfer rates (below) and inbound data is free.
  • Data Transfer Rates: This depends on your level of utilization. The first 5GB of outgoing/month is free and the prices are set on a curve thereafter.
  • Virtual Network Pricing: VNets are free; you can have up to 50 VNets per subscription across all regions.
  • IP Addresses: You'll be using at least one IP address. The first 5 static in a given region are free, additional and dynamic are charged at a rate of $.004/hr.

Overall the cost of a "standard" class data tunnel each month for a single IP address, no additional support, and without including outgoing bandwidth, is about $140/month. 

Note: Costs as of 10/26/2016, subject to change. Up to date pricing information is available here.


Dynamic IP? Changing Your IP Address

There is no reason that this IPsec tunnel will not work without a dnymic IP, but each time the IP changes you'll need to take a series of steps to restore tunnel functionality. These are:
  1. In Azure, the local network gateway specifies your IP, change it under "Local network gateway-><NAME>->Configuration->IP address".
  2. In pfSense, the Phase 1 tunnel definition under "My identifier" needs to reflect your current external IP address.
  3. If there are any implications on upstream  routers you'll need to handle that as well.

After taking care of these you will need to restart the tunnel. This could all be automated with PowerShell and SSH if you like, but I won't be covering that here.


A Note on Effective Security

As mentioned earlier, this set up does have a couple security issues; the impact of which I would like to discuss briefly. Without an optimal security configuration, including the support of Perfect Forward Secrecy, this tunnel may not be strong enough to stand up to attacks of a state-sponsored actor over a long period of time. Because of that I cannot recommend this solution if your traffic may be subject to that level of attack, for example traffic facilitating substantial financial transaction activity.

With that said, this tunnel is still (for better or worse) more secure than the configuration I have seen at many clients, and should be suitable for most traffic. It also performs very well; added latency between my modestly equipped pfSense devices and Azure is trivial.

For science!

References


Azure

Microsoft: Azure Virtual Network Overview
Microsoft: Azure Resource Manager Cmdlets (PowerShell)
Microsoft: Create a VNet with a Site-to-Site connection using the Azure portal
Microsoft: Create a VNet with a Site-to-Site connection using PowerShell
Steve Buchanan: Azure & RRAS Site to Site VPN Setup (Azure Resource Manager)
Microsoft: What is a Network Security Group?
Microsoft: VPN Gateway FAQ
Microsoft: About VPN Gateway
Microsoft: How to install and configure Azure PowerShell
Microsoft: About VPN Devices for Site-to-Site VPN Gateway Connections (Critical article; contains IPsec specs of Azure side)

pfSense

pfSense Doco: Routing internet traffic through a site-to-site IPsec tunnel
pfSense Doco: VPN Capability IPsec

IPsec

PFS RFC
strongSwan Wiki: Security recommendations
Johannes Webber: Considerations About IPsec Pre-Shared Keys
Cisco Support: How Does NAT-T Work With IPsec (Only needed if behind NAT)

Tuesday, April 19, 2016

Installing Chef on A Raspberry Pi 2/3


Introduction

So you’ve got 1,103 Raspberry Pis that you need to manage. Two things:
  1. Why?
  2. Wanna hang out Saturday? No? You’re busy managing all your mini computers manually? I can help you with that!
RaspChef_Big

Scope

In this article we’ll cover installing and configuring the Chef version 12 client on a Raspberry Pi. This has been tested on Raspberry Pi versions 2 and 3; in theory it should work on a 1 as well, albeit  slowly.

Assumptions

  • Raspberry Pi with Rasbian, Hypriot, or similar build with connections to interwebs
  • Chef server/org you would like to point clients to
  • Chef workstation capable of bootstrapping clients; if needed see this excellent article by Digital Ocean.

Execution

The main point of this article is really the installation of Ruby, which is the foundation on which Chef is based. Because the Ruby package in the Rasbian locations is out of date (2.1 as of this writing) we need to compile our own from source. Chef 12 requires Ruby 2.0 or greater, but Rack, which is installed with Chef, requires 2.2.2.

Step 1: Install Ruby

Clearly this should be scripted for optimal efficiency, but for learning purposes we’ll do it step by step to see exactly what is going on first hand. Log onto your Raspi via SSH and execute the following:
  1. Most commands we’ll be executing require root, so let’s elevate our session:
    sudo su
    
    Where sudo = "super user do" and su ="super user"; using root privs to assume root identity.

  2. Pull the newest package lists from configured repositories to ensure we get the newest packages:
    apt-get update
    
    2016-05-07 00_03_01-_home_toby_
  3. Install pre-requisites for Ruby: gcc, make, and libssl-dev with their dependencies. 
    apt-get install gcc make libssl-dev
    
    2016-05-07 00_06_47-_home_toby_
    Note: On some distros, such as Raspbian, gcc and make are installed by default. It won't hurt to include the in the command line none the less, and including it here will cover most distros.

  4. Download the Ruby source to the /usr/src directory:
    cd /usr/src
    wget https://cache.ruby-lang.org/pub/ruby/2.2/ruby-2.2.5.tar.gz
    
    Note: 2.2.5 is the newest 2.2 version at the time of this writing, but you should make sure there isn't a newer version avialable. Check the Ruby page here.

  5. Extract the source & navigate to the directory (Make sure you update filenames/directory names for differing versions):
    tar -xvzf ruby-2.2.5.tar.gz
    cd ruby-2.2.5
    
  6. Prepare to compile with configure, omitting unnecessary components:
    ./configure --enable-shared --disable-install-doc --disable-install-rdoc --disable-install-capi
    
    2016-05-07 00_12_03-_home_toby_
    Note: This will take between 2 and 10 minutes depending on which Pi and the speed of your SD card.

  7. Compile it using make!
    make -j4 ; make install
    
    2016-05-07 15_28_19-Window
    Note: "make -j4" will multi-thread the execution, using each of the Raspi's processors. This will take between 15 and 30 minutes depending on your Pi and SD card.
hot_hot_hot

Step 2: Install Chef

Now we’ll use the gem install command to get Chef
  1. Execute gem install chef as root (sudo if not).
    gem install chef
    
    Note: This will take between 5 and 25 minutes depending on which Pi, SD card, and network connection.

    2016-05-07 16_44_05-Window
  2. Relinquish root privileges as they are no longer needed. This should only exit the root session and not the SSH session itself. If you’re logged in directly as root ignore this, but don’t do that next time!
    exit
    
  3. Test the install to ensure it worked
    chef-client --version
    
    2016-05-07 16_46_29-Window

Step 3: Configure Chef

For this step move to your Chef workstation and logon using your account that is configured to manage your organization. 
  1. Use the knife command to boostrap the newly installed client
    knife bootstrap rasdock02.truckchase.lan -N rasdock02.truckchase.lan -x {user} -P {password}
    
    Where: {user} is a user on the target platform with root privs and {password} is the password for that account.

    Note: It is normal to see errors on the first portion on the bootstrap since the Chef ARM client will not be found in the Chef repo, but the second phase should work utilizing the client we just installed.

    2016-05-07 16_50_56-Window

That’s it! For further verification you can check against the Chef server using your workstation (knife client show {node name}) or even better yet, use Chef Manage if you have it available.

2016-05-07 16_53_23-Chef Manage

References

Sunday, April 5, 2015

PowerShell Networking Cheat Sheet

Foreward

I haven't located a good cheat sheet for basic PowerShell commands to manage your Network Card/IP stack, so I thought I should make one. If you like this please share it with your mom because I'm sure she's been waiting to set a static IP address via PowerShell for some time. Also you should call her more.


My campaign against stock photography continues.

Assumptions


  • Windows 2012/Windows 8 or higher
  • Administrator access to the machine in question
  • Powershell ran as administrator on the machine in question


CMDLet List:

Most of the commands used to manipulate IP settings can be found by typing Get-Command -Module NetTCPIP

PS C:\WINDOWS\system32> Get-Command -Module NetTCPIP

CommandType     Name                                               ModuleName
-----------     ----                                               ----------
Function        Find-NetRoute                                      NetTCPIP
Function        Get-NetCompartment                                 NetTCPIP
Function        Get-NetIPAddress                                   NetTCPIP
Function        Get-NetIPConfiguration                             NetTCPIP
Function        Get-NetIPInterface                                 NetTCPIP
Function        Get-NetIPv4Protocol                                NetTCPIP
Function        Get-NetIPv6Protocol                                NetTCPIP
Function        Get-NetNeighbor                                    NetTCPIP
Function        Get-NetOffloadGlobalSetting                        NetTCPIP
Function        Get-NetPrefixPolicy                                NetTCPIP
Function        Get-NetRoute                                       NetTCPIP
Function        Get-NetTCPConnection                               NetTCPIP
Function        Get-NetTCPSetting                                  NetTCPIP
Function        Get-NetTransportFilter                             NetTCPIP
Function        Get-NetUDPEndpoint                                 NetTCPIP
Function        Get-NetUDPSetting                                  NetTCPIP
Function        New-NetIPAddress                                   NetTCPIP
Function        New-NetNeighbor                                    NetTCPIP
Function        New-NetRoute                                       NetTCPIP
Function        New-NetTransportFilter                             NetTCPIP
Function        Remove-NetIPAddress                                NetTCPIP
Function        Remove-NetNeighbor                                 NetTCPIP
Function        Remove-NetRoute                                    NetTCPIP
Function        Remove-NetTransportFilter                          NetTCPIP
Function        Set-NetIPAddress                                   NetTCPIP
Function        Set-NetIPInterface                                 NetTCPIP
Function        Set-NetIPv4Protocol                                NetTCPIP
Function        Set-NetIPv6Protocol                                NetTCPIP
Function        Set-NetNeighbor                                    NetTCPIP
Function        Set-NetOffloadGlobalSetting                        NetTCPIP
Function        Set-NetRoute                                       NetTCPIP
Function        Set-NetTCPSetting                                  NetTCPIP
Function        Set-NetUDPSetting                                  NetTCPIP
Function        Test-NetConnection                                 NetTCPIP
To find out more about any given command, type Get-Help <cmdlet name>

Common Tasks


Here are walkthroughs of some of the more common tasks you may want to perform.

List Network Adapters

Gets a list of all adapters in the machine; you'll need to know your adapter name or index # (both listed) for some of the commands below.
Get-NetAdapter

Change an Adapter Friendly Name

You may want to consider changing the friendly name of the adapter you intend to manipulate. By giving it shorter, more meaningful name you'll have an easier time going forward. Use the old name you got from the last command. Syntax is Rename-NetAdapter -Name "<Current Name>" -NewName <"New Name">
Rename-NetAdapter -Name "Local Area Connection" -NewName WiFi

Get the Current IP Address

Gets all IPv4 addresses on the machine; you'll almost always have multiple as your loopback interface (127.0.0.1) will be listed. Ignore that guy.
Get-NetIPAddress -AddressFamily ipv4
Optionally, you can specify -InterfaceAlias <friendly name> or -InterfaceIndex <index #> to limit the command to a single adapter.

Assign a Static IP Address to your Network Adapter

This command will set the one and only (overwriting what is there) IP address for the specified network adapter. You also can (and should) set the subnet mask with this command. The subnet mask is set via CIDR using the -PrefixLength; see the link for more info about CIDR, but if you're not familiar with CIDR it is likely that you want -PrefixLength 24 which translates to 255.255.255.0 meaning the first three octets are the network while the last is the host. Syntax is New-NetIPAddress -InterfaceAlias <name> -IPAddress <IP address> -PrefixLength <CIDR> -DefaultGateway <Gateway IP>  . You can substitute -InterfaceIndex <index #> for -InterfaceAlias .
New-NetIPAddress -InterfaceAlias WiFi -IPAddress 192.168.1.10 -PrefixLength 24 -DefaultGateway 192.168.1.1
Note: You will get an error if you already have a static IP address with a default gateway. To fix this problem see "Delete Existing Static IP" below and then try again.
Note2: We're not using "Set-NetIPAddress" here because it doesn't allow you to set a default gateway. BOOOO.

Set DNS Servers for your Adapter

To look up names you'll need to set DNS server(s). Syntax is Set-DNSClientServerAddress -InterfaceAlias <name> -ServerAddresses ("<IP Address 1","IP Address 2") . You can set as many DNS servers as you like. You can substitute -InterfaceIndex <index #> for -InterfaceAlias .
Set-DNSClientServerAddress –interfaceAlias WiFi –ServerAddresses (“192.168.1.5”,”192.168.1.6”)

Set a Default Gateway

It's generally easier to set the default gateway as part of the New-NetIPAddress command above, but if you want to set one separately see "Set a Static Route" below.

Delete Existing Static IP (to prep for a new)

This is a two step process; you need to delete the IP, then the gateway. No need to worry about the DNS servers here as it works to overwrite them with the command above. You will need to know the IP address you want to delete first; use get-netipaddress (above) to get it (write it down to use below if necessary). You'll then need to know the NextHop of the gateway. To get this, use the get-netroute command and write down the entry(ies) that have a nexthop of the gateway you intend to remove (see screenshot).



The syntax for these commands are
Remove-NetAddress <IPAddress> -Confirm:$False
Remove-NetRoute -NextHop <Gateway IPAddress> -Confirm:$False

Remove-NetAddress 192.168.1.10 -Confirm:$False
Remove-NetRoute -NextHop 192.168.1.1 -Confirm:$False
Note: If you have multiple routes set with that default gateway it will delete them all. If you haven't manually set routes, don't worry about it (you just have the one).

Set Your Adapter to Use DHCP

This is another two step process; first set the desired adapter IP/Gateway to DHCP then set the DNS servers to pull from DHCP as well.

The syntax for these commands are:
Set-NetIPInterface -InterfaceAlias <name> -Dhcp Enabled
Set-DNSClientServerAddress -InterfaceAlias <name> -ResetServerAddress
You can substitute -InterfaceIndex <index #> for -IterfaceAlias if you prefer.
Set-NetIPInterface -InterfaceAlias WiFi -Dhcp Enabled
Set-DNSClientServerAddress -InterfaceAlias Wifi -ResetServerAddress
Note: If you have a static gateway set you'll need to perform the second step "Remove-NetRoute" from the step above as well. 


Advanced Tasks


Here are walkthroughs of some of the more common tasks you may want to perform. My assumption here is that you know what you want to do so I won't be discussing the details of what each of these means.

Add/Delete a Static Route

Add: (use -RouteMetric to specify metric or -PolicyStore to control persistence through reboots)
New-NetRoute -DestinationPrefix 192.168.2.0/24 -InterfaceAlias WiFi -NextHop 192.168.2.1
Add default route:
New-NetRoute -DestinationPrefix 0.0.0.0/0 -InterfaceAlias WiFi -NextHop 192.168.2.1
Delete: (while this command is very specific you can be more generic; see above)
Remove-NetRoute -DestinationPrefix 192.168.2.0/24 -InterfaceAlias WiFi -NextHop 192.168.2.1

Test Network Connectivity (Ping)

Test-Connection replaces ping.exe. In addition to the ping functionality Test-Connection supports authentication (if firewall is set accordingly), multiple targets in a single command, running as a job, as well as more detailed returns.
Test-Connection myhost.mydomain.com

Assign a DNS Suffix

This is the domain under which your IP will be registered and under most circumstances will be used to append to hostname searches. Note this is per-adapter. You can substitute InterfaceIndex for InterfaceAlias if you like.
Set-DnsClient -InterfaceAlias WiFi -ConnectionSpecificSuffix mydomain.com

Assign an Additional IP Address to your NIC

If you want to add another IP (usually only applicable on a server)
New-NetIPAddress -InterfaceAlias Ethernet1 -IPAddress 192.168.1.101 -PrefixLength 24


References/More Information

TechNet: Net TCP/IP Cmdlets in Windows PowerShell
TechNet: DNS Client Cmdlets in Windows PowerShell
TechNet: Network Adapter Cmdlets

You did it, hug a puppy!

Saturday, January 3, 2015

Remotely Restart Serivices via PowerShell Without Admin Credentials

Scope


Ever wanted to allow a non-admin to remotely restart a service? No? K.

.... well if sometime you feel like it, this article is for you. We'll walk through enabling PowerShell Remoting (part of WinRM), allowing the non-administrators to connect, and allowing the specified user or group to restart the service in question.


(Install and) Enable PowerShell Remoting


On Windows server 2012, this step is already completed by default. On earlier versions of Windows you may need to install software and then enable connection. That is an article in and of itself, but fortunately it has already been published by Ian Farr. Consult that article to determine what software, if any, you need to install.

After installs:

Note: This may not be necessary on 2012 and up but will not hurt none the less.
  1. Remote to the server you wish to remotely manipulate services on
  2. Open an elevated PowerShell prompt


  3. Run the Enable-PSRemoting cmdlet


  4. For each query, type "y" and press enter.


The Enable-PSRemoting cmdlet will run a few different sub cmdlets to enable the proper services and setup initial permissions.


Set up Security Principals and Grant Permissions to use PowerShell Remoting


Caution: This grants access to connect to the PSRemote interface and attempt to issue commands. While this isn't enough to stop services, etc. by itself it should be done with caution.

There are two levels of security you will need to address with this operation: the ability to connect to the PSRemote interface/attempt to issue commands, and the permissions on the service to be restarted. It is important to consider that these two groups may not be the same; there is a lot one can do with PSRemoting outside of restarting a specific service. Thus it is likely you will want to create a larger (in scope and size) group to access PSRemoting and a smaller group to restart this specific service.

If you don't much care and just want to get going you can just make the principals the same... even just a single user. With that said:

  1. Determine what security principals you want to have access to PSRemoting and create them if necessary. I use an Active Directory group named appropriately and populated with the desired users, but you could use anything down to a single user if you like.


  2. Ensure you still have the PowerShell prompt open on your target server; if not open it as in the section above.
  3. Execute Set-PSSessionConfiguration -ShowSecurityDescriptorUI -Name Microsoft.PowerShell which will bring up the Windows security dialog box.


  4. Add your desired security principal (AD Group, Local Group, User, etc) and grant it "Full Control" access.


  5. Click "OK" to close the box and apply the permissions, then answer y + <enter> to restart the service.

Note that on 64-bit systems where you have elected to use the 32-bit version of PowerShell you will also need to execute steps 3-5 using the Set-PSSessionConfiguration -ShowSecurityDescriptorUI -Name Microsoft.PowerShell32 command as well. 

Now you can hit the PSRemote interface with the specified accounts; if you haven't figured it out yet this could be used for many things other than just restarting a service... this is why PSRemoting is so great!


Grant Service Restart Access to the Appropriate Security Principal


In my example my service will be called "MyService". Normally this is an operation that would be somewhat complex with the built-in Windows tools, so I'm going to offer up a simpler way.

  1. Download and install Carbon (more below) on your target server(s). 
  2. Per the install instructions, unblock the zip, extract, and import the module after install with Import-Module Carbon or the import-carbon.ps1 script. (Note: You may need to set-executionpolicy unrestricted to use Carbon; make sure you set-executionpolicy Restricted when you're done)


  3. Execute the Grant-ServiceControlPermission, i.e. Grant-ServiceControlPermission -ServiceName MyService -Identity MyDomain\MyGroup where MyService is the short name of the target service and MyDomain\MyGroup is the target security principal. 


Aaron Jensen has done a great job with the Carbon module and I highly recommend it to anyone operating in a DevOps space on Windows Server. That one command in point #3 could have been an article about SDDLs in and of itself.


Test and Provide Restart Commands to Target User(s)


Rather than having your target users learn the appropriate PowerShell commands, it may be easier to provide them with a script. Here's a sample one that can restart multiple services on multiple servers after executing the steps above on each.



$servers="Server1","Server2"
$services="MyService1","MyService2"

Write-Warning 'What would you like this script to do? '
Write-Warning '1. Stop Services'
Write-Warning '2. Start Services'
Write-Warning '3. Re-start Services'
Write-Warning '4. Quit'
while($ActionType.length -lt 1)
{
 $ActionType=Read-Host "Enter the number to correspond with your desired action. " 
 $ActionType=[string]$ActionType.trim()
}
 
switch ($ActionType)
{
 1 
 {
  $stop=$true
  $start=$false
 }
 2 
 {
  $stop=$false
  $start=$true
 }
 3
 {
  $stop=$true
  $start=$true
 }
 4 {Exit}
 default {throw ("Unexpected input from action type inquriy.")}
}
 
foreach ($server in $servers)
{
 if ($stop)
 {
  foreach ($service in $services)
  {
   Write-Debug "Stopping $service on $Server"
   Invoke-Command -ComputerName $server -ScriptBlock {Stop-Service -Name $args[0]} -ArgumentList $service
   if ($? -eq $false){throw "TERMINATING SCRIPT; UNABLE TO STOP SERVICE. Check permissions and ensure target machine $server is up"}
   Write-Warning "Service $service stopped on $server"
  }
 }
 start-sleep -Milliseconds 500

 if ($start)
 {
  #Start Services
  foreach ($service in $services)
  {
   Write-Debug "Starting $service on $server"
   Invoke-Command -ComputerName $server -ScriptBlock {Start-Service -Name $args[0]} -ArgumentList $service
   if ($? -eq $false){throw "TERMINATING SCRIPT; UNABLE TO START SERVICE. Check permissions and ensure target machine $server is up"}
   Write-Warning "Service $service started on $server"
  }
 }
}
Write-Warning "All actions complete!"

This script is really just a starting point; there are many different directions you could take it based on your needs. For example, you could use the get-credential cmdlet to allow the user to enter alternative credentials. To use it as-is simply replace the server(s) and service(s) names at the top of the script with those you are targeting.

... and that's it! Thanks to carbon and a couple simple commands this operation is pretty easy to achieve. This has been a big help in lower environments for letting users, developers, etc. restart services as needed without having to interrupt their support personnel. Hopefully it helps you as well.

References/Additional Information


TechNet: Enable-PSRemoting
Stack Overflow: WSMan and Basic Authorization
Stack Overflow: PowerShell Remoting Giving Access is Denied Error
TechNet: Introduction to WinRM
Ondrej Sevecek: Enabling remote WMI and PowerShell access over WinRM for non-administrators (Much of the inspiration for this article)
Carbon (PowerShell Dev-Ops module by Aaron Jensen)

Monday, December 22, 2014

Mumble + Splunk

Who's on the Server? Splunk it!


Mumble is a great VOIP solution for latency sensitive situations; I run several servers for different applications, and monitoring those has always been a bit of a challenge since it only generates text logs and doesn't do any historic usage tracking. Fortunately we've got a tool to solve that problem in both real time and historic situations: Splunk.

In this article we'll walk through a simple example of data ingestion, parsing, and dashboard creation in Splunk. When we're finished we'll be able to tell who is online at any given time and how popular your server has been in the recent past.


Step 1: Data Ingestion

This section assumes the following:

  • Mumble (or another app if using this as a general guideline) installed and logging to a consistent location.
  • Logging is not set to rename logfiles as part of the log rollover process; doing so would complicate our source setup. (note this is achievable) 
  • The Splunk forwarder is installed on the target machine(s) and is already configured to output to the indexer(s). (install location referred to henceforth as $SPLUNK_HOME) What's that? You've got thousands of boxes and no time to install? We can fix that!

I'm using Windows boxes in this example, but there's no reason all of this won't work on Linux as well with some minor tweaks.


To get the data into Splunk we'll first need to identify where the data is and how to ensure it gets into the target indexer(s). In my case I'll be targeting the following (4 instances) data sources from one server:
  • C:\Apps\Murmur1LowQual\murmur.log
  • C:\Apps\Murmur2LowQual\murmur.log
  • C:\Apps\Murmur1HighQual\murmur.log
  • C:\Apps\Murmur2HighQual\murmur.log
  • Performance data: Network interface, CPU Usage

If the application to be monitored is distributed densely and consistently in your enterprise you will want to make the input changes in an "app" for deployment assuming you use "Forwarder Management". In this case it is a one-off configuration so I will be specifying these inputs manually.


For our one-off case, open the $SPLUNK_HOME/etc/system/local/inputs.conf file for editing (for background, make sure you understand "About Configuration Files") and add the following entries:

    [monitor://C:\Apps\MurmurLowQual\murmur.log]
    disabled = false
    sourcetype = murmur
    
    [monitor://C:\Apps\MurmurHighQual\murmur.log]
    disabled = false
    sourcetype = murmur

    [monitor://C:\Apps\Murmur2HighQual\murmur.log]
    disabled = false
    sourcetype = murmur
    
    [monitor://C:\Apps\Murmur2LowQual\murmur.log]
    disabled = false
    sourcetype = murmur
    
    [perfmon://CPU Load]
    counters = % Processor Time
    instances = _Total
    interval = 20
    object = Processor
    
    [perfmon://Network Interface]
    counters = Bytes Received/sec;Bytes Sent/sec
    instances = *
    interval = 15
    object = Network Interface
    
Note these entries are Windows specific; the paths and the perfmon data would need be changed on a Linux host.

Where:
  • monitor://<logfilePath> specifies the path to the logfile to be ingested (Murmur is Mumble's server component)
  • sourcetype = murmur specifies the sourcetype to store this information under in Splunk. This is critical to properly sorting data.
  • perfmon://<counter> , counters = <counter1>;<counter2> , instances = *, and object = <object in question> specify the performance information we want to bring in. This line is Windows specific and needs to be changed for *nix.
  • interval = <interval in seconds> is the interval to bring that performance data in at. Lower interval = more granular performance information but more data to store.  

This may or may not be sufficient in your case. For more information, see "Edit inputs.conf" on the Splunk site


Step 2: Pre-Format Data


Now we need to take the steps necessary to easily create searches against this data. At a minimum any Splunk Admin should take the time to do proper field extractions from your new data source. Creating custom fields ensure there will be meaningful information for users to search on. This part of Splunk operation is often the make-or-break point in many organizations, as proper field extraction can be the difference between an end user figuring out how to create meaningful searches vs. giving up and going to the original log files.

In this example I'll use one regex to extract three fields from the Murmur logfile under the "search" app context. To set up this field extraction:

  1. Log in to your Splunk web interface
  2. Change to the "search" app context
  3. Navigate to "Settings -> Fields


  4. Click on "Field extractions"


  5. Click "New"
  6. Leave the "destination app"as "search" (We'll work in search for now but you could make this into an app)
  7. Name it per your enterprise standard. In my case that is <AppName_fields extracted>, so "Murmur_sessionID_UserName_AdminStatus".
  8. Change "Apply to" to "sourcetype" named "murmur"
  9. Keep "Type" to "Inline"
  10. For the Extraction/Transform insert your regex statement. To extract the Session ID (as session_ID), UserName (as u_name), and AdminStatus (as userIsAdmin) from a Murmur logfile use this: "=> <(?<session_ID>[^:]+):(?<u_name>[^\(]+)\((?<userIsAdmin>[^\)]+)"
  11. Click "Save"


  12. To make this usable for everyone, click "Permissions" under "Sharing" to the right of the name on the "Field extractions" screen. 


  13. Select "This app only(search)" (change later if you use a different app) and check "Read" under "Everyone" Click "Save". 

We have now configured field extractions for Mumble. While there is much more one should do (data sizing, business use cases, etc.) to on-board an application this will be enough for now to develop a basic dashboard.


Step 3: Searches and a Dashboard!


Now we'll make use of this data. This is the beauty of Splunk; you can format almost all the data in a meaningful way, and even create new data points inferred from the other available data. To illustrate what I mean, one of these searches will determine who is online right now from the logon/logoff information in the log file. Let's tackle that and a few others:


Search 1: Who is online right now?

sourcetype=murmur |transaction session_ID,u_name maxspan=24h|search authenticated NOT closed|eval AdminRange = case(userIsAdmin < 0, "False", userIsAdmin >= 1, "True")|table u_name,session_ID,AdminRange,_time | rename u_name as "User Name", session_ID as "Session ID", AdminRange as "User is Administrator"

Will generate a table like this:


Where:

sourcetype=murmur : only search the appropriate sourcetype. Note you may want to further limit by host, index, or other locations.

transaction session_ID,u_name maxspan=24h : uses the powerful "transaction" command to string events into a transaction by the listed fields. Note: This command can be computationally expensive so be careful when using it!

search authenticated NOT closed : look for sessions illustrating a user connecting but not yet closed Note: again, since we're using a "NOT" clause this search could be expensive depending on your data volume from the previous parts of the search. Fortunately we should be very limited data wise in this search string by now.

eval AdminRange = case(userIsAdmin < 0, "False", userIsAdmin >= 1, "True") : use the eval command to determine admin status as Boolean

table u_name,session_ID,AdminRange,_time : Render as a table

rename u_name as "User Name", session_ID as "Session ID", AdminRange as "User is Administrator" : Rename table fields to be more meaningful to the end user

Since we're limiting a session span to 24hours in the transaction search, you may as well reduce the time for this search to 24 hours as well. For a slight performance tweak on searches that display a table or graphic of specific fields you can change the search mode from "smart" to "fast".  Now let's add this to a new dashboard:
  1. After the results come up, click "Save As" the "Dashboard Panel"  on the upper right hand side.
  2. Select "New"
  3. Insert an appropriate title, i.e. "Mumble Statistics" following enterprise standards if applicable. Generally it is OK to let the Dashboard ID auto-populate with this name.
  4. Write a description if desired and change the Dashboard Permissions to "Shared in App" so we can share this information with others. 
  5. Type an appropriate Panel Tile such as "Users Online Now!". Accept the defaults for the remaining and click "Save".
For fun you can make a little tachometer displaying this information by opening the saved search and saving it to the existing dashboard under a different name, then changing the display to "Radial Gauge". End users generally like these "at a glance" graphics for important information, especially at the top of a dashboard.

Search 2: How many people have logged on per day for the last 30 days?

sourcetype=murmur u_name=* authenticated |timechart count

Will generate:


Where:

sourcetype=murmur u_name=* authenticated
: only search the appropriate sourcetype for events where u_name is populated and the word "authenticated" is present.

timechart count : charts all results using the very easy timechart command. Make sure you limit the search scope (below).

Set the search scope for 30 days, execute, then save to our existing Mumble Statistics dashboard as a panel named "Logins Per Day - Last 30 Days". I prefer this as a bar chart, which you can select at the dashboard level.

Search 3: How much bandwidth did the server use in the last day?

host=hostname_here sourcetype="Perfmon:Network Interface"|eval DataSrc=case((instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Received/sec"),"ETH0bitsSecIN",(instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Sent/sec"),"ETH0bitsSecOUT")|eval Kbits_Sec=Value*.008| timechart span=5m avg(Kbits_Sec) by DataSrc| rename LANbitsSecIN as "LAN Kbits/sec IN" LANbitsSecOUT as "LAN Kbits/sec OUT" WANbitsSecIN as "WAN Kbits/sec IN" WANbitsSecOUT as "WAN Kbits/sec OUT"

Will generate:


Where:

host=hostname_here sourcetype="Perfmon:Network Interface"
:  Specify events to search. You will need to change the hostname and potentially the sourcetype depending on your host platform, etc.

eval DataSrc=case((instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Received/sec"),"ETH0bytesSecIN",(instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Sent/sec"),"ETH0bytesSecOUT") : Here is where we use eval to map interfaces to directions. If you have multiple interfaces you'll need to address them on this line and also note you will need to change the "names" of the adapters to match your data.

eval Kbits_Sec=Value*.008 : Convert Bytes/Sec to Kbits/Sec

timechart span=5m avg(Kbits_Sec) by DataSrc : Chart the data by interface direction on a 5 minute average. Note if you made a longer-term chart you'll need to change the average to calculate on a wider basis to keep your datapoints low enough to chart.

rename ETH0bytesSecIN as "ETH0 Kbits/sec IN" ETH0bytesSecOUT as "ETH0 Kbits/sec OUT" : Rename the datapoints relative to our eval statement above.

Set the search scope for 24 hours, execute, then save to our existing Mumble Statistics dashboard as a panel named "Network Traffic 5m Average Last 24 Hours"

How many users on the sever? Plenty.

Those three should get you started; clearly there is substantially more one could do with all the data available to us. After you decide what else to add make sure you go through your dashboard and reposition/edit each panel as necessary. Keep in mind that you can rename x/y axis as well as change the way data is rendered. Hopefully this tutorial has demonstrated that even with a simple application you can use a tool like Splunk to make the day-to-day use and impact on an organization much more transparent. This methodology could easily be turned into an app and distributed throughout your enterprise and/or the Splunk community.