Mumble + Splunk

Who's on the Server? Splunk it!

Mumble is a great VOIP solution for latency sensitive situations; I run several servers for different applications, and monitoring those has always been a bit of a challenge since it only generates text logs and doesn't do any historic usage tracking. Fortunately we've got a tool to solve that problem in both real time and historic situations: Splunk.

In this article we'll walk through a simple example of data ingestion, parsing, and dashboard creation in Splunk. When we're finished we'll be able to tell who is online at any given time and how popular your server has been in the recent past.

Step 1: Data Ingestion

This section assumes the following:

• Mumble (or another app if using this as a general guideline) installed and logging to a consistent location.
• Logging is not set to rename logfiles as part of the log rollover process; doing so would complicate our source setup. (note this is achievable) 
• The Splunk forwarder is installed on the target machine(s) and is already configured to output to the indexer(s). (install location referred to henceforth as $SPLUNK_HOME) What's that? You've got thousands of boxes and no time to install? We can fix that!

I'm using Windows boxes in this example, but there's no reason all of this won't work on Linux as well with some minor tweaks.

To get the data into Splunk we'll first need to identify where the data is and how to ensure it gets into the target indexer(s). In my case I'll be targeting the following (4 instances) data sources from one server:

• C:\Apps\Murmur1LowQual\murmur.log
• C:\Apps\Murmur2LowQual\murmur.log
• C:\Apps\Murmur1HighQual\murmur.log
• C:\Apps\Murmur2HighQual\murmur.log
• Performance data: Network interface, CPU Usage

If the application to be monitored is distributed densely and consistently in your enterprise you will want to make the input changes in an "app" for deployment assuming you use "Forwarder Management". In this case it is a one-off configuration so I will be specifying these inputs manually.

For our one-off case, open the $SPLUNK_HOME/etc/system/local/inputs.conf file for editing (for background, make sure you understand "About Configuration Files") and add the following entries:

    [monitor://C:\Apps\MurmurLowQual\murmur.log]
    disabled = false
    sourcetype = murmur
    
    [monitor://C:\Apps\MurmurHighQual\murmur.log]
    disabled = false
    sourcetype = murmur

    [monitor://C:\Apps\Murmur2HighQual\murmur.log]
    disabled = false
    sourcetype = murmur
    
    [monitor://C:\Apps\Murmur2LowQual\murmur.log]
    disabled = false
    sourcetype = murmur
    
    [perfmon://CPU Load]
    counters = % Processor Time
    instances = _Total
    interval = 20
    object = Processor
    
    [perfmon://Network Interface]
    counters = Bytes Received/sec;Bytes Sent/sec
    instances = *
    interval = 15
    object = Network Interface
    

Note these entries are Windows specific; the paths and the perfmon data would need be changed on a Linux host.

Where:

• monitor://<logfilePath> specifies the path to the logfile to be ingested (Murmur is Mumble's server component)
• sourcetype = murmur specifies the sourcetype to store this information under in Splunk. This is critical to properly sorting data.
• perfmon://<counter> , counters = <counter1>;<counter2> , instances = *, and object = <object in question> specify the performance information we want to bring in. This line is Windows specific and needs to be changed for *nix.
• interval = <interval in seconds> is the interval to bring that performance data in at. Lower interval = more granular performance information but more data to store.  

This may or may not be sufficient in your case. For more information, see "Edit inputs.conf" on the Splunk site. 

Step 2: Pre-Format Data

Now we need to take the steps necessary to easily create searches against this data. At a minimum any Splunk Admin should take the time to do proper field extractions from your new data source. Creating custom fields ensure there will be meaningful information for users to search on. This part of Splunk operation is often the make-or-break point in many organizations, as proper field extraction can be the difference between an end user figuring out how to create meaningful searches vs. giving up and going to the original log files.

In this example I'll use one regex to extract three fields from the Murmur logfile under the "search" app context. To set up this field extraction:

1. Log in to your Splunk web interface
2. Change to the "search" app context
3. Navigate to "Settings -> Fields" 





4. Click on "Field extractions"





5. Click "New"
6. Leave the "destination app"as "search" (We'll work in search for now but you could make this into an app)
7. Name it per your enterprise standard. In my case that is <AppName_fields extracted>, so "Murmur_sessionID_UserName_AdminStatus".
8. Change "Apply to" to "sourcetype" named "murmur"
9. Keep "Type" to "Inline"
10. For the Extraction/Transform insert your regex statement. To extract the Session ID (as session_ID), UserName (as u_name), and AdminStatus (as userIsAdmin) from a Murmur logfile use this: "=> <(?<session_ID>[^:]+):(?<u_name>[^\(]+)\((?<userIsAdmin>[^\)]+)"
11. Click "Save"





12. To make this usable for everyone, click "Permissions" under "Sharing" to the right of the name on the "Field extractions" screen. 





13. Select "This app only(search)" (change later if you use a different app) and check "Read" under "Everyone" Click "Save". 

We have now configured field extractions for Mumble. While there is much more one should do (data sizing, business use cases, etc.) to on-board an application this will be enough for now to develop a basic dashboard.

Step 3: Searches and a Dashboard!

Now we'll make use of this data. This is the beauty of Splunk; you can format almost all the data in a meaningful way, and even create new data points inferred from the other available data. To illustrate what I mean, one of these searches will determine who is online right now from the logon/logoff information in the log file. Let's tackle that and a few others:

Search 1: Who is online right now?

sourcetype=murmur |transaction session_ID,u_name maxspan=24h|search authenticated NOT closed|eval AdminRange = case(userIsAdmin < 0, "False", userIsAdmin >= 1, "True")|table u_name,session_ID,AdminRange,_time | rename u_name as "User Name", session_ID as "Session ID", AdminRange as "User is Administrator"

Will generate a table like this:

Where:

sourcetype=murmur : only search the appropriate sourcetype. Note you may want to further limit by host, index, or other locations.

transaction session_ID,u_name maxspan=24h : uses the powerful "transaction" command to string events into a transaction by the listed fields. Note: This command can be computationally expensive so be careful when using it!

search authenticated NOT closed : look for sessions illustrating a user connecting but not yet closed Note: again, since we're using a "NOT" clause this search could be expensive depending on your data volume from the previous parts of the search. Fortunately we should be very limited data wise in this search string by now.

eval AdminRange = case(userIsAdmin < 0, "False", userIsAdmin >= 1, "True") : use the eval command to determine admin status as Boolean

table u_name,session_ID,AdminRange,_time : Render as a table

rename u_name as "User Name", session_ID as "Session ID", AdminRange as "User is Administrator" : Rename table fields to be more meaningful to the end user

Since we're limiting a session span to 24hours in the transaction search, you may as well reduce the time for this search to 24 hours as well. For a slight performance tweak on searches that display a table or graphic of specific fields you can change the search mode from "smart" to "fast".  Now let's add this to a new dashboard:

1. After the results come up, click "Save As" the "Dashboard Panel"  on the upper right hand side.
2. Select "New"
3. Insert an appropriate title, i.e. "Mumble Statistics" following enterprise standards if applicable. Generally it is OK to let the Dashboard ID auto-populate with this name.
4. Write a description if desired and change the Dashboard Permissions to "Shared in App" so we can share this information with others. 
5. Type an appropriate Panel Tile such as "Users Online Now!". Accept the defaults for the remaining and click "Save".

For fun you can make a little tachometer displaying this information by opening the saved search and saving it to the existing dashboard under a different name, then changing the display to "Radial Gauge". End users generally like these "at a glance" graphics for important information, especially at the top of a dashboard.

Search 2: How many people have logged on per day for the last 30 days?

sourcetype=murmur u_name=* authenticated |timechart count

Will generate:

Where:

sourcetype=murmur u_name=* authenticated : only search the appropriate sourcetype for events where u_name is populated and the word "authenticated" is present.

timechart count : charts all results using the very easy timechart command. Make sure you limit the search scope (below).

Set the search scope for 30 days, execute, then save to our existing Mumble Statistics dashboard as a panel named "Logins Per Day - Last 30 Days". I prefer this as a bar chart, which you can select at the dashboard level.

Search 3: How much bandwidth did the server use in the last day?

host=hostname_here sourcetype="Perfmon:Network Interface"|eval DataSrc=case((instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Received/sec"),"ETH0bitsSecIN",(instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Sent/sec"),"ETH0bitsSecOUT")|eval Kbits_Sec=Value*.008| timechart span=5m avg(Kbits_Sec) by DataSrc| rename LANbitsSecIN as "LAN Kbits/sec IN" LANbitsSecOUT as "LAN Kbits/sec OUT" WANbitsSecIN as "WAN Kbits/sec IN" WANbitsSecOUT as "WAN Kbits/sec OUT"

Will generate:

Where:

host=hostname_here sourcetype="Perfmon:Network Interface" :  Specify events to search. You will need to change the hostname and potentially the sourcetype depending on your host platform, etc.

eval DataSrc=case((instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Received/sec"),"ETH0bytesSecIN",(instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Sent/sec"),"ETH0bytesSecOUT") : Here is where we use eval to map interfaces to directions. If you have multiple interfaces you'll need to address them on this line and also note you will need to change the "names" of the adapters to match your data.

eval Kbits_Sec=Value*.008 : Convert Bytes/Sec to Kbits/Sec

timechart span=5m avg(Kbits_Sec) by DataSrc : Chart the data by interface direction on a 5 minute average. Note if you made a longer-term chart you'll need to change the average to calculate on a wider basis to keep your datapoints low enough to chart.

rename ETH0bytesSecIN as "ETH0 Kbits/sec IN" ETH0bytesSecOUT as "ETH0 Kbits/sec OUT" : Rename the datapoints relative to our eval statement above.

Set the search scope for 24 hours, execute, then save to our existing Mumble Statistics dashboard as a panel named "Network Traffic 5m Average Last 24 Hours"

How many users on the sever? Plenty.

Those three should get you started; clearly there is substantially more one could do with all the data available to us. After you decide what else to add make sure you go through your dashboard and reposition/edit each panel as necessary. Keep in mind that you can rename x/y axis as well as change the way data is rendered. Hopefully this tutorial has demonstrated that even with a simple application you can use a tool like Splunk to make the day-to-day use and impact on an organization much more transparent. This methodology could easily be turned into an app and distributed throughout your enterprise and/or the Splunk community.

Monitoring Windows: Granular Service Rights in an Enterprise Environment

Foreword

I'm a big fan of Splunk and similar tools for network and systems monitoring. This new generation of machine data analysis brings a myriad of product and monitoring opportunities. These systems work by indexing large swaths of machine logging (logs, performance information, etc.) and providing an incredibly effective interface for reporting of that information.To achieve this, however, you need to provide very broad access to target systems.

If for any reason you can't use Universal Forwarders on Windows platforms, data collection can be difficult. While Splunk provides some great documentation, it suffers from the same access (and granting) problem as other platforms needing similar access. Generally this problem is solved in one of two ways:

• Grant local administrator access on every machine to be monitored to a single service account. Clearly this won't fly in organizations concerned with granting the least amount of access needed for their application portfolio. A single service account/group that has administrator everywhere is a significant security risk. 
• Use Group Policy to grant rights, ACLs, etc. The problem with this approach is that some things cannot be addressed by GPO (WMI) and if granting user rights often you cannot be granular enough since each right defined must list all accounts/groups that get that right. Across an enterprise that is a difficult and potentially costly proposition. The main problem with setting user rights via Group Policy is that the rights are not additive; they overwrite all entries including required system or other application entries. This makes central management of thousands of machines impractical. Say you need to define "act as part of the operating system" for your monitoring software; you would then need to centrally define every account/group that has that right and any machine that needs to define that right in a different way would need to obtain an override GPO for the machines in question. While this is possible, many very large organizations find it difficult to execute.

Neither one of these solutions is desirable in a medium to large sized enterprise due to security requirements and management complexities. To address this issue in a more granular way than the two options listed above, we can use scripts to automate local configuration where necessary.

I'll be using Powershell scripts to perform the following tasks:

• Add Active Directory groups/users to local groups
• Grant WMI rights to Active Directory groups/users
• Grant local user rights to  Active Directory groups/users

Assuming the affected security principals are not set elsewhere, we can set these items with scripts that can be triggered via group policy. I'll be using the GPO startup script functionality, but you can feel free to use any other trigger mechanism you prefer (logon script, etc.). Note that this solution will work for products similar to Splunk that require similar rights.

Execution

First things first, we need to establish what we'll be doing and principals to which we want to grant access.

Assumptions

• Splunk indexer (or your other monitoring tool) installed and ready to go
• You have administrative rights on all machines in question and Active Directory rights sufficient to create the service account/group(s)
• You would rather use the Splunk Indexer to collect information rather than install the Splunk Universal Installer on your target machines (Good option as well). 
• Powershell 2.0 or newer & Windows 2003 or newer. This could be done with batch/executables or VBScript but I won't be covering that.

Service Account/Group Setup

Per the Splunk instructions, create a service account to run Splunk. This will require administrative access on the machine running Splunk, but nothing else (yet). It would be best to use a managed service account but if you cannot a standard account will be fine.

Splunk instructions deem that an Active Directory group need be created to contain the service account. This isn't really necessary, but there are potential advantages so we'll stick to it.

After creating/configuring that service account, create the service account group and make your service account a member. For the purposes of our demonstration I've named this group "Splunk Accounts"

You should monitor this group to ensure no other accounts are added since doing so would grant those accounts unwanted rights.

Now let's configure services. The service account will need to be local administrator on machines running the Splunk software (splunkd, splunkweb) directly (more on this another time). After granting local admin, stop the Splunkd and splunkweb services, change the logon to your new service account and start them back up.

Now let's tackle the interesting part: granting granular rights to the service account group.

Objective 

Our objective is to grant the appropriate rights to the service account group on every machine in our enterprise that we wish to monitor. Per the documentation from Splunk, the minimum rights to collect Event Log and WMI data remotely on a Windows platform are: 

Misc

• WMI rights under root/cimv2
• DCOM launch permissions

User Rights

• Access this Computer from the Network
• Act as part of the operating system (I'd like to confirm this one; not sure why)
• Log on as a batch job
• Log on as a service
• Profile System Performance
• Replace a process level token (not sure about this one either)

Local Group Membership

• Distributed COM Users
• Event Log Readers
• Performance Log Users

I'm sure you can see the why the official documentation prefers local admin rights. :)  Fortunately, we've got automation to do our heavy lifting.

Scripts

Let's examine the scripts we'll be using. I've prepared four scripts for this task: 

• Powershell script to grant WMI and DCOM rights (Grant-WMIACL.ps1)
• Powershell script to grant User Rights (Grant-Rights.ps1)
• Powershell script to add local group membership and tie the others together (Setup-Splunkuser.ps1)
• Batch file to launch powershell script correctly on all platforms (SetupSplunkuser.bat)

Grant-WMIACL.ps1

As discussed, this script will grant the appropriate WMI and DCOM access. This is based heavily on the work of Steve Lee and Karl Mitschke. Note that I've left options in the script and commented out to change the level of access given to the WMI objects, including the ability to toggle inheritance. I've left these in to help you decipher the SDDL structure. That said, the line that is not commented out in this will work fine for our application. This script could be run remotely (hence the $strComputer="localhost" line) but we'll be using it locally (more on that below).

Grant-WMIACL.ps1 (Click to Expand)  + 

if ($args[0] -eq $null)
{
 Write-Warning "No account or group name provided as an argument. Re-launch with the principal to grant rights to as the first argument."
 Throw "I don't know what to do, please specify username, i.e. mydomain\myuser"
}
$Uname=$args[0]

Write-Warning "Granting WMI and DCOM access to $Uname..."

#This function gets a SID from an account name
function get-sid
{
 Param ($DSIdentity)
 $ID = new-object System.Security.Principal.NTAccount($DSIdentity)
 return $ID.Translate( [System.Security.Principal.SecurityIdentifier] ).toString()
}
$sid = get-sid $Uname

#Enable Account, Remote Enable
#maps to "CC"  SDDL_CREATE_CHILD  ADS_RIGHT_DS_CREATE_CHILD , "WP"  SDDL_WRITE_PROPERTY  ADS_RIGHT_DS_WRITE_PROP 
#$SDDL = "A;;CCWP;;;$sid"

#Provider Write(RP), Enable Account, Remote Enable
#maps to "RP"  SDDL_READ_PROPERTY  ADS_RIGHT_DS_READ_PROP , "CC"  SDDL_CREATE_CHILD  ADS_RIGHT_DS_CREATE_CHILD , "WP"  SDDL_WRITE_PROPERTY  ADS_RIGHT_DS_WRITE_PROP 
#$SDDL = "A;;CCWPRP;;;$sid"

#Execute Methods(DC), Provider Write(RP), Enable Account, Remote Enable
#maps to "DC"  SDDL_DELETE_CHILD  ADS_RIGHT_DS_DELETE_CHILD , "RP"  SDDL_READ_PROPERTY  ADS_RIGHT_DS_READ_PROP , "CC"  SDDL_CREATE_CHILD  ADS_RIGHT_DS_CREATE_CHILD , "WP"  SDDL_WRITE_PROPERTY  ADS_RIGHT_DS_WRITE_PROP 
#$SDDL = "A;;CCWPRPDC;;;$sid"

#Execute Methods(DC), Provider Write(RP), Enable Account, Remote Enable WITH INHERITENCE!! (note CI is in the header) 
#maps to "CI"  SDDL_CONTAINER_INHERIT  CONTAINER_INHERIT_ACE , "DC"  SDDL_DELETE_CHILD  ADS_RIGHT_DS_DELETE_CHILD , "RP"  SDDL_READ_PROPERTY  ADS_RIGHT_DS_READ_PROP , "CC"  SDDL_CREATE_CHILD  ADS_RIGHT_DS_CREATE_CHILD , "WP"  SDDL_WRITE_PROPERTY  ADS_RIGHT_DS_WRITE_PROP 
$SDDL = "A;CI;CCWPRPDC;;;$sid"

#Local Launch and Remote Activation
$DCOMSDDL = "A;;CCDCRP;;;$sid"

#Local Launch, Local Activation, Remote Launch, remote Activation
#$DCOMSDDL = “A;;CCDCLCSWRP;;;$sid”

#$computers = Get-Content "computers.txt"
#foreach ($strcomputer in $computers)
#{
$strComputer="localhost"
$Reg = [WMIClass]"\\$strComputer\root\default:StdRegProv"
$DCOM = $Reg.GetBinaryValue(2147483650,"software\microsoft\ole","MachineLaunchRestriction").uValue
$security = Get-WmiObject -ComputerName $strcomputer -Namespace root/cimv2 -Class __SystemSecurity
$converter = new-object system.management.ManagementClass Win32_SecurityDescriptorHelper
$binarySD = @($null)
$result = $security.PsBase.InvokeMethod("GetSD",$binarySD)
$outsddl = $converter.BinarySDToSDDL($binarySD[0])
$outDCOMSDDL = $converter.BinarySDToSDDL($DCOM)
$newSDDL = $outsddl.SDDL += "(" + $SDDL + ")"
$newDCOMSDDL = $outDCOMSDDL.SDDL += "(" + $DCOMSDDL + ")"
$WMIbinarySD = $converter.SDDLToBinarySD($newSDDL)
$WMIconvertedPermissions = ,$WMIbinarySD.BinarySD
$DCOMbinarySD = $converter.SDDLToBinarySD($newDCOMSDDL)
$DCOMconvertedPermissions = ,$DCOMbinarySD.BinarySD
#Set WMI Permissions
$result = $security.PsBase.InvokeMethod("SetSD",$WMIconvertedPermissions)
#Set DCOM Permissions
$result = $Reg.SetBinaryValue(2147483650,"software\microsoft\ole","MachineLaunchRestriction", $DCOMbinarySD.binarySD)

She's a beaut eh? Now for Grant-Rights.ps1:

This one is huge because we're using full .NET code in the beginning. That code is a combination of snippets from here, here, and here. Note the comments displaying the short names for some of the rights we can assign which I've left in to facilitate applying this approach to systems other than Spunk. More can be found here.

Grant-Rights.ps1 (Click to Expand)  + 

Add-Type @'
using System;
using System.Collections.Generic;
using System.Text;

namespace MyLsaWrapper
{
    using System.Runtime.InteropServices;
    using System.Security;
    using System.Management;
    using System.Runtime.CompilerServices;
    using System.ComponentModel;

    using LSA_HANDLE = IntPtr;

    [StructLayout(LayoutKind.Sequential)]
    struct LSA_OBJECT_ATTRIBUTES
    {
        internal int Length;
        internal IntPtr RootDirectory;
        internal IntPtr ObjectName;
        internal int Attributes;
        internal IntPtr SecurityDescriptor;
        internal IntPtr SecurityQualityOfService;
    }
    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
    struct LSA_UNICODE_STRING
    {
        internal ushort Length;
        internal ushort MaximumLength;
        [MarshalAs(UnmanagedType.LPWStr)]
        internal string Buffer;
    }
    sealed class Win32Sec
    {
        [DllImport("advapi32", CharSet = CharSet.Unicode, SetLastError = true),
        SuppressUnmanagedCodeSecurityAttribute]
        internal static extern uint LsaOpenPolicy(
        LSA_UNICODE_STRING[] SystemName,
        ref LSA_OBJECT_ATTRIBUTES ObjectAttributes,
        int AccessMask,
        out IntPtr PolicyHandle
        );

        [DllImport("advapi32", CharSet = CharSet.Unicode, SetLastError = true),
        SuppressUnmanagedCodeSecurityAttribute]
        internal static extern uint LsaAddAccountRights(
        LSA_HANDLE PolicyHandle,
        IntPtr pSID,
        LSA_UNICODE_STRING[] UserRights,
        int CountOfRights
        );

        [DllImport("advapi32", CharSet = CharSet.Unicode, SetLastError = true),
        SuppressUnmanagedCodeSecurityAttribute]
        internal static extern int LsaLookupNames2(
        LSA_HANDLE PolicyHandle,
        uint Flags,
        uint Count,
        LSA_UNICODE_STRING[] Names,
        ref IntPtr ReferencedDomains,
        ref IntPtr Sids
        );

        [DllImport("advapi32")]
        internal static extern int LsaNtStatusToWinError(int NTSTATUS);

        [DllImport("advapi32")]
        internal static extern int LsaClose(IntPtr PolicyHandle);

        [DllImport("advapi32")]
        internal static extern int LsaFreeMemory(IntPtr Buffer);

    }
    /// 

    /// This class is used to grant "Log on as a service", "Log on as a batchjob", "Log on localy" etc.
    /// to a user.
    /// 
    public sealed class LsaWrapper : IDisposable
    {
        [StructLayout(LayoutKind.Sequential)]
        struct LSA_TRUST_INFORMATION
        {
            internal LSA_UNICODE_STRING Name;
            internal IntPtr Sid;
        }
        [StructLayout(LayoutKind.Sequential)]
        struct LSA_TRANSLATED_SID2
        {
            internal SidNameUse Use;
            internal IntPtr Sid;
            internal int DomainIndex;
            uint Flags;
        }

        [StructLayout(LayoutKind.Sequential)]
        struct LSA_REFERENCED_DOMAIN_LIST
        {
            internal uint Entries;
            internal LSA_TRUST_INFORMATION Domains;
        }

        enum SidNameUse : int
        {
            User = 1,
            Group = 2,
            Domain = 3,
            Alias = 4,
            KnownGroup = 5,
            DeletedAccount = 6,
            Invalid = 7,
            Unknown = 8,
            Computer = 9
        }

        enum Access : int
        {
            POLICY_READ = 0x20006,
            POLICY_ALL_ACCESS = 0x00F0FFF,
            POLICY_EXECUTE = 0X20801,
            POLICY_WRITE = 0X207F8
        }
        const uint STATUS_ACCESS_DENIED = 0xc0000022;
        const uint STATUS_INSUFFICIENT_RESOURCES = 0xc000009a;
        const uint STATUS_NO_MEMORY = 0xc0000017;

        IntPtr lsaHandle;

        public LsaWrapper()
            : this(null)
        { }
        // // local system if systemName is null
        public LsaWrapper(string systemName)
        {
            LSA_OBJECT_ATTRIBUTES lsaAttr;
            lsaAttr.RootDirectory = IntPtr.Zero;
            lsaAttr.ObjectName = IntPtr.Zero;
            lsaAttr.Attributes = 0;
            lsaAttr.SecurityDescriptor = IntPtr.Zero;
            lsaAttr.SecurityQualityOfService = IntPtr.Zero;
            lsaAttr.Length = Marshal.SizeOf(typeof(LSA_OBJECT_ATTRIBUTES));
            lsaHandle = IntPtr.Zero;
            LSA_UNICODE_STRING[] system = null;
            if (systemName != null)
            {
                system = new LSA_UNICODE_STRING[1];
                system[0] = InitLsaString(systemName);
            }

            uint ret = Win32Sec.LsaOpenPolicy(system, ref lsaAttr,
            (int)Access.POLICY_ALL_ACCESS, out lsaHandle);
            if (ret == 0)
                return;
            if (ret == STATUS_ACCESS_DENIED)
            {
                throw new UnauthorizedAccessException();
            }
            if ((ret == STATUS_INSUFFICIENT_RESOURCES) || (ret == STATUS_NO_MEMORY))
            {
                throw new OutOfMemoryException();
            }
            throw new Win32Exception(Win32Sec.LsaNtStatusToWinError((int)ret));
        }

        public void AddPrivileges(string account, string privilege)
        {
            LSA_UNICODE_STRING[] names = new LSA_UNICODE_STRING[1];
            LSA_TRANSLATED_SID2 lts;
            IntPtr tsids = IntPtr.Zero;
            IntPtr tdom = IntPtr.Zero;
            names[0] = InitLsaString(account);
            lts.Sid = IntPtr.Zero;
            Console.WriteLine("String account: {0}", names[0].Length);
            int ret1 = Win32Sec.LsaLookupNames2(lsaHandle, 0, 1, names, ref tdom, ref tsids);
            if (ret1 != 0)
                throw new Win32Exception(Win32Sec.LsaNtStatusToWinError(ret1));
            lts = (LSA_TRANSLATED_SID2)Marshal.PtrToStructure(tsids, typeof(LSA_TRANSLATED_SID2));
            IntPtr pSid = lts.Sid;
            //IntPtr pSid = GetSIDInformation(account);
            LSA_UNICODE_STRING[] privileges = new LSA_UNICODE_STRING[1];
            privileges[0] = InitLsaString(privilege);
            uint ret = Win32Sec.LsaAddAccountRights(lsaHandle, pSid, privileges, 1);
            Win32Sec.LsaFreeMemory(tsids);
            Win32Sec.LsaFreeMemory(tdom);
            if (ret == 0)
                return;
            if (ret == STATUS_ACCESS_DENIED)
            {
                throw new UnauthorizedAccessException();
            }
            if ((ret == STATUS_INSUFFICIENT_RESOURCES) || (ret == STATUS_NO_MEMORY))
            {
                throw new OutOfMemoryException();
            }
            throw new Win32Exception(Win32Sec.LsaNtStatusToWinError((int)ret));
        }

        public void Dispose()
        {
            if (lsaHandle != IntPtr.Zero)
            {
                Win32Sec.LsaClose(lsaHandle);
                lsaHandle = IntPtr.Zero;
            }
            GC.SuppressFinalize(this);
        }
        ~LsaWrapper()
        {
            Dispose();
        }
        // helper functions

        IntPtr GetSIDInformation(string account)
        {
            LSA_UNICODE_STRING[] names = new LSA_UNICODE_STRING[1];
            LSA_TRANSLATED_SID2 lts;
            IntPtr tsids = IntPtr.Zero;
            IntPtr tdom = IntPtr.Zero;
            names[0] = InitLsaString(account);
            lts.Sid = IntPtr.Zero;
            Console.WriteLine("String account: {0}", names[0].Length);
            int ret = Win32Sec.LsaLookupNames2(lsaHandle, 0, 1, names, ref tdom, ref tsids);
            if (ret != 0)
                throw new Win32Exception(Win32Sec.LsaNtStatusToWinError(ret));
            lts = (LSA_TRANSLATED_SID2)Marshal.PtrToStructure(tsids,
            typeof(LSA_TRANSLATED_SID2));
            Win32Sec.LsaFreeMemory(tsids);
            Win32Sec.LsaFreeMemory(tdom);
            return lts.Sid;
        }

        static LSA_UNICODE_STRING InitLsaString(string s)
        {
            // Unicode strings max. 32KB
            if (s.Length > 0x7ffe)
                throw new ArgumentException("String too long");
            LSA_UNICODE_STRING lus = new LSA_UNICODE_STRING();
            lus.Buffer = s;
            lus.Length = (ushort)(s.Length * sizeof(char));
            lus.MaximumLength = (ushort)(lus.Length + sizeof(char));
            return lus;
        }
    }
    public class LsaWrapperCaller
    {
        public static void AddPrivileges(string account, string privilege)
        {
            using (LsaWrapper lsaWrapper = new LsaWrapper())
            {
                lsaWrapper.AddPrivileges(account, privilege);
            }
        }
    }
}

'@


#Access this Computer from the Network
#SeNetworkLogonRight
#
#Act as part of the operating system
#SeTcbPrivilege
#
#Log on as a batch job
#SeBatchLogonRight
#
#Log on as a service
#SeServiceLogonRight
#
#Profile System Performance
#SeSystemProfilePrivilege
#
#Replace a process level token
#SeAssignPrimaryTokenPrivilege

if ($args[0] -eq $null -or $args[1] -eq $null)
{
 Write-Warning "Available privs:"
 Write-Host ""
 Write-Host "Access this Computer from the Network"
 Write-Warning "SeNetworkLogonRight"
 Write-Host ""
 Write-Host "Act as part of the operating system"
 Write-Warning "SeTcbPrivilege"
 Write-Host ""
 Write-Host "Log on as a batch job"
 Write-Warning "SeBatchLogonRight"
 Write-Host ""
 Write-Host "Log on as a service"
 Write-Warning "SeServiceLogonRight"
 Write-Host ""
 Write-Host "Profile System Performance"
 Write-Warning "SeSystemProfilePrivilege"
 Write-Host ""
 Write-Host "Replace a process level token"
 Write-Warning "SeAssignPrimaryTokenPrivilege"
 Throw "I don't know what to do, please specify username,right, i.e. mydomain\myuser, SeServiceLogonRight"
}

$Uname=$args[0]
$Right=$args[1]

Write-Warning "Giving $Right to $Uname..."
[MyLsaWrapper.LsaWrapperCaller]::AddPrivileges($Uname, $Right)

Don't let the length of that code scare you off; the only part you need to understand is the last few lines. Now let's examine Setup-Splunkuser.ps1:

This script does a few things: defines the target for these privileges, adds those to a series of groups, and ties the other scripts together. The one thing you'll definitely want to modify is the entry at the top, $SplunkAcct="MYDOMAIN\Splunk Accounts". This should be the domain and group you created to host the service account(s). The next line, $localGroups="Distributed COM Users","Event Log Readers","Performance Log Users" local groups to which you want to add AD groups to, each in quotes and separated by commas. What I've set it to here is perfect for Splunk, but feel free to change it for your needs.

Setup-Splunkuser.ps1 (Click to Expand)  + 

$SplunkAcct="MYDOMAIN\Splunk Accounts"
$localGroups="Distributed COM Users","Event Log Readers","Performance Log Users"

#Get-ScriptDirectory Function #
#############################################################

function Get-ScriptDirectory
{
$Invocation = (Get-Variable MyInvocation -Scope 1).Value
$path=Split-Path $Invocation.MyCommand.Path
# INSERT ESCAPE CHARACTERS FOR USING PATHS IN COMMANDS
$path=$path.replace('{','`{')
$path=$path.replace('}','`}')
$path=$path.replace('[','`[')
$path=$path.replace(']','`]')
$path

}
#**********************************************************************************************************************

#get directory
$ScriptDir=Get-ScriptDirectory

#grant rights
Invoke-Expression "$scriptdir\Grant-Rights.ps1 '$SplunkAcct' 'SeNetworkLogonRight'"
Invoke-Expression "$scriptdir\Grant-Rights.ps1 '$SplunkAcct' 'SeTcbPrivilege'"
Invoke-Expression "$scriptdir\Grant-Rights.ps1 '$SplunkAcct' 'SeBatchLogonRight'"
Invoke-Expression "$scriptdir\Grant-Rights.ps1 '$SplunkAcct' 'SeServiceLogonRight'"
Invoke-Expression "$scriptdir\Grant-Rights.ps1 '$SplunkAcct' 'SeSystemProfilePrivilege'"
Invoke-Expression "$scriptdir\Grant-Rights.ps1 '$SplunkAcct' 'SeAssignPrimaryTokenPrivilege'"

#grant WMI ACLs
Invoke-Expression "$scriptdir\Grant-WMIACL.ps1 '$SplunkAcct'"

#add local group access

#NOTE WE ARE CHANGING THE ACCT VARIABLE TO A FORWARDSLASH NOW DO ALL OTHER FUNCTIONS BEFORE THIS

$SplunkAcct=$SplunkAcct.replace('\','/')
foreach ($localGroup in $localGroups)
{
 $lclGrp=[ADSI]"WinNT://$env:COMPUTERNAME/$localGroup,group"
 $lclGrp.psbase.Invoke("Add",([ADSI]"WinNT://$SplunkAcct").path)
}

Now for the final piece to the puzzle: SetupSplunkuser.bat:

This script exists to provide compatibility with 2008 (not R2) and lower. Native Powershell scripts run correctly on 2008R2 and higher, but a batch launched script will run from 2003 up with no problem. This will be the script we'll use to kick everything off.

powershell.exe -nologo -noexit -file %~dp0\Setup-SplunkUser.ps1

Now that we have our scripts in place, we need to do the hard part. Run them everywhere we would like in the enterprise.

Running the Scripts on Target Machines

As discussed earlier, there are several ways to accomplish this. The solution I'll be using is Group Policy by utilizing the Startup Script functionality in the Computer portion of the policy. Setting up the group policy is easy, the difficult part is getting the scripts to run correctly, but we'll worry about that when we get there. 

Determine Your Target Machines

If you want to configure Splunk access rights across all machines in your domain you can plan on using the default domain policy GPO to roll out these changes. If you're reading this article, however, I suspect you want to be more targeted with your application of changes.

I recommend using security filtering against the desired groups of computers. If you don't already have groups you would like to use, create a security group in Active Directory and add your desired machines to it. To do this on a large scale, you can use something like this.

After you have your target group(s) set up, you need to configure the group policy you plan on using to execute the script. If this GPO doesn't yet exist, create it and apply the security filtering listed above. Edit the GPO using GPMC:

1. After opening the GPO, navigate to "Computer Configuration->Policies->Windows Settings->Scripts (Startup/Shutdown)





2. Select "Startup". 
3. Click "Show Files..."; this will open a folder on your sysvol volume associated with this Group Policy Object. 





4. After modifying accordingly (Setup-Splunkuser.ps1) copy all four scripts into the folder. These will be automatically replicated to all domain controllers in the domain. 





5. Back on the "Startup Properties" screen, ensure the "Scripts" tab is selected (not Powershell Scripts) and click "Add". 
6. Type or browse to SetupSplunkUser.bat. And click "OK" twice. 

This now is triggered to run on reboot for all impacted machines, but we're not done yet. We still have to navigate the oddly challenging world of running Powershell scripts in the Windows environment.

Running Powershell Scripts in a Distributed Environment - The Problem

The hurdle using this solution is a larger problem: By default, Powershell scripts are super scary and won't run in your environment for two reasons:

• By default, running scripts requires those scripts be signed. 
• By default, scripts will not run from the "internet", and for some reason the UNC path to the domain controller that may have just authenticated you is considered the "internet" by Windows. 

More here. There are two ways to solve this problem. 

• Secure Way/PKI(Recommended): If you have an internal PKI you can sign all your scripts AND add the signer cert(s) to the trusted publisher store on all targeted machines in Active Directory. 
• Not-So Secure Way/Disable Security: You can run scripts from your DC by setting the execution policy in Powershell to "Unrestricted" and adding your domain controller sysvol share to the trusted sites zone in Internet Explorer configuration. 

I will briefly discuss each solution. 

Secure Solution - PKI

This is the preferred and most secure method. While you could do this with certificates from a public authority, it is recommended that you use your own Active Directory integrated PKI for this. Here is a very high-level overview of the steps:

1. Acquire a code signer cert from your PKI. If using a Windows PKI you can use the "Code Signing" template, but it really should be customized first. Export or save the public key (.cer) because we'll need it later. 
2. Sign each script with the following code (substituting scriptname.ps1 with the location of your script): 

#This assumes you have only one code signing script
#use Get-Childitem cert:\CurrentUser\My -codesigning to see all and change index if necessary

$cert=@(Get-Childitem cert:\CurrentUser\My -codesigning)[0]

#Do this for each script and change scriptname.ps1 to the actual script name

Set-AuthenticodeSignature .\scriptname.ps1 $cert

3. Copy the scripts into the sysvol share as outlined above. 
4. Edit the GPO you would like to use to distribute the script... the same one you're using to run the scripts above would be the perfect match. 
5. Navigate to "Computer Configuration->Policies->Windows Settings->Security Settings->Public Key Policies->Trusted Publishers"
6. Right click->Import and select your public key (.cer) corresponding to your code signing cert. 

Close the GPO and you should now be good. Once the certificate propagates to the target machines they'll run the signed scripts without issue. 

Alternative Solution A - Disable Security for this Script

1. Edit the SetupSplunkUser.bat script and change the line to:

powershell.exe -ExecutionPolicy Bypass -nologo -noexit -file %~dp0\Setup-SplunkUser.ps1

2. Upload the file to your sysvol share as described above. 
3. If you have issues running the other Powershell scripts that are called from the first, you may have to use some trickery: such as this.

Alternative Solution B - Disable Security Permanently

1. Edit the GPO you would like to use to distribute the script... the same one you're using to run the scripts would be the perfect match. 
2. Navigate to "Computer Configuration->Policies->Administrative Templates->Windows Components->Windows Powershell"
3. Edit "Turn on Script Execution" and change it to "Enabled" and "Allow all scripts". Click OK and close the GPO. 
4. Add your domain sysvol path (ensure you use your domain name, not your domain controller name) to the trusted sites zone: see here and here. OR change the UACAsInternet property; see here. 

That should do it! Remote eventlog, WMI, and most other types of remote collection should now work using Splunk.

Troubleshooting

If you have issues getting or determining if the scripts are running successfully, try the following:

• Try executing SetupSplunkUser.bat from an elevated command prompt on one of the target machines. Note any issues. 
• To audit running in a test group, you can add a line like "ran script Setup-SplunkUser.ps1" | Out-File "c:\temp\temp.txt" to create a log of the script execution on each machine. 
• Script execution should be logged in the Application Log of each server. 

With this information you should be able to better control the service rights of Splunk or similar software in a large enterprise environment and still keep your auditors happy. For an added bonus, you could use Splunk itself for security auditing now that your're done.