Foreword
I'm a big fan of Splunk and similar tools for network and systems monitoring. This new generation of machine data analysis brings a myriad of product and monitoring opportunities. These systems work by indexing large swaths of machine logging (logs, performance information, etc.) and providing an incredibly effective interface for reporting of that information.To achieve this, however, you need to provide very broad access to target systems.
If for any reason you can't use Universal Forwarders on Windows platforms, data collection can be difficult. While Splunk provides some great documentation, it suffers from the same access (and granting) problem as other platforms needing similar access. Generally this problem is solved in one of two ways:
- Grant local administrator access on every machine to be monitored to a single service account. Clearly this won't fly in organizations concerned with granting the least amount of access needed for their application portfolio. A single service account/group that has administrator everywhere is a significant security risk.
- Use Group Policy to grant rights, ACLs, etc. The problem with this approach is that some things cannot be addressed by GPO (WMI) and if granting user rights often you cannot be granular enough since each right defined must list all accounts/groups that get that right. Across an enterprise that is a difficult and potentially costly proposition. The main problem with setting user rights via Group Policy is that the rights are not additive; they overwrite all entries including required system or other application entries. This makes central management of thousands of machines impractical. Say you need to define "act as part of the operating system" for your monitoring software; you would then need to centrally define every account/group that has that right and any machine that needs to define that right in a different way would need to obtain an override GPO for the machines in question. While this is possible, many very large organizations find it difficult to execute.
I'll be using Powershell scripts to perform the following tasks:
- Add Active Directory groups/users to local groups
- Grant WMI rights to Active Directory groups/users
- Grant local user rights to Active Directory groups/users
Execution
First things first, we need to establish what we'll be doing and principals to which we want to grant access.
Assumptions
- Splunk indexer (or your other monitoring tool) installed and ready to go
- You have administrative rights on all machines in question and Active Directory rights sufficient to create the service account/group(s)
- You would rather use the Splunk Indexer to collect information rather than install the Splunk Universal Installer on your target machines (Good option as well).
- Powershell 2.0 or newer & Windows 2003 or newer. This could be done with batch/executables or VBScript but I won't be covering that.
Service Account/Group Setup
Per the Splunk instructions, create a service account to run Splunk. This will require administrative access on the machine running Splunk, but nothing else (yet). It would be best to use a managed service account but if you cannot a standard account will be fine.
Splunk instructions deem that an Active Directory group need be created to contain the service account. This isn't really necessary, but there are potential advantages so we'll stick to it.
After creating/configuring that service account, create the service account group and make your service account a member. For the purposes of our demonstration I've named this group "Splunk Accounts"
You should monitor this group to ensure no other accounts are added since doing so would grant those accounts unwanted rights.
Now let's configure services. The service account will need to be local administrator on machines running the Splunk software (splunkd, splunkweb) directly (more on this another time). After granting local admin, stop the Splunkd and splunkweb services, change the logon to your new service account and start them back up.
Now let's tackle the interesting part: granting granular rights to the service account group.
Objective
Our objective is to grant the appropriate rights to the service account
group on every machine in our enterprise that we wish to monitor. Per the
documentation from Splunk, the minimum rights to collect Event Log and WMI
data remotely on a Windows platform are:
Misc
- WMI rights under root/cimv2
- DCOM launch permissions
User Rights
- Access this Computer from the Network
- Act as part of the operating system (I'd like to confirm this one; not sure why)
- Log on as a batch job
- Log on as a service
- Profile System Performance
- Replace a process level token (not sure about this one either)
Local Group Membership
- Distributed COM Users
- Event Log Readers
- Performance Log Users
Scripts
Let's examine the scripts we'll be using. I've prepared four scripts for
this task:
- Powershell script to grant WMI and DCOM rights (Grant-WMIACL.ps1)
- Powershell script to grant User Rights (Grant-Rights.ps1)
- Powershell script to add local group membership and tie the others together (Setup-Splunkuser.ps1)
- Batch file to launch powershell script correctly on all platforms (SetupSplunkuser.bat)
Grant-WMIACL.ps1
As discussed, this script will grant the appropriate WMI and DCOM
access. This is based heavily on the work of Steve Lee and Karl Mitschke. Note that I've left options in the
script and commented out to change the level of access given to the WMI
objects, including the ability to toggle inheritance. I've left these in
to help you decipher the SDDL structure. That said, the line that is not
commented out in this will work fine for our application. This script
could be run remotely (hence the $strComputer="localhost" line) but
we'll be using it locally (more on that below).
Grant-WMIACL.ps1 (Click to Expand) +
She's a beaut eh? Now for Grant-Rights.ps1:
This one is huge because we're using full .NET code in the beginning. That code is a combination of snippets from here, here, and here. Note the comments displaying the short names for some of the rights we can assign which I've left in to facilitate applying this approach to systems other than Spunk. More can be found here.
Grant-Rights.ps1 (Click to Expand) +
Don't let the length of that code scare you off; the only part you need to understand is the last few lines. Now let's examine Setup-Splunkuser.ps1:
This script does a few things: defines the target for these privileges, adds those to a series of groups, and ties the other scripts together. The one thing you'll definitely want to modify is the entry at the top, $SplunkAcct="MYDOMAIN\Splunk Accounts". This should be the domain and group you created to host the service account(s). The next line, $localGroups="Distributed COM Users","Event Log Readers","Performance Log Users" local groups to which you want to add AD groups to, each in quotes and separated by commas. What I've set it to here is perfect for Splunk, but feel free to change it for your needs.
Setup-Splunkuser.ps1 (Click to Expand)
+
Now for the final piece to the puzzle: SetupSplunkuser.bat:
This script exists to provide compatibility with 2008 (not R2) and lower. Native Powershell scripts run correctly on 2008R2 and higher, but a batch launched script will run from 2003 up with no problem. This will be the script we'll use to kick everything off.
powershell.exe -nologo -noexit -file %~dp0\Setup-SplunkUser.ps1
Now that we have our scripts in place, we need to do the hard part. Run them everywhere we would like in the enterprise.
Running the Scripts on Target Machines
As discussed earlier, there are several ways to accomplish this. The solution I'll be using is Group Policy by utilizing the Startup Script functionality in the Computer portion of the policy. Setting up the group policy is easy, the difficult part is getting the scripts to run correctly, but we'll worry about that when we get there.
Determine Your Target Machines
If you want to configure Splunk access rights across all machines in your domain you can plan on using the default domain policy GPO to roll out these changes. If you're reading this article, however, I suspect you want to be more targeted with your application of changes.
I recommend using security filtering against the desired groups of computers. If you don't already have groups you would like to use, create a security group in Active Directory and add your desired machines to it. To do this on a large scale, you can use something like this.
After you have your target group(s) set up, you need to configure the group policy you plan on using to execute the script. If this GPO doesn't yet exist, create it and apply the security filtering listed above. Edit the GPO using GPMC:
- After opening the GPO, navigate to "Computer Configuration->Policies->Windows Settings->Scripts (Startup/Shutdown)
- Select "Startup".
- Click "Show Files..."; this will open a folder on your sysvol volume associated with this Group Policy Object.
- After modifying accordingly (Setup-Splunkuser.ps1) copy all four scripts into the folder. These will be automatically replicated to all domain controllers in the domain.
- Back on the "Startup Properties" screen, ensure the "Scripts" tab is selected (not Powershell Scripts) and click "Add".
- Type or browse to SetupSplunkUser.bat. And click "OK" twice.
This now is triggered to run on reboot for all impacted machines, but we're not done yet. We still have to navigate the oddly challenging world of running Powershell scripts in the Windows environment.
Running Powershell Scripts in a Distributed Environment - The Problem
The hurdle using this solution is a larger problem: By default, Powershell scripts are super scary and won't run in your environment for two reasons:
- By default, running scripts requires those scripts be signed.
- By default, scripts will not run from the "internet", and for some reason the UNC path to the domain controller that may have just authenticated you is considered the "internet" by Windows.
More here. There are two ways to solve this
problem.
- Secure Way/PKI(Recommended): If you have an internal PKI you can sign all your scripts AND add the signer cert(s) to the trusted publisher store on all targeted machines in Active Directory.
- Not-So Secure Way/Disable Security: You can run scripts from your DC by setting the execution policy in Powershell to "Unrestricted" and adding your domain controller sysvol share to the trusted sites zone in Internet Explorer configuration.
I will briefly discuss each solution.
Secure Solution - PKI
This is the preferred and most secure method. While you could do this with certificates from a public authority, it is recommended that you use your own Active Directory integrated PKI for this. Here is a very high-level overview of the steps:
- Acquire a code signer cert from your PKI. If using a Windows PKI you can use the "Code Signing" template, but it really should be customized first. Export or save the public key (.cer) because we'll need it later.
- Sign each script with the following code (substituting scriptname.ps1 with the location of your script):
- Copy the scripts into the sysvol share as outlined above.
- Edit the GPO you would like to use to distribute the script... the same one you're using to run the scripts above would be the perfect match.
- Navigate to "Computer Configuration->Policies->Windows Settings->Security Settings->Public Key Policies->Trusted Publishers"
- Right click->Import and select your public key (.cer) corresponding to your code signing cert.
#This assumes you have only one code signing script
#use Get-Childitem cert:\CurrentUser\My -codesigning to see all and change index if necessary
$cert=@(Get-Childitem cert:\CurrentUser\My -codesigning)[0]
#Do this for each script and change scriptname.ps1 to the actual script name
Set-AuthenticodeSignature .\scriptname.ps1 $cert
Close the GPO and you should now be good. Once the certificate
propagates to the target machines they'll run the signed scripts
without issue.
Alternative Solution A - Disable Security for this Script
- Edit the SetupSplunkUser.bat script and change
the line to:
powershell.exe -ExecutionPolicy Bypass -nologo -noexit -file %~dp0\Setup-SplunkUser.ps1
- Upload the file to your sysvol share as described above.
- If you have issues running the other Powershell scripts that
are called from the first, you may have to use some trickery:
such as this.
powershell.exe -ExecutionPolicy Bypass -nologo -noexit -file %~dp0\Setup-SplunkUser.ps1
Alternative Solution B - Disable Security Permanently
- Edit the GPO you would like to use to distribute the script... the same one you're using to run the scripts would be the perfect match.
- Navigate to "Computer Configuration->Policies->Administrative Templates->Windows Components->Windows Powershell"
- Edit "Turn on Script Execution" and change it to "Enabled" and "Allow all scripts". Click OK and close the GPO.
- Add your domain sysvol path (ensure you use your domain name, not your domain controller name) to the trusted sites zone: see here and here. OR change the UACAsInternet property; see here.
That should do it! Remote eventlog, WMI, and most other types of remote collection should now work using Splunk.
Troubleshooting
If you have issues getting or determining if the scripts are running successfully, try the following:
- Try executing SetupSplunkUser.bat from an elevated command prompt on one of the target machines. Note any issues.
- To audit running in a test group, you can add a line like "ran script Setup-SplunkUser.ps1" | Out-File "c:\temp\temp.txt" to create a log of the script execution on each machine.
- Script execution should be logged in the Application Log of each server.
With this information you should be able to better control the service
rights of Splunk or similar software in a large enterprise environment
and still keep your auditors happy. For an added bonus, you could use
Splunk itself for security
auditing now that your're done.
