Who's on the Server? Splunk it!
Mumble is a great VOIP solution for latency sensitive situations; I run several servers for different applications, and monitoring those has always been a bit of a challenge since it only generates text logs and doesn't do any historic usage tracking. Fortunately we've got a tool to solve that problem in both real time and historic situations: Splunk.
In this article we'll walk through a simple example of data ingestion, parsing, and dashboard creation in Splunk. When we're finished we'll be able to tell who is online at any given time and how popular your server has been in the recent past.
Step 1: Data Ingestion
This section assumes the following:- Mumble (or another app if using this as a general guideline) installed and logging to a consistent location.
- Logging is not set to rename logfiles as part of the log rollover process; doing so would complicate our source setup. (note this is achievable)
- The Splunk forwarder is installed on the target machine(s) and is already configured to output to the indexer(s). (install location referred to henceforth as $SPLUNK_HOME) What's that? You've got thousands of boxes and no time to install? We can fix that!
I'm using Windows boxes in this example, but there's no reason all of this won't work on Linux as well with some minor tweaks.
To get the data into Splunk we'll first need to identify where the data is and how to ensure it gets into the target indexer(s). In my case I'll be targeting the following (4 instances) data sources from one server:
- C:\Apps\Murmur1LowQual\murmur.log
- C:\Apps\Murmur2LowQual\murmur.log
- C:\Apps\Murmur1HighQual\murmur.log
- C:\Apps\Murmur2HighQual\murmur.log
- Performance data: Network interface, CPU Usage
If the application to be monitored is distributed densely and consistently in your enterprise you will want to make the input changes in an "app" for deployment assuming you use "Forwarder Management". In this case it is a one-off configuration so I will be specifying these inputs manually.
For our one-off case, open the $SPLUNK_HOME/etc/system/local/inputs.conf file for editing (for background, make sure you understand "About Configuration Files") and add the following entries:
[monitor://C:\Apps\MurmurLowQual\murmur.log]
disabled = false
sourcetype = murmur
[monitor://C:\Apps\MurmurHighQual\murmur.log]
disabled = false
sourcetype = murmur
[monitor://C:\Apps\Murmur2HighQual\murmur.log]
disabled = false
sourcetype = murmur
[monitor://C:\Apps\Murmur2LowQual\murmur.log]
disabled = false
sourcetype = murmur
[perfmon://CPU Load]
counters = % Processor Time
instances = _Total
interval = 20
object = Processor
[perfmon://Network Interface]
counters = Bytes Received/sec;Bytes Sent/sec
instances = *
interval = 15
object = Network Interface
Where:
- monitor://<logfilePath> specifies the path to the logfile to be ingested (Murmur is Mumble's server component)
- sourcetype = murmur specifies the sourcetype to store this information under in Splunk. This is critical to properly sorting data.
- perfmon://<counter> , counters = <counter1>;<counter2> , instances = *, and object = <object in question> specify the performance information we want to bring in. This line is Windows specific and needs to be changed for *nix.
- interval = <interval in seconds> is the interval to bring that performance data in at. Lower interval = more granular performance information but more data to store.
This may or may not be sufficient in your case. For more information, see "Edit inputs.conf" on the Splunk site.
Step 2: Pre-Format Data
Now we need to take the steps necessary to easily create searches against this data. At a minimum any Splunk Admin should take the time to do proper field extractions from your new data source. Creating custom fields ensure there will be meaningful information for users to search on. This part of Splunk operation is often the make-or-break point in many organizations, as proper field extraction can be the difference between an end user figuring out how to create meaningful searches vs. giving up and going to the original log files.
In this example I'll use one regex to extract three fields from the Murmur logfile under the "search" app context. To set up this field extraction:
- Log in to your Splunk web interface
- Change to the "search" app context
- Navigate to "Settings -> Fields"
- Click on "Field extractions"
- Click "New"
- Leave the "destination app"as "search" (We'll work in search for now but you could make this into an app)
- Name it per your enterprise standard. In my case that is <AppName_fields extracted>, so "Murmur_sessionID_UserName_AdminStatus".
- Change "Apply to" to "sourcetype" named "murmur"
- Keep "Type" to "Inline"
- For the Extraction/Transform insert your regex statement. To extract the Session ID (as session_ID), UserName (as u_name), and AdminStatus (as userIsAdmin) from a Murmur logfile use this: "=> <(?<session_ID>[^:]+):(?<u_name>[^\(]+)\((?<userIsAdmin>[^\)]+)"
- Click "Save"
- To make this usable for everyone, click "Permissions" under "Sharing" to the right of the name on the "Field extractions" screen.
- Select "This app only(search)" (change later if you use a different app) and check "Read" under "Everyone" Click "Save".
We have now configured field extractions for Mumble. While there is much more one should do (data sizing, business use cases, etc.) to on-board an application this will be enough for now to develop a basic dashboard.
Step 3: Searches and a Dashboard!
Now we'll make use of this data. This is the beauty of Splunk; you can format almost all the data in a meaningful way, and even create new data points inferred from the other available data. To illustrate what I mean, one of these searches will determine who is online right now from the logon/logoff information in the log file. Let's tackle that and a few others:
Search 1: Who is online right now?
sourcetype=murmur |transaction session_ID,u_name maxspan=24h|search authenticated NOT closed|eval AdminRange = case(userIsAdmin < 0, "False", userIsAdmin >= 1, "True")|table u_name,session_ID,AdminRange,_time | rename u_name as "User Name", session_ID as "Session ID", AdminRange as "User is Administrator"Will generate a table like this:
Where:
sourcetype=murmur : only search the appropriate sourcetype. Note you may want to further limit by host, index, or other locations.
transaction session_ID,u_name maxspan=24h : uses the powerful "transaction" command to string events into a transaction by the listed fields. Note: This command can be computationally expensive so be careful when using it!
search authenticated NOT closed : look for sessions illustrating a user connecting but not yet closed Note: again, since we're using a "NOT" clause this search could be expensive depending on your data volume from the previous parts of the search. Fortunately we should be very limited data wise in this search string by now.
eval AdminRange = case(userIsAdmin < 0, "False", userIsAdmin >= 1, "True") : use the eval command to determine admin status as Boolean
table u_name,session_ID,AdminRange,_time : Render as a table
rename u_name as "User Name", session_ID as "Session ID", AdminRange as "User is Administrator" : Rename table fields to be more meaningful to the end user
Since we're limiting a session span to 24hours in the transaction search, you may as well reduce the time for this search to 24 hours as well. For a slight performance tweak on searches that display a table or graphic of specific fields you can change the search mode from "smart" to "fast". Now let's add this to a new dashboard:
- After the results come up, click "Save As" the "Dashboard Panel" on the upper right hand side.
- Select "New"
- Insert an appropriate title, i.e. "Mumble Statistics" following enterprise standards if applicable. Generally it is OK to let the Dashboard ID auto-populate with this name.
- Write a description if desired and change the Dashboard Permissions to "Shared in App" so we can share this information with others.
- Type an appropriate Panel Tile such as "Users Online Now!". Accept the defaults for the remaining and click "Save".
Search 2: How many people have logged on per day for the last 30 days?
sourcetype=murmur u_name=* authenticated |timechart count
Will generate:
Where:
sourcetype=murmur u_name=* authenticated : only search the appropriate sourcetype for events where u_name is populated and the word "authenticated" is present.
timechart count : charts all results using the very easy timechart command. Make sure you limit the search scope (below).
Set the search scope for 30 days, execute, then save to our existing Mumble Statistics dashboard as a panel named "Logins Per Day - Last 30 Days". I prefer this as a bar chart, which you can select at the dashboard level.
Search 3: How much bandwidth did the server use in the last day?
host=hostname_here sourcetype="Perfmon:Network Interface"|eval DataSrc=case((instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Received/sec"),"ETH0bitsSecIN",(instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Sent/sec"),"ETH0bitsSecOUT")|eval Kbits_Sec=Value*.008| timechart span=5m avg(Kbits_Sec) by DataSrc| rename LANbitsSecIN as "LAN Kbits/sec IN" LANbitsSecOUT as "LAN Kbits/sec OUT" WANbitsSecIN as "WAN Kbits/sec IN" WANbitsSecOUT as "WAN Kbits/sec OUT"
Will generate:
Where:
host=hostname_here sourcetype="Perfmon:Network Interface" : Specify events to search. You will need to change the hostname and potentially the sourcetype depending on your host platform, etc.
eval DataSrc=case((instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Received/sec"),"ETH0bytesSecIN",(instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Sent/sec"),"ETH0bytesSecOUT") : Here is where we use eval to map interfaces to directions. If you have multiple interfaces you'll need to address them on this line and also note you will need to change the "names" of the adapters to match your data.
eval Kbits_Sec=Value*.008 : Convert Bytes/Sec to Kbits/Sec
timechart span=5m avg(Kbits_Sec) by DataSrc : Chart the data by interface direction on a 5 minute average. Note if you made a longer-term chart you'll need to change the average to calculate on a wider basis to keep your datapoints low enough to chart.
rename ETH0bytesSecIN as "ETH0 Kbits/sec IN" ETH0bytesSecOUT as "ETH0 Kbits/sec OUT" : Rename the datapoints relative to our eval statement above.
Set the search scope for 24 hours, execute, then save to our existing Mumble Statistics dashboard as a panel named "Network Traffic 5m Average Last 24 Hours"
 |
| How many users on the sever? Plenty. |
Those three should get you started; clearly there is substantially more one could do with all the data available to us. After you decide what else to add make sure you go through your dashboard and reposition/edit each panel as necessary. Keep in mind that you can rename x/y axis as well as change the way data is rendered. Hopefully this tutorial has demonstrated that even with a simple application you can use a tool like Splunk to make the day-to-day use and impact on an organization much more transparent. This methodology could easily be turned into an app and distributed throughout your enterprise and/or the Splunk community.
