Mumble + Splunk

Who's on the Server? Splunk it!

Mumble is a great VOIP solution for latency sensitive situations; I run several servers for different applications, and monitoring those has always been a bit of a challenge since it only generates text logs and doesn't do any historic usage tracking. Fortunately we've got a tool to solve that problem in both real time and historic situations: Splunk.

In this article we'll walk through a simple example of data ingestion, parsing, and dashboard creation in Splunk. When we're finished we'll be able to tell who is online at any given time and how popular your server has been in the recent past.

Step 1: Data Ingestion

This section assumes the following:

• Mumble (or another app if using this as a general guideline) installed and logging to a consistent location.
• Logging is not set to rename logfiles as part of the log rollover process; doing so would complicate our source setup. (note this is achievable) 
• The Splunk forwarder is installed on the target machine(s) and is already configured to output to the indexer(s). (install location referred to henceforth as $SPLUNK_HOME) What's that? You've got thousands of boxes and no time to install? We can fix that!

I'm using Windows boxes in this example, but there's no reason all of this won't work on Linux as well with some minor tweaks.

To get the data into Splunk we'll first need to identify where the data is and how to ensure it gets into the target indexer(s). In my case I'll be targeting the following (4 instances) data sources from one server:

• C:\Apps\Murmur1LowQual\murmur.log
• C:\Apps\Murmur2LowQual\murmur.log
• C:\Apps\Murmur1HighQual\murmur.log
• C:\Apps\Murmur2HighQual\murmur.log
• Performance data: Network interface, CPU Usage

If the application to be monitored is distributed densely and consistently in your enterprise you will want to make the input changes in an "app" for deployment assuming you use "Forwarder Management". In this case it is a one-off configuration so I will be specifying these inputs manually.

For our one-off case, open the $SPLUNK_HOME/etc/system/local/inputs.conf file for editing (for background, make sure you understand "About Configuration Files") and add the following entries:

    [monitor://C:\Apps\MurmurLowQual\murmur.log]
    disabled = false
    sourcetype = murmur
    
    [monitor://C:\Apps\MurmurHighQual\murmur.log]
    disabled = false
    sourcetype = murmur

    [monitor://C:\Apps\Murmur2HighQual\murmur.log]
    disabled = false
    sourcetype = murmur
    
    [monitor://C:\Apps\Murmur2LowQual\murmur.log]
    disabled = false
    sourcetype = murmur
    
    [perfmon://CPU Load]
    counters = % Processor Time
    instances = _Total
    interval = 20
    object = Processor
    
    [perfmon://Network Interface]
    counters = Bytes Received/sec;Bytes Sent/sec
    instances = *
    interval = 15
    object = Network Interface
    

Note these entries are Windows specific; the paths and the perfmon data would need be changed on a Linux host.

Where:

• monitor://<logfilePath> specifies the path to the logfile to be ingested (Murmur is Mumble's server component)
• sourcetype = murmur specifies the sourcetype to store this information under in Splunk. This is critical to properly sorting data.
• perfmon://<counter> , counters = <counter1>;<counter2> , instances = *, and object = <object in question> specify the performance information we want to bring in. This line is Windows specific and needs to be changed for *nix.
• interval = <interval in seconds> is the interval to bring that performance data in at. Lower interval = more granular performance information but more data to store.  

This may or may not be sufficient in your case. For more information, see "Edit inputs.conf" on the Splunk site. 

Step 2: Pre-Format Data

Now we need to take the steps necessary to easily create searches against this data. At a minimum any Splunk Admin should take the time to do proper field extractions from your new data source. Creating custom fields ensure there will be meaningful information for users to search on. This part of Splunk operation is often the make-or-break point in many organizations, as proper field extraction can be the difference between an end user figuring out how to create meaningful searches vs. giving up and going to the original log files.

In this example I'll use one regex to extract three fields from the Murmur logfile under the "search" app context. To set up this field extraction:

1. Log in to your Splunk web interface
2. Change to the "search" app context
3. Navigate to "Settings -> Fields" 





4. Click on "Field extractions"





5. Click "New"
6. Leave the "destination app"as "search" (We'll work in search for now but you could make this into an app)
7. Name it per your enterprise standard. In my case that is <AppName_fields extracted>, so "Murmur_sessionID_UserName_AdminStatus".
8. Change "Apply to" to "sourcetype" named "murmur"
9. Keep "Type" to "Inline"
10. For the Extraction/Transform insert your regex statement. To extract the Session ID (as session_ID), UserName (as u_name), and AdminStatus (as userIsAdmin) from a Murmur logfile use this: "=> <(?<session_ID>[^:]+):(?<u_name>[^\(]+)\((?<userIsAdmin>[^\)]+)"
11. Click "Save"





12. To make this usable for everyone, click "Permissions" under "Sharing" to the right of the name on the "Field extractions" screen. 





13. Select "This app only(search)" (change later if you use a different app) and check "Read" under "Everyone" Click "Save". 

We have now configured field extractions for Mumble. While there is much more one should do (data sizing, business use cases, etc.) to on-board an application this will be enough for now to develop a basic dashboard.

Step 3: Searches and a Dashboard!

Now we'll make use of this data. This is the beauty of Splunk; you can format almost all the data in a meaningful way, and even create new data points inferred from the other available data. To illustrate what I mean, one of these searches will determine who is online right now from the logon/logoff information in the log file. Let's tackle that and a few others:

Search 1: Who is online right now?

sourcetype=murmur |transaction session_ID,u_name maxspan=24h|search authenticated NOT closed|eval AdminRange = case(userIsAdmin < 0, "False", userIsAdmin >= 1, "True")|table u_name,session_ID,AdminRange,_time | rename u_name as "User Name", session_ID as "Session ID", AdminRange as "User is Administrator"

Will generate a table like this:

Where:

sourcetype=murmur : only search the appropriate sourcetype. Note you may want to further limit by host, index, or other locations.

transaction session_ID,u_name maxspan=24h : uses the powerful "transaction" command to string events into a transaction by the listed fields. Note: This command can be computationally expensive so be careful when using it!

search authenticated NOT closed : look for sessions illustrating a user connecting but not yet closed Note: again, since we're using a "NOT" clause this search could be expensive depending on your data volume from the previous parts of the search. Fortunately we should be very limited data wise in this search string by now.

eval AdminRange = case(userIsAdmin < 0, "False", userIsAdmin >= 1, "True") : use the eval command to determine admin status as Boolean

table u_name,session_ID,AdminRange,_time : Render as a table

rename u_name as "User Name", session_ID as "Session ID", AdminRange as "User is Administrator" : Rename table fields to be more meaningful to the end user

Since we're limiting a session span to 24hours in the transaction search, you may as well reduce the time for this search to 24 hours as well. For a slight performance tweak on searches that display a table or graphic of specific fields you can change the search mode from "smart" to "fast".  Now let's add this to a new dashboard:

1. After the results come up, click "Save As" the "Dashboard Panel"  on the upper right hand side.
2. Select "New"
3. Insert an appropriate title, i.e. "Mumble Statistics" following enterprise standards if applicable. Generally it is OK to let the Dashboard ID auto-populate with this name.
4. Write a description if desired and change the Dashboard Permissions to "Shared in App" so we can share this information with others. 
5. Type an appropriate Panel Tile such as "Users Online Now!". Accept the defaults for the remaining and click "Save".

For fun you can make a little tachometer displaying this information by opening the saved search and saving it to the existing dashboard under a different name, then changing the display to "Radial Gauge". End users generally like these "at a glance" graphics for important information, especially at the top of a dashboard.

Search 2: How many people have logged on per day for the last 30 days?

sourcetype=murmur u_name=* authenticated |timechart count

Will generate:

Where:

sourcetype=murmur u_name=* authenticated : only search the appropriate sourcetype for events where u_name is populated and the word "authenticated" is present.

timechart count : charts all results using the very easy timechart command. Make sure you limit the search scope (below).

Set the search scope for 30 days, execute, then save to our existing Mumble Statistics dashboard as a panel named "Logins Per Day - Last 30 Days". I prefer this as a bar chart, which you can select at the dashboard level.

Search 3: How much bandwidth did the server use in the last day?

host=hostname_here sourcetype="Perfmon:Network Interface"|eval DataSrc=case((instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Received/sec"),"ETH0bitsSecIN",(instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Sent/sec"),"ETH0bitsSecOUT")|eval Kbits_Sec=Value*.008| timechart span=5m avg(Kbits_Sec) by DataSrc| rename LANbitsSecIN as "LAN Kbits/sec IN" LANbitsSecOUT as "LAN Kbits/sec OUT" WANbitsSecIN as "WAN Kbits/sec IN" WANbitsSecOUT as "WAN Kbits/sec OUT"

Will generate:

Where:

host=hostname_here sourcetype="Perfmon:Network Interface" :  Specify events to search. You will need to change the hostname and potentially the sourcetype depending on your host platform, etc.

eval DataSrc=case((instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Received/sec"),"ETH0bytesSecIN",(instance=="Microsoft Hyper-V Network Adapter" AND counter=="Bytes Sent/sec"),"ETH0bytesSecOUT") : Here is where we use eval to map interfaces to directions. If you have multiple interfaces you'll need to address them on this line and also note you will need to change the "names" of the adapters to match your data.

eval Kbits_Sec=Value*.008 : Convert Bytes/Sec to Kbits/Sec

timechart span=5m avg(Kbits_Sec) by DataSrc : Chart the data by interface direction on a 5 minute average. Note if you made a longer-term chart you'll need to change the average to calculate on a wider basis to keep your datapoints low enough to chart.

rename ETH0bytesSecIN as "ETH0 Kbits/sec IN" ETH0bytesSecOUT as "ETH0 Kbits/sec OUT" : Rename the datapoints relative to our eval statement above.

Set the search scope for 24 hours, execute, then save to our existing Mumble Statistics dashboard as a panel named "Network Traffic 5m Average Last 24 Hours"

How many users on the sever? Plenty.

Those three should get you started; clearly there is substantially more one could do with all the data available to us. After you decide what else to add make sure you go through your dashboard and reposition/edit each panel as necessary. Keep in mind that you can rename x/y axis as well as change the way data is rendered. Hopefully this tutorial has demonstrated that even with a simple application you can use a tool like Splunk to make the day-to-day use and impact on an organization much more transparent. This methodology could easily be turned into an app and distributed throughout your enterprise and/or the Splunk community.

Monitoring Windows: Granular Service Rights in an Enterprise Environment

Foreword

I'm a big fan of Splunk and similar tools for network and systems monitoring. This new generation of machine data analysis brings a myriad of product and monitoring opportunities. These systems work by indexing large swaths of machine logging (logs, performance information, etc.) and providing an incredibly effective interface for reporting of that information.To achieve this, however, you need to provide very broad access to target systems.

If for any reason you can't use Universal Forwarders on Windows platforms, data collection can be difficult. While Splunk provides some great documentation, it suffers from the same access (and granting) problem as other platforms needing similar access. Generally this problem is solved in one of two ways:

• Grant local administrator access on every machine to be monitored to a single service account. Clearly this won't fly in organizations concerned with granting the least amount of access needed for their application portfolio. A single service account/group that has administrator everywhere is a significant security risk. 
• Use Group Policy to grant rights, ACLs, etc. The problem with this approach is that some things cannot be addressed by GPO (WMI) and if granting user rights often you cannot be granular enough since each right defined must list all accounts/groups that get that right. Across an enterprise that is a difficult and potentially costly proposition. The main problem with setting user rights via Group Policy is that the rights are not additive; they overwrite all entries including required system or other application entries. This makes central management of thousands of machines impractical. Say you need to define "act as part of the operating system" for your monitoring software; you would then need to centrally define every account/group that has that right and any machine that needs to define that right in a different way would need to obtain an override GPO for the machines in question. While this is possible, many very large organizations find it difficult to execute.

Neither one of these solutions is desirable in a medium to large sized enterprise due to security requirements and management complexities. To address this issue in a more granular way than the two options listed above, we can use scripts to automate local configuration where necessary.

I'll be using Powershell scripts to perform the following tasks:

• Add Active Directory groups/users to local groups
• Grant WMI rights to Active Directory groups/users
• Grant local user rights to  Active Directory groups/users

Assuming the affected security principals are not set elsewhere, we can set these items with scripts that can be triggered via group policy. I'll be using the GPO startup script functionality, but you can feel free to use any other trigger mechanism you prefer (logon script, etc.). Note that this solution will work for products similar to Splunk that require similar rights.

Execution

First things first, we need to establish what we'll be doing and principals to which we want to grant access.

Assumptions

• Splunk indexer (or your other monitoring tool) installed and ready to go
• You have administrative rights on all machines in question and Active Directory rights sufficient to create the service account/group(s)
• You would rather use the Splunk Indexer to collect information rather than install the Splunk Universal Installer on your target machines (Good option as well). 
• Powershell 2.0 or newer & Windows 2003 or newer. This could be done with batch/executables or VBScript but I won't be covering that.

Service Account/Group Setup

Per the Splunk instructions, create a service account to run Splunk. This will require administrative access on the machine running Splunk, but nothing else (yet). It would be best to use a managed service account but if you cannot a standard account will be fine.

Splunk instructions deem that an Active Directory group need be created to contain the service account. This isn't really necessary, but there are potential advantages so we'll stick to it.

After creating/configuring that service account, create the service account group and make your service account a member. For the purposes of our demonstration I've named this group "Splunk Accounts"

You should monitor this group to ensure no other accounts are added since doing so would grant those accounts unwanted rights.

Now let's configure services. The service account will need to be local administrator on machines running the Splunk software (splunkd, splunkweb) directly (more on this another time). After granting local admin, stop the Splunkd and splunkweb services, change the logon to your new service account and start them back up.

Now let's tackle the interesting part: granting granular rights to the service account group.

Objective 

Our objective is to grant the appropriate rights to the service account group on every machine in our enterprise that we wish to monitor. Per the documentation from Splunk, the minimum rights to collect Event Log and WMI data remotely on a Windows platform are: 

Misc

• WMI rights under root/cimv2
• DCOM launch permissions

User Rights

• Access this Computer from the Network
• Act as part of the operating system (I'd like to confirm this one; not sure why)
• Log on as a batch job
• Log on as a service
• Profile System Performance
• Replace a process level token (not sure about this one either)

Local Group Membership

• Distributed COM Users
• Event Log Readers
• Performance Log Users

I'm sure you can see the why the official documentation prefers local admin rights. :)  Fortunately, we've got automation to do our heavy lifting.

Scripts

Let's examine the scripts we'll be using. I've prepared four scripts for this task: 

• Powershell script to grant WMI and DCOM rights (Grant-WMIACL.ps1)
• Powershell script to grant User Rights (Grant-Rights.ps1)
• Powershell script to add local group membership and tie the others together (Setup-Splunkuser.ps1)
• Batch file to launch powershell script correctly on all platforms (SetupSplunkuser.bat)

Grant-WMIACL.ps1

As discussed, this script will grant the appropriate WMI and DCOM access. This is based heavily on the work of Steve Lee and Karl Mitschke. Note that I've left options in the script and commented out to change the level of access given to the WMI objects, including the ability to toggle inheritance. I've left these in to help you decipher the SDDL structure. That said, the line that is not commented out in this will work fine for our application. This script could be run remotely (hence the $strComputer="localhost" line) but we'll be using it locally (more on that below).

Grant-WMIACL.ps1 (Click to Expand)  + 

if ($args[0] -eq $null)
{
 Write-Warning "No account or group name provided as an argument. Re-launch with the principal to grant rights to as the first argument."
 Throw "I don't know what to do, please specify username, i.e. mydomain\myuser"
}
$Uname=$args[0]

Write-Warning "Granting WMI and DCOM access to $Uname..."

#This function gets a SID from an account name
function get-sid
{
 Param ($DSIdentity)
 $ID = new-object System.Security.Principal.NTAccount($DSIdentity)
 return $ID.Translate( [System.Security.Principal.SecurityIdentifier] ).toString()
}
$sid = get-sid $Uname

#Enable Account, Remote Enable
#maps to "CC"  SDDL_CREATE_CHILD  ADS_RIGHT_DS_CREATE_CHILD , "WP"  SDDL_WRITE_PROPERTY  ADS_RIGHT_DS_WRITE_PROP 
#$SDDL = "A;;CCWP;;;$sid"

#Provider Write(RP), Enable Account, Remote Enable
#maps to "RP"  SDDL_READ_PROPERTY  ADS_RIGHT_DS_READ_PROP , "CC"  SDDL_CREATE_CHILD  ADS_RIGHT_DS_CREATE_CHILD , "WP"  SDDL_WRITE_PROPERTY  ADS_RIGHT_DS_WRITE_PROP 
#$SDDL = "A;;CCWPRP;;;$sid"

#Execute Methods(DC), Provider Write(RP), Enable Account, Remote Enable
#maps to "DC"  SDDL_DELETE_CHILD  ADS_RIGHT_DS_DELETE_CHILD , "RP"  SDDL_READ_PROPERTY  ADS_RIGHT_DS_READ_PROP , "CC"  SDDL_CREATE_CHILD  ADS_RIGHT_DS_CREATE_CHILD , "WP"  SDDL_WRITE_PROPERTY  ADS_RIGHT_DS_WRITE_PROP 
#$SDDL = "A;;CCWPRPDC;;;$sid"

#Execute Methods(DC), Provider Write(RP), Enable Account, Remote Enable WITH INHERITENCE!! (note CI is in the header) 
#maps to "CI"  SDDL_CONTAINER_INHERIT  CONTAINER_INHERIT_ACE , "DC"  SDDL_DELETE_CHILD  ADS_RIGHT_DS_DELETE_CHILD , "RP"  SDDL_READ_PROPERTY  ADS_RIGHT_DS_READ_PROP , "CC"  SDDL_CREATE_CHILD  ADS_RIGHT_DS_CREATE_CHILD , "WP"  SDDL_WRITE_PROPERTY  ADS_RIGHT_DS_WRITE_PROP 
$SDDL = "A;CI;CCWPRPDC;;;$sid"

#Local Launch and Remote Activation
$DCOMSDDL = "A;;CCDCRP;;;$sid"

#Local Launch, Local Activation, Remote Launch, remote Activation
#$DCOMSDDL = “A;;CCDCLCSWRP;;;$sid”

#$computers = Get-Content "computers.txt"
#foreach ($strcomputer in $computers)
#{
$strComputer="localhost"
$Reg = [WMIClass]"\\$strComputer\root\default:StdRegProv"
$DCOM = $Reg.GetBinaryValue(2147483650,"software\microsoft\ole","MachineLaunchRestriction").uValue
$security = Get-WmiObject -ComputerName $strcomputer -Namespace root/cimv2 -Class __SystemSecurity
$converter = new-object system.management.ManagementClass Win32_SecurityDescriptorHelper
$binarySD = @($null)
$result = $security.PsBase.InvokeMethod("GetSD",$binarySD)
$outsddl = $converter.BinarySDToSDDL($binarySD[0])
$outDCOMSDDL = $converter.BinarySDToSDDL($DCOM)
$newSDDL = $outsddl.SDDL += "(" + $SDDL + ")"
$newDCOMSDDL = $outDCOMSDDL.SDDL += "(" + $DCOMSDDL + ")"
$WMIbinarySD = $converter.SDDLToBinarySD($newSDDL)
$WMIconvertedPermissions = ,$WMIbinarySD.BinarySD
$DCOMbinarySD = $converter.SDDLToBinarySD($newDCOMSDDL)
$DCOMconvertedPermissions = ,$DCOMbinarySD.BinarySD
#Set WMI Permissions
$result = $security.PsBase.InvokeMethod("SetSD",$WMIconvertedPermissions)
#Set DCOM Permissions
$result = $Reg.SetBinaryValue(2147483650,"software\microsoft\ole","MachineLaunchRestriction", $DCOMbinarySD.binarySD)

She's a beaut eh? Now for Grant-Rights.ps1:

This one is huge because we're using full .NET code in the beginning. That code is a combination of snippets from here, here, and here. Note the comments displaying the short names for some of the rights we can assign which I've left in to facilitate applying this approach to systems other than Spunk. More can be found here.

Grant-Rights.ps1 (Click to Expand)  + 

Add-Type @'
using System;
using System.Collections.Generic;
using System.Text;

namespace MyLsaWrapper
{
    using System.Runtime.InteropServices;
    using System.Security;
    using System.Management;
    using System.Runtime.CompilerServices;
    using System.ComponentModel;

    using LSA_HANDLE = IntPtr;

    [StructLayout(LayoutKind.Sequential)]
    struct LSA_OBJECT_ATTRIBUTES
    {
        internal int Length;
        internal IntPtr RootDirectory;
        internal IntPtr ObjectName;
        internal int Attributes;
        internal IntPtr SecurityDescriptor;
        internal IntPtr SecurityQualityOfService;
    }
    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
    struct LSA_UNICODE_STRING
    {
        internal ushort Length;
        internal ushort MaximumLength;
        [MarshalAs(UnmanagedType.LPWStr)]
        internal string Buffer;
    }
    sealed class Win32Sec
    {
        [DllImport("advapi32", CharSet = CharSet.Unicode, SetLastError = true),
        SuppressUnmanagedCodeSecurityAttribute]
        internal static extern uint LsaOpenPolicy(
        LSA_UNICODE_STRING[] SystemName,
        ref LSA_OBJECT_ATTRIBUTES ObjectAttributes,
        int AccessMask,
        out IntPtr PolicyHandle
        );

        [DllImport("advapi32", CharSet = CharSet.Unicode, SetLastError = true),
        SuppressUnmanagedCodeSecurityAttribute]
        internal static extern uint LsaAddAccountRights(
        LSA_HANDLE PolicyHandle,
        IntPtr pSID,
        LSA_UNICODE_STRING[] UserRights,
        int CountOfRights
        );

        [DllImport("advapi32", CharSet = CharSet.Unicode, SetLastError = true),
        SuppressUnmanagedCodeSecurityAttribute]
        internal static extern int LsaLookupNames2(
        LSA_HANDLE PolicyHandle,
        uint Flags,
        uint Count,
        LSA_UNICODE_STRING[] Names,
        ref IntPtr ReferencedDomains,
        ref IntPtr Sids
        );

        [DllImport("advapi32")]
        internal static extern int LsaNtStatusToWinError(int NTSTATUS);

        [DllImport("advapi32")]
        internal static extern int LsaClose(IntPtr PolicyHandle);

        [DllImport("advapi32")]
        internal static extern int LsaFreeMemory(IntPtr Buffer);

    }
    /// 

    /// This class is used to grant "Log on as a service", "Log on as a batchjob", "Log on localy" etc.
    /// to a user.
    /// 
    public sealed class LsaWrapper : IDisposable
    {
        [StructLayout(LayoutKind.Sequential)]
        struct LSA_TRUST_INFORMATION
        {
            internal LSA_UNICODE_STRING Name;
            internal IntPtr Sid;
        }
        [StructLayout(LayoutKind.Sequential)]
        struct LSA_TRANSLATED_SID2
        {
            internal SidNameUse Use;
            internal IntPtr Sid;
            internal int DomainIndex;
            uint Flags;
        }

        [StructLayout(LayoutKind.Sequential)]
        struct LSA_REFERENCED_DOMAIN_LIST
        {
            internal uint Entries;
            internal LSA_TRUST_INFORMATION Domains;
        }

        enum SidNameUse : int
        {
            User = 1,
            Group = 2,
            Domain = 3,
            Alias = 4,
            KnownGroup = 5,
            DeletedAccount = 6,
            Invalid = 7,
            Unknown = 8,
            Computer = 9
        }

        enum Access : int
        {
            POLICY_READ = 0x20006,
            POLICY_ALL_ACCESS = 0x00F0FFF,
            POLICY_EXECUTE = 0X20801,
            POLICY_WRITE = 0X207F8
        }
        const uint STATUS_ACCESS_DENIED = 0xc0000022;
        const uint STATUS_INSUFFICIENT_RESOURCES = 0xc000009a;
        const uint STATUS_NO_MEMORY = 0xc0000017;

        IntPtr lsaHandle;

        public LsaWrapper()
            : this(null)
        { }
        // // local system if systemName is null
        public LsaWrapper(string systemName)
        {
            LSA_OBJECT_ATTRIBUTES lsaAttr;
            lsaAttr.RootDirectory = IntPtr.Zero;
            lsaAttr.ObjectName = IntPtr.Zero;
            lsaAttr.Attributes = 0;
            lsaAttr.SecurityDescriptor = IntPtr.Zero;
            lsaAttr.SecurityQualityOfService = IntPtr.Zero;
            lsaAttr.Length = Marshal.SizeOf(typeof(LSA_OBJECT_ATTRIBUTES));
            lsaHandle = IntPtr.Zero;
            LSA_UNICODE_STRING[] system = null;
            if (systemName != null)
            {
                system = new LSA_UNICODE_STRING[1];
                system[0] = InitLsaString(systemName);
            }

            uint ret = Win32Sec.LsaOpenPolicy(system, ref lsaAttr,
            (int)Access.POLICY_ALL_ACCESS, out lsaHandle);
            if (ret == 0)
                return;
            if (ret == STATUS_ACCESS_DENIED)
            {
                throw new UnauthorizedAccessException();
            }
            if ((ret == STATUS_INSUFFICIENT_RESOURCES) || (ret == STATUS_NO_MEMORY))
            {
                throw new OutOfMemoryException();
            }
            throw new Win32Exception(Win32Sec.LsaNtStatusToWinError((int)ret));
        }

        public void AddPrivileges(string account, string privilege)
        {
            LSA_UNICODE_STRING[] names = new LSA_UNICODE_STRING[1];
            LSA_TRANSLATED_SID2 lts;
            IntPtr tsids = IntPtr.Zero;
            IntPtr tdom = IntPtr.Zero;
            names[0] = InitLsaString(account);
            lts.Sid = IntPtr.Zero;
            Console.WriteLine("String account: {0}", names[0].Length);
            int ret1 = Win32Sec.LsaLookupNames2(lsaHandle, 0, 1, names, ref tdom, ref tsids);
            if (ret1 != 0)
                throw new Win32Exception(Win32Sec.LsaNtStatusToWinError(ret1));
            lts = (LSA_TRANSLATED_SID2)Marshal.PtrToStructure(tsids, typeof(LSA_TRANSLATED_SID2));
            IntPtr pSid = lts.Sid;
            //IntPtr pSid = GetSIDInformation(account);
            LSA_UNICODE_STRING[] privileges = new LSA_UNICODE_STRING[1];
            privileges[0] = InitLsaString(privilege);
            uint ret = Win32Sec.LsaAddAccountRights(lsaHandle, pSid, privileges, 1);
            Win32Sec.LsaFreeMemory(tsids);
            Win32Sec.LsaFreeMemory(tdom);
            if (ret == 0)
                return;
            if (ret == STATUS_ACCESS_DENIED)
            {
                throw new UnauthorizedAccessException();
            }
            if ((ret == STATUS_INSUFFICIENT_RESOURCES) || (ret == STATUS_NO_MEMORY))
            {
                throw new OutOfMemoryException();
            }
            throw new Win32Exception(Win32Sec.LsaNtStatusToWinError((int)ret));
        }

        public void Dispose()
        {
            if (lsaHandle != IntPtr.Zero)
            {
                Win32Sec.LsaClose(lsaHandle);
                lsaHandle = IntPtr.Zero;
            }
            GC.SuppressFinalize(this);
        }
        ~LsaWrapper()
        {
            Dispose();
        }
        // helper functions

        IntPtr GetSIDInformation(string account)
        {
            LSA_UNICODE_STRING[] names = new LSA_UNICODE_STRING[1];
            LSA_TRANSLATED_SID2 lts;
            IntPtr tsids = IntPtr.Zero;
            IntPtr tdom = IntPtr.Zero;
            names[0] = InitLsaString(account);
            lts.Sid = IntPtr.Zero;
            Console.WriteLine("String account: {0}", names[0].Length);
            int ret = Win32Sec.LsaLookupNames2(lsaHandle, 0, 1, names, ref tdom, ref tsids);
            if (ret != 0)
                throw new Win32Exception(Win32Sec.LsaNtStatusToWinError(ret));
            lts = (LSA_TRANSLATED_SID2)Marshal.PtrToStructure(tsids,
            typeof(LSA_TRANSLATED_SID2));
            Win32Sec.LsaFreeMemory(tsids);
            Win32Sec.LsaFreeMemory(tdom);
            return lts.Sid;
        }

        static LSA_UNICODE_STRING InitLsaString(string s)
        {
            // Unicode strings max. 32KB
            if (s.Length > 0x7ffe)
                throw new ArgumentException("String too long");
            LSA_UNICODE_STRING lus = new LSA_UNICODE_STRING();
            lus.Buffer = s;
            lus.Length = (ushort)(s.Length * sizeof(char));
            lus.MaximumLength = (ushort)(lus.Length + sizeof(char));
            return lus;
        }
    }
    public class LsaWrapperCaller
    {
        public static void AddPrivileges(string account, string privilege)
        {
            using (LsaWrapper lsaWrapper = new LsaWrapper())
            {
                lsaWrapper.AddPrivileges(account, privilege);
            }
        }
    }
}

'@


#Access this Computer from the Network
#SeNetworkLogonRight
#
#Act as part of the operating system
#SeTcbPrivilege
#
#Log on as a batch job
#SeBatchLogonRight
#
#Log on as a service
#SeServiceLogonRight
#
#Profile System Performance
#SeSystemProfilePrivilege
#
#Replace a process level token
#SeAssignPrimaryTokenPrivilege

if ($args[0] -eq $null -or $args[1] -eq $null)
{
 Write-Warning "Available privs:"
 Write-Host ""
 Write-Host "Access this Computer from the Network"
 Write-Warning "SeNetworkLogonRight"
 Write-Host ""
 Write-Host "Act as part of the operating system"
 Write-Warning "SeTcbPrivilege"
 Write-Host ""
 Write-Host "Log on as a batch job"
 Write-Warning "SeBatchLogonRight"
 Write-Host ""
 Write-Host "Log on as a service"
 Write-Warning "SeServiceLogonRight"
 Write-Host ""
 Write-Host "Profile System Performance"
 Write-Warning "SeSystemProfilePrivilege"
 Write-Host ""
 Write-Host "Replace a process level token"
 Write-Warning "SeAssignPrimaryTokenPrivilege"
 Throw "I don't know what to do, please specify username,right, i.e. mydomain\myuser, SeServiceLogonRight"
}

$Uname=$args[0]
$Right=$args[1]

Write-Warning "Giving $Right to $Uname..."
[MyLsaWrapper.LsaWrapperCaller]::AddPrivileges($Uname, $Right)

Don't let the length of that code scare you off; the only part you need to understand is the last few lines. Now let's examine Setup-Splunkuser.ps1:

This script does a few things: defines the target for these privileges, adds those to a series of groups, and ties the other scripts together. The one thing you'll definitely want to modify is the entry at the top, $SplunkAcct="MYDOMAIN\Splunk Accounts". This should be the domain and group you created to host the service account(s). The next line, $localGroups="Distributed COM Users","Event Log Readers","Performance Log Users" local groups to which you want to add AD groups to, each in quotes and separated by commas. What I've set it to here is perfect for Splunk, but feel free to change it for your needs.

Setup-Splunkuser.ps1 (Click to Expand)  + 

$SplunkAcct="MYDOMAIN\Splunk Accounts"
$localGroups="Distributed COM Users","Event Log Readers","Performance Log Users"

#Get-ScriptDirectory Function #
#############################################################

function Get-ScriptDirectory
{
$Invocation = (Get-Variable MyInvocation -Scope 1).Value
$path=Split-Path $Invocation.MyCommand.Path
# INSERT ESCAPE CHARACTERS FOR USING PATHS IN COMMANDS
$path=$path.replace('{','`{')
$path=$path.replace('}','`}')
$path=$path.replace('[','`[')
$path=$path.replace(']','`]')
$path

}
#**********************************************************************************************************************

#get directory
$ScriptDir=Get-ScriptDirectory

#grant rights
Invoke-Expression "$scriptdir\Grant-Rights.ps1 '$SplunkAcct' 'SeNetworkLogonRight'"
Invoke-Expression "$scriptdir\Grant-Rights.ps1 '$SplunkAcct' 'SeTcbPrivilege'"
Invoke-Expression "$scriptdir\Grant-Rights.ps1 '$SplunkAcct' 'SeBatchLogonRight'"
Invoke-Expression "$scriptdir\Grant-Rights.ps1 '$SplunkAcct' 'SeServiceLogonRight'"
Invoke-Expression "$scriptdir\Grant-Rights.ps1 '$SplunkAcct' 'SeSystemProfilePrivilege'"
Invoke-Expression "$scriptdir\Grant-Rights.ps1 '$SplunkAcct' 'SeAssignPrimaryTokenPrivilege'"

#grant WMI ACLs
Invoke-Expression "$scriptdir\Grant-WMIACL.ps1 '$SplunkAcct'"

#add local group access

#NOTE WE ARE CHANGING THE ACCT VARIABLE TO A FORWARDSLASH NOW DO ALL OTHER FUNCTIONS BEFORE THIS

$SplunkAcct=$SplunkAcct.replace('\','/')
foreach ($localGroup in $localGroups)
{
 $lclGrp=[ADSI]"WinNT://$env:COMPUTERNAME/$localGroup,group"
 $lclGrp.psbase.Invoke("Add",([ADSI]"WinNT://$SplunkAcct").path)
}

Now for the final piece to the puzzle: SetupSplunkuser.bat:

This script exists to provide compatibility with 2008 (not R2) and lower. Native Powershell scripts run correctly on 2008R2 and higher, but a batch launched script will run from 2003 up with no problem. This will be the script we'll use to kick everything off.

powershell.exe -nologo -noexit -file %~dp0\Setup-SplunkUser.ps1

Now that we have our scripts in place, we need to do the hard part. Run them everywhere we would like in the enterprise.

Running the Scripts on Target Machines

As discussed earlier, there are several ways to accomplish this. The solution I'll be using is Group Policy by utilizing the Startup Script functionality in the Computer portion of the policy. Setting up the group policy is easy, the difficult part is getting the scripts to run correctly, but we'll worry about that when we get there. 

Determine Your Target Machines

If you want to configure Splunk access rights across all machines in your domain you can plan on using the default domain policy GPO to roll out these changes. If you're reading this article, however, I suspect you want to be more targeted with your application of changes.

I recommend using security filtering against the desired groups of computers. If you don't already have groups you would like to use, create a security group in Active Directory and add your desired machines to it. To do this on a large scale, you can use something like this.

After you have your target group(s) set up, you need to configure the group policy you plan on using to execute the script. If this GPO doesn't yet exist, create it and apply the security filtering listed above. Edit the GPO using GPMC:

1. After opening the GPO, navigate to "Computer Configuration->Policies->Windows Settings->Scripts (Startup/Shutdown)





2. Select "Startup". 
3. Click "Show Files..."; this will open a folder on your sysvol volume associated with this Group Policy Object. 





4. After modifying accordingly (Setup-Splunkuser.ps1) copy all four scripts into the folder. These will be automatically replicated to all domain controllers in the domain. 





5. Back on the "Startup Properties" screen, ensure the "Scripts" tab is selected (not Powershell Scripts) and click "Add". 
6. Type or browse to SetupSplunkUser.bat. And click "OK" twice. 

This now is triggered to run on reboot for all impacted machines, but we're not done yet. We still have to navigate the oddly challenging world of running Powershell scripts in the Windows environment.

Running Powershell Scripts in a Distributed Environment - The Problem

The hurdle using this solution is a larger problem: By default, Powershell scripts are super scary and won't run in your environment for two reasons:

• By default, running scripts requires those scripts be signed. 
• By default, scripts will not run from the "internet", and for some reason the UNC path to the domain controller that may have just authenticated you is considered the "internet" by Windows. 

More here. There are two ways to solve this problem. 

• Secure Way/PKI(Recommended): If you have an internal PKI you can sign all your scripts AND add the signer cert(s) to the trusted publisher store on all targeted machines in Active Directory. 
• Not-So Secure Way/Disable Security: You can run scripts from your DC by setting the execution policy in Powershell to "Unrestricted" and adding your domain controller sysvol share to the trusted sites zone in Internet Explorer configuration. 

I will briefly discuss each solution. 

Secure Solution - PKI

This is the preferred and most secure method. While you could do this with certificates from a public authority, it is recommended that you use your own Active Directory integrated PKI for this. Here is a very high-level overview of the steps:

1. Acquire a code signer cert from your PKI. If using a Windows PKI you can use the "Code Signing" template, but it really should be customized first. Export or save the public key (.cer) because we'll need it later. 
2. Sign each script with the following code (substituting scriptname.ps1 with the location of your script): 

#This assumes you have only one code signing script
#use Get-Childitem cert:\CurrentUser\My -codesigning to see all and change index if necessary

$cert=@(Get-Childitem cert:\CurrentUser\My -codesigning)[0]

#Do this for each script and change scriptname.ps1 to the actual script name

Set-AuthenticodeSignature .\scriptname.ps1 $cert

3. Copy the scripts into the sysvol share as outlined above. 
4. Edit the GPO you would like to use to distribute the script... the same one you're using to run the scripts above would be the perfect match. 
5. Navigate to "Computer Configuration->Policies->Windows Settings->Security Settings->Public Key Policies->Trusted Publishers"
6. Right click->Import and select your public key (.cer) corresponding to your code signing cert. 

Close the GPO and you should now be good. Once the certificate propagates to the target machines they'll run the signed scripts without issue. 

Alternative Solution A - Disable Security for this Script

1. Edit the SetupSplunkUser.bat script and change the line to:

powershell.exe -ExecutionPolicy Bypass -nologo -noexit -file %~dp0\Setup-SplunkUser.ps1

2. Upload the file to your sysvol share as described above. 
3. If you have issues running the other Powershell scripts that are called from the first, you may have to use some trickery: such as this.

Alternative Solution B - Disable Security Permanently

1. Edit the GPO you would like to use to distribute the script... the same one you're using to run the scripts would be the perfect match. 
2. Navigate to "Computer Configuration->Policies->Administrative Templates->Windows Components->Windows Powershell"
3. Edit "Turn on Script Execution" and change it to "Enabled" and "Allow all scripts". Click OK and close the GPO. 
4. Add your domain sysvol path (ensure you use your domain name, not your domain controller name) to the trusted sites zone: see here and here. OR change the UACAsInternet property; see here. 

That should do it! Remote eventlog, WMI, and most other types of remote collection should now work using Splunk.

Troubleshooting

If you have issues getting or determining if the scripts are running successfully, try the following:

• Try executing SetupSplunkUser.bat from an elevated command prompt on one of the target machines. Note any issues. 
• To audit running in a test group, you can add a line like "ran script Setup-SplunkUser.ps1" | Out-File "c:\temp\temp.txt" to create a log of the script execution on each machine. 
• Script execution should be logged in the Application Log of each server. 

With this information you should be able to better control the service rights of Splunk or similar software in a large enterprise environment and still keep your auditors happy. For an added bonus, you could use Splunk itself for security auditing now that your're done. 

Starting Small: Set Up a Hadoop Compute Cluster Using Raspberry Pis

What is Hadoop?

Hadoop is a big data computing framework that generally refers to the main components: the core, HDFS, and MapReduce. There are several other projects under the umbrella as well. For more information, see this interview with Cloudera CSO Mike Olson.

What is a Raspberry Pi?

The Pi is a small, inexpensive ($39) ARM based computer. It is meant primarily as an educational tool.

Is Hadoop on the Pi practical?

Nope! Compute performance is horrendous.

Then why?

It's a great learning opportunity to work with Hadoop and multiple nodes. It's also cool to be able to put your "compute cluster" in a lunch box.
In reality though, this article is much more about setting up your first Hadoop compute cluster than it is about the Pi.

Could I use this guide to setup a non-Raspberry Pi based (Real) Hadoop Cluster?

Absolutely, please do. I'll make notes to that effect throughout.

I've been wanting to do this anyhow, let's get started!

Yeah, that's what I said.

Hardware Used

• 3x Raspberry Pis
• 1x 110v Power Splitter
• 1x 4 port 10/100 Switch
• 1x Powered 7 port USB hub
• 3x 2' Cat 5 cables
• 3x 2' USB "power" cables
• 3x Class 10 16GB SDHC Cards

Total cost: about $170 bucks. I picked it all up at my local Microcenter, including the Pis!

Pi Hadoop Cluster; caution: may be slow.

If you would like a a high performance Hadoop cluster just pick one of these up on the way home:

This will cost at least $59.95. Maybe more. Probably more.

Raspberry Pi Preparation

We'll do the master node by itself first so that this guide can be used for a single or multi-node setup. Note: On a non-Pi installation skip to Networking Preparation (though you may want to update your OS manually).

Initial Config

1. Download the "Raspbian wheezy" image from here and follow these directions to set up your first SD card. Soft-float isn't necessary.
2. Hook up a monitor and keyboard and start up the device by connecting up the USB power.
3. When the Raspberry Pi config tool launches, change the keyboard layout first. If you set passwords, etc. with the wrong KB layout you may have a hard time with that later.
4. Change the timezone, then the locale. (Language, etc.)
5. Change the default user password
6. Configure the Pi to disable the X environment on boot. (boot_behavior->straight to desktop->no)
7. Enable the SSH Daemon (Advanced-> A4 SSH) then exit the raspi-config tool
8. Reboot the Pi. (sudo reboot)

Update The OS

Note: It is assumed from henceforth you are connecting to the Pi or your OS via SSH. You can continue on the direct terminal if you like until we get to multiple nodes.

1. Log on to your Pi using the Raspberry user with the password you set earlier.
2. To pull the newest sources, execute sudo apt-get update
3. To pull upgrades for the OS and packages, execute sudo apt-get upgrade
4. Reboot the pi. (sudo reboot)

Split the Memory/Overclock

1. After logging back into the Pi, execute sudo raspi-config to bring up the Rasperry Pi config tool.
2. Select Option 8, "Advanced Options"
3. Select Option A3, "Memory Split"
4. When prompted for how much memory the GPU should have, enter the minimum, "16" and hit enter.
5. (If you would like to overclock) Select Option 7, "Overclock", hit "OK" to the warning, and select what speed you would like.
6. On the main menu, select "finish" and hit "Yes" to reboot.

Networking Preparation

Each Hadoop node must have a unique name and static IP.  I'll be following the Debian instructions since the Rasbian build is based on Debian. If you're running another distro follow the instructions for that. (Redhat, CentOS, Ubuntu)

First we need to change the machine hostname by editing /etc/hostname

sudo nano /etc/hostname

Note: I'll be using nano throughout the article; I'm sure some of you will be substituting vi instead. :)
Change the hostname to what you would like and save the file; I used "Node 1" throughout this document. If you're setting up the second node make sure to use a different hostname.

Now to change to a static IP by editing /etc/network/interfaces

sudo nano /etc/network/interfaces

Change the iface eth0 to a static IP by replacing "auto eth0" and/or "iface eth0 inet dhcp" with:

iface eth0 inet static
        address 192.168.1.40
        netmask 255.255.255.0
        gateway 192.168.1.1

Note: Substitute the correct address (IP), netmask, and gateway for your environment.

Make sure you can resolve DNS queries correctly by editing your /etc/resolv.conf:

sudo nano /etc/resolv.conf

Change the content to match below, substituting the correct information for your environment:

domain company.com
search company.com
nameserver 192.168.1.20
nameserver 192.168.1.30

domain=domain suffix for this machine
search=appends when FQDN not specified
nameserver=list DNS servers in order of precedence

We could restart networking etc, but for the sake of simplicity we'll bounce the box:

sudo reboot

Install Java

If you're going for performance you can install the SunJDK, but getting that to work requires a bit of extra effort. Since we're on the Pi and performance isn't our goal, I'll be using OpenJDK which installs easily. If you're installing on a "real" machine/VM, you may want to diverge a bit here and go for the real deal (Probably Java ver 6).
After connecting to the new IP via SSH again, execute:

sudo apt-get install openjdk-7-jdk

Ensure that 7 is the version you want. If not substitute the package accordingly.

Create and Config Hadoop User and Groups

Create the hadoop group:

sudo addgroup hadoop

Create hadoop user "hduser" and place it in the hadoop group; make sure you remember the password!

sudo adduser --ingroup hadoop hduser

Give hduser the ability to sudo:

sudo adduser hduser sudo

Now logout:

logout

Log back in as the newly created hduser. From hence forth everything will be executed as that user.

Setup SSH Certs

SSH is used between Hadoop nodes and services (even on the same node) to coordinate work. These SSH sessions will be run as the Hadoop user, "hduser" that we created earlier. We need to generate a keypair to use for EACH node. Let's do this one first:

Create the certs for use with ssh:

ssh-keygen -t rsa -P ""

When prompted, save the key to the default directory (will be /home/USERNAME/.ssh/id_rsa). If done correctly you will be shown the fingerprint and randomart image.

Copy the public key into the user's authorized keys store (~ represents the home directory of the current user, >> appends to that file to preserve any keys already there) 

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

SSH to localhost and add to the list of known hosts.

ssh localhost

When prompted, type "yes" to add to the list of known hosts.

Download/Install Hadoop!

Note: As of this writing, the newest version of the 1.x branch (which we're covering) is 1.2. (Ahem, 1.2.1, they're quick) You should check to see what the newest stable release is and change the download link below accordingly. Versions and mirrors can be found here.

Download to home dir:

cd ~
wget http://mirror.catn.com/pub/apache/hadoop/core/hadoop-1.2.0/hadoop-1.2.0.tar.gz

Unzip to the /usr/local dir:

sudo tar vxzf hadoop-1.2.0.tar.gz -C /usr/local

Change the dir name of the unzipped hadoop dir. Note that if your version # is different you'll need to adjust the command

cd /usr/local
sudo mv hadoop-1.2.0/ hadoop

Set the hadoop dir owner to be the hadoop user (hduser):

sudo chown -R hduser:hadoop hadoop

Configure the User Environment

Now we'll config the environment for the user (hduser) to run Hadoop. This assumes you're logged on as that user.
Change to the home dir and open the .bashrc file for editing:

cd ~
nano .bashrc

Add the following to the .bashrc file. It doesn't matter where: (note if you're using Sun Java on a real box you need to put in the right dir for JAVA_HOME)

export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-armhf
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin

Bounce the box (you could actually just log off/on, but what the hell eh?):

sudo reboot

Configure Hadoop!

(Finally!) It's time to setup Hadoop on the machine. With these steps we'll set up a single node and we'll discuss multiple node configuration afterward.
First let's ensure your Hadoop install is in place correctly. After logging in as hduser, execute:

hadoop version

you should see something like: (differs depending on version)

Hadoop 1.2.0
Subversion https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1479473
Compiled by hortonfo on Mon May  6 06:59:37 UTC 2013
From source with checksum 2e0dac51ede113c1f2ca8e7d82fb3405
This command was run using /usr/local/hadoop/hadoop-core-1.2.0.jar

Good. Now let's configure. First up we'll set up the environment file that defines some overall runtime parameters:

nano /usr/local/hadoop/conf/hadoop-env.sh

Add the following lines to set the Java Home parameter (specifies where Java is) and how much memory the Hadoop processes are allowed to use. There should be sample lines in your existing hadoop-env.sh that you can just un-comment and modify.
Note: If you're using a "real" machine rather than a Pi, change the heapsize to a number appropriate to your machine (physical mem - (OS + caching overhead)) and ensure you have the right directory for JAVA_HOME because it will likely be different. If you have a Raspberry Pi version "A" rather than "B", you'll heapsize will need to be much lower since the RAM is halved.

export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-armhf
export HADOOP_HEAPSIZE=272

Now let's configure core-site.xml. This file defines critical operational parameters for a Hadoop site.
Note: You'll notice we're using "localhost" in the configuration. That will change when we go to multi-node, but we'll leave it as localhost for a demonstration of an exportable local only configuration.

nano /usr/local/hadoop/conf/core-site.xml

Add the following lines. If there is already information in the file make sure you respect XML format rules. The "description" field is not required, but I've included it to help this tutorial make sense.

<configuration>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/fs/hadoop/tmp</value>    <description>Sets the operating directory for Hadoop data.
    </description>
  </property>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:54310</value>    <description>The name of the default file system.  A URI whose
    scheme and authority determine the FileSystem implementation.
    The URI's scheme determines the config property (fs.SCHEME.impl) naming
    the FileSystem implementation class.  The URI's authority is used to
    determine the host, port, etc. for a filesystem.  
    </description>
  </property>
</configuration>

It's time to configure the mapred-site.xml. This file determines where the mapreduce job tracker(s) run. Again, we're using localhost for now and we'll change it when we go to multi-node in a bit.

nano /usr/local/hadoop/conf/mapred-site.xml

Add the following lines:

<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>localhost:54311</value>
    <description>The host and port that the MapReduce job tracker runs
    at.  If "local", then jobs are run in-process as a single map
    and reduce task.
    </description>
  </property>
</configuration>

Now on to hdfs-site.xml. This file configures the HDFS parameters.

nano /usr/local/hadoop/conf/hdfs-site.xml

Add the following lines:
Note: The dfs.replication value sets how many copies of any given file should exist in the configured HDFS site. We'll do 1 for now since there is only one node, but again when we configure multiple nodes we'll change this.

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
    <description>Default block replication.
    The actual number of replications can be specified when the file is created.
    The default is used if replication is not specified in create time.
    </description>
  </property>
</configuration>

Create the working directory and set permissions

sudo mkdir -p /fs/hadoop/tmp
sudo chown hduser:hadoop /fs/hadoop/tmp
sudo chmod 750 /fs/hadoop/tmp/

Format the working directory

/usr/local/hadoop/bin/hadoop namenode -format

Start Hadoop and Run Your First Job

Let's roll, this will be fun.
Start it up:

cd /usr/local/hadoop
bin/start-all.sh

Now let's make sure it's running successfully. First we'll use the jps command (Java PS) to determine what Java processes are running:

jps

You should see the following processes: (ignore the number in front, that's a process ID)

4863 Jps
4003 SecondaryNameNode
4192 TaskTracker
3893 DataNode
3787 NameNode
4079 JobTracker

If there are any missing processes, you'll need to review logs. By default, logs are located in:

/usr/local/hadoop/logs

There are two types of log files: .out and .log. .out files detail process information while .log files are the logging output from the process. Generally .log files are used for troubleshooting. The logfile name standard is:

hadoop-(username)-(processtype)-(machinename).log

i.e.

hadoop-hduser-jobtracker-node1.log

In this, like many other articles, we'll use the included wordcount example. For the wordcount example you will need plain text large enough to be interesting. For this purpose, you can download plain text books from Project Gutenberg.

After downloading 1 or more books (make sure you selected "Plain Text UTF-8" format) copy them to the local filesystem on your Hadoop node using SSH (via SCP or similar). In my example I've copied the books to /tmp/books.

Now let's copy the books onto the HDFS filesystem by using the Hadoop dfs command.

cd /usr/local/hadoop
bin/hadoop dfs -copyFromLocal /tmp/books /fs/hduser/books

After copying in the book(s), execute the wordcount example:

bin/hadoop jar hadoop*examples*.jar wordcount /fs/hduser/books /fs/hduser/books-output

You'll see the job run; it may take awhile... remember these Pis don't perform all that well. Upon completion you should see something like this:

13/06/17 15:46:54 INFO mapred.JobClient: Job complete: job_201306170244_0001
13/06/17 15:46:55 INFO mapred.JobClient: Counters: 29
13/06/17 15:46:55 INFO mapred.JobClient:   Job Counters
13/06/17 15:46:55 INFO mapred.JobClient:     Launched reduce tasks=1
13/06/17 15:46:55 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=566320
13/06/17 15:46:55 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/06/17 15:46:55 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/06/17 15:46:55 INFO mapred.JobClient:     Launched map tasks=1
13/06/17 15:46:55 INFO mapred.JobClient:     Data-local map tasks=1
13/06/17 15:46:55 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=112626
13/06/17 15:46:55 INFO mapred.JobClient:   File Output Format Counters
13/06/17 15:46:55 INFO mapred.JobClient:     Bytes Written=229278
...

The results from the run, if you're curious, should be in the last part of the command you executed earlier. (/fs/hduser/books-output in our case) Update: to check the output use the hadoop dfs command a la:

hduser@rasdoop1 /usr/local/hadoop $ bin/hadoop dfs -ls /fs/hduser/books-output
Found 3 items
-rw-r--r--   3 hduser supergroup          0 2015-05-05 05:00 /fs/hduser/books-output/_SUCCESS
drwxr-xr-x   - hduser supergroup          0 2015-05-05 04:35 /fs/hduser/books-output/_logs
-rw-r--r--   3 hduser supergroup     926451 2015-05-05 04:58 /fs/hduser/books-output/part-r-00000

hduser@rasdoop1 /usr/local/hadoop $bin/hadoop dfs -cat /fs/hduser/books-output/part-r-00000
...

Congrats! You just ran your first Hadoop job, welcome to the world of (tiny)big data!

Multiple Nodes

Hadoop has been successfully run on thousands of nodes; this is where we derive the power of the platform. The first step to getting to many nodes is getting to 2, so let's do that.

Set up the Secondary Pi

I'll cover cloning a node for the third Pi, so for this one we'll assume you've set up another Pi the same as the first using the instructions above. After that, we just need make appropriate changes to the configuration files. Important Note: I'll be referring to the nodes as Node 1 and Node 2 and making the assumption that you have named them that. Feel free to use different names, just make sure you substitute them in below. Name resolution will be an issue; we'll touch on that below.

Node 1 will run:

• NameNode
• Secondary NameNode (In a large production cluster this would be somewhere else)
• DataNode
• JobTracker (In a large production cluster this would be somewhere else)
• TaskTracker

Node 2 will run:

• DataNode
• TaskTracker

Normally this is where I planned on writing up an overview of the different pieces of Hadoop, but upon researching I found an article that exceeded what I planned to write it by such a magnitude I figured why bother, I'll just link it here. Excellent post by Brad Hedlund, be sure to check it out.
On Node 1, edit masters

nano /usr/local/hadoop/conf/masters

remove "localhost" and add the first node FQDN and save the file:

Node1.domain.ext

Note that conf/masters does NOT determine which node holds "master" roles, i.e. NameNode, JobTracker, etc. It only defines which node will attempt to contact the nodes in slaves (below) to initiate start-up.
On Node 1, edit slaves

nano /usr/local/hadoop/conf/slaves

remove "localhost" add the internal FQDN of all nodes in the cluster and save the file:

Node1.domain.ext
Node2.domain.ext

On ALL (both) cluster members edit core-site.xml

nano /usr/local/hadoop/conf/core-site.xml

and change the fs.default.name value to reflect the location of the NameNode:

<property>
 <name>fs.default.name</name>
 <value>hdfs://node1:54310</value>
</property>

On ALL (both) cluster members edit mapred-site.xml

nano /usr/local/hadoop/conf/mapred-site.xml

and change the mapred.job.tracker value to reflect the location of the JobTracker:

<property>
 <name>mapred.job.tracker</name>
 <value>node1:54311</value>
</property>

On ALL (both) cluster members edit hdfs-site.xml

nano /usr/local/hadoop/conf/hdfs-site.xml

and change the dfs.replication value to reflect the location of the JobTracker:

<property>
 <name>dfs.replication</name>
 <value>2</value>
</property>

The dfs.replication value determines how many copies of any given piece of data will exist in the HDFS site. By default this is set to 3 which is optimal for many small->medium clusters. In this case we set it to the number of nodes we'll have right now, 2.

Now we need to ensure the master node can talk to the slave node(s) by adding the master key to the authorized keys file on each node (just 1 for now)
On Node 1, cat your public key and copy it to the clipboard or some other medium for transfer to the slave node

cat ~/.ssh/id_rsa.pub

On Node 2

nano ~/.ssh/authorized_keys

and paste in the key from the master and save the file to allow it to take commands.

Name Lookup

The Hadoop nodes should be able to resolve the names of the nodes in the cluster. To accomplish this we have two options: the right way and the quick way.

The right way: DNS

The best way to enable name lookup works correctly is by adding the nodes and their IPs to your internal DNS. Not only does this facilitate standard Hadoop operation, but it makes it possible to browse around the Hadoop status web pages (more on that below) from a non-node member, which I suspect you'll want to do. Those sites are automatically created by Hadoop all the hyperlinks use the node DNS names, so you need to be able to resolve those names from a "client" machine. This can scale too; with the right tool set it's very easy to automate DNS updates when spinning up a node.

To do this option, add your Hadoop hostnames to the DNS zone they reside in.

The quick way: HOSTS

If you don't have access to update your internal DNS zone, you'll need to use the hosts files on the nodes. This will allow the nodes to talk to each other via name. There are scenarios where you may want to automate the updating of node based HOSTS files for performance or fault tolerance reasons as well, but I won't go into that here.

To use this option do the following:

On ALL (both) cluster members edit /etc/hosts

nano /etc/hosts

and add each of your nodes with the appropriate IP addresses (mine are only examples)

192.168.1.40    node1
192.168.1.41    node2

Now that the configuration is done we need to format the HDFS filesystem. Note: THIS IS DESTRUCTIVE. Anything currently on your single node Hadoop system will be deleted.
On Node 1

bin/hadoop namenode -format

When that is done, let's start the cluster:

cd /usr/local/hadoop
bin/start-all.sh

Just as before, use the jps command to check processes, but this time on both nodes.
On Node 1

jps

You should see the following processes: (ignore the number in front, that's a process ID)

4863 Jps
4003 SecondaryNameNode
4192 TaskTracker
3893 DataNode
3787 NameNode
4079 JobTracker

On Node 2

jps

You should see the following processes: (ignore the number in front, that's a process ID)

6365 TaskTracker
7248 Jps
6279 DataNode

If there are problems review the logs. (for detailed location see reference above) Note that by default each node contains its own logs.

Now let's run the wordcount example in a cluster. To facilitate this you'll need enough data to split, so when downloading UTF-8 books from Project Gutenberg get at least 6 very long books (that should do with default settings).

Copy them to the local filesystem on your master Hadoop node using SSH (via SCP or similar). In my example I've copied the books to /tmp/books.

Now let's copy the books onto the HDFS filesystem by using the Hadoop dfs command. This will automatically replicate data to all nodes.

cd /usr/local/hadoop
bin/hadoop dfs -copyFromLocal /tmp/books /fs/hduser/books

After copying in the book(s) wait a couple minutes for all blocks to replicate (more on monitoring below) then execute the wordcount example:

bin/hadoop jar hadoop*examples*.jar wordcount /fs/hduser/books /fs/hduser/books-output

Monitoring HDFS/Jobs

Hadoop includes a great set of management web sites that will allow you to monitor jobs, check logfiles, browse the filesystem, etc. Let's take a moment to examine the three sites.

JobTracker
By default, the job tracker site can be found at http://(jobtrackernodename.domain.ext):50030/jobtracker.jsp ; i.e. http://node1.company.com:50030/jobtracker.jsp

On the Job Tracker you can see information regarding the cluster, running map and reduce tasks, node status, running and completed jobs, and you can also drill into specific node and task status.

DFSHealth
By default, the DFSHealth site can be found at http://(NameNodename.domain.ext):50070/dfshealth.jsp ; i.e. http://node1.company.com:50070/dfshealth.jsp

The DFSHealth site allows you to browse the HDFS system, view nodes, look at space consumption, and check NameNode logs.

TaskTracker
By default, the task tracker site can be found at http://(nodename.domain.ext):50060/tasktracker.jsp ; i.e. http://node2.company.com:50060/tasktracker.jsp

The Task Tracker site runs on each node that can run a task to provide more detailed information about what task is running on that node at the time.

Cloning a Node and Adding it to the Cluster

Now we'll explore how to get our third node into the cluster the quickest way possible: cloning the drive. This will cover, at a high level, the SD card cloning process for the Pi. If you're using a real machine replace the drive cloning steps with your favorite (*cough*Clonzilla*cough*) cloning software, or copy the virtual drive if you are using VMs.

You should always clone a slave node unless you intend on making a new cluster. To this end, do the following:

1. Shut down Node 2 (sudo init 0) and remove the SD card.
2. Insert the SD card into a PC to so we can clone it; I'm using a Windows 7 machine so if you're using Linux we'll diverge here. (I'll meet you at the pass!)
3. Capture an image to your hdd using your favorite SDcard imaging software. I use either Roadkill's Disk Image or dd for windows. 
4. Remove the Node 2 SD card and place it back into Node 2. Do not power up yet.
5. Insert a new SD card of the same size to your imaging PC.
6. Delete any partitions on the card to ensure error-free imaging. I use Diskpart , Select Disk x , Clean , exit where "x" is the disk number. Use list disk to find it... don't screw up or you'll wipe out your HDD!
7. Image the new SD card with the image from Node 2 and remove it from the PC when complete.





8. Insert the newly cloned SD card to Node 3. Power up that node and leave Node 2 off for now or the IPs will conflict.

Log into the new node (same IP as the cloned node) via ssh with the hduser (Hadoop user) account and perform the following tasks:

Change the hostname:

sudo nano /etc/hostname

For this example change "Node2" to "Node3" and save the file.

Change the IP address:

sudo nano /etc/network/interfaces

For this example change "192.168.1.41" to "192.168.1.42" and save the file. Switch to your own IP address range if necessary.

If you're using HOSTS for name resolution, add Node 3 with the proper IP address to the hosts file

sudo nano /etc/hosts

Repeat this on the master node and for the sake of completeness the other node(s) should probably be updated as well. In a large scale deployment this would all be automated.
If you are using DNS, make sure to add the Node3 entry to your zone.

Reboot Node 3 to enact the IP and hostname changes

sudo reboot

Power up Node 2 again at this time if you had it powered down. Log back into Node 3 (now with the new IP!) with the hduser account via SSH.

Generate a new key:

ssh-keygen -t rsa -P ""

Copy the key to its own trusted store:

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

If you aren't already, log into Node1 (Master) as hduser via SSH and do the following:
Add Node 3 to the slaves list

nano /usr/local/hadoop/conf/slaves

SSH to Node 3 to add it to known hosts

ssh rasdoop3

When prompted re: continue connecting type yes

An ALL nodes edit hdfs-site.xml

nano /usr/local/hadoop/conf/hdfs-site.xml

Change it to replicate to 3 nodes. Note if you add more nodes to the cluster after this you would most likely NOT increase this above 3. It's not 1:1 that we're going for, it's the number of replicas we want in the entire cluster. Once you're running a cluster with > 3 nodes you'll generally know if you want dfs.replication set to more than 3.

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>3</value>
    <description>Default block replication.
    The actual number of replications can be specified when the file is created.
    The default is used if replication is not specified in create time.
    </description>
  </property>
</configuration>

Now we need to format the filesystem. It is possible, but slightly more complicated, to clone/add a node without re-formatting the whole HDFS cluster, but that's a topic for another blog post. Since we have virtually no data here, we'll address all concerns by wiping clean:
On Node 1:

cd /usr/local/hadoop
bin/hadoop namenode -format

Now we should be able to start up all three nodes. Let's give it a shot!
On Node 1:

bin/start-all.sh

Note: If you have issues starting up the Datanodes you may need to delete the sub-directories under the directory listed in core-site.xml as "hadoop.tmp.dir" and then re-format again (see above). 

As for startup troubleshooting and running the wordcount example, copy out the instructions above after the initial cluster setup. There should be nothing different from running with two nodes save the fact that you may need more books to get it working on all 3 nodes.

Postmortem/Additional Reading

You did it! You've now got the experience of setting up a Hadoop cluster under your belt. There's a ton of directions to go from here; this relatively new tech is changing the way a lot of companies view data and business intelligence.

There is so much great community content out there; here's just a small list of the references I used and some additional reading.

References:

Raspberry Pi forums: Hadoop + HDFS + MR on Pi cluster - works!
Michael G. Noll: Running Hadoop on Ubuntu Linux Single-Node Cluster
Michael G. Noll: Running Hadoop on Ubuntu Linux Multi-Note Cluster
Hadoop Wiki: Getting Started with Hadoop
Hadoop Wiki: Wordcount Example
University of Glasgow's Raspberry Pi Hadoop Project
Hadoop Wiki: Cluster Setup
Hadoop Wiki: Java Versions
All Things Hadoop: Tips, Tricks and Pointers When Setting Up..
Hadoop Wiki: FAQ
Code/Google Hadoop Toolkit: Hadoop Performance Monitoring
Jeremy Morgan: How to Overclock the Raspberry Pi