SplunkLive 2011 Advanced Session

Technical Workshops
Advanced User Training

Jeff Blake

May 17, 2011

Agenda
Assumptions
Building Apps
Users and Roles
Splunk By Example – Deconstructing the Demo
Support and Community

Copyright © 2011, Splunk Inc. 2 Listen to your data.

You Are in This Session Because….

• You have developed advanced searches with Splunk to manipulate
and present data
• You have mastered sourcetyping and extracting fields
• You have built reports beyond | timechart count
• You have created dashboards of some kind
• You have http://www.splunk.com/base/Documentation bookmarked
• You have seen all of the Splunk Ninja videos


What is An App, Anyway?
• Terminology
• Apps – A workspace that solves a specific use case with a navigable view
• Add-on – A reusable Splunk component that does not contain a view
• Example
• Splunk for Cisco Security is an App
• The collection of field extractions/sourcetypes/transforms/eventypes that
map raw firewall logs is an Add-on


What You Need to Create an App
• Some data
• Add-ons (Use ours or build your own)
• Splunk Objects
• Saved Searches
• Reports
• Dashboards
• Views
• Text Editor (Komodo Edit, Text Wrangler, vi, Notepad) for Advanced XML
• Web development tools (Firebug, etc)


Getting Started is Easy

This will build the necessary directory structure in
$SPLUNK_HOME/etc/apps


app.conf
The Interesting Stuff:
• version = <version string> - Give your App a version number. Important if you plan to
put on Splunkbase
• id = <appid> - Must be same as folder name where your app lives. With version, used
to check for App updates on Splunkbase
• state_change_requires_restart = true | false - Install Apps without restarting! Check
$SPLUNK_HOME/etc/system/default/apps.conf. If Splunk touches any endpoint not
listed, restart required
• build = <integer> - Change with version so browsers don’t use cached copies of old
static files


Users and Roles
Integrate authentication with LDAP and Active Directory.

LDAP, AD Splunk Flexible Roles Capabilities & Filters
Users and Groups Manage
Indexes
Share
Searches Save
Searches
Problem Investigation Problem Investigation Problem Investigation
Manage
Users
NOT
tag=PCI
App=ERP
…
Map LDAP & AD groups to flexible Splunk roles. Define any search as a filter.

Maps Roles to Apps
Options for Logical Data Partitioning
• Map Roles to Indexes
• Map Roles to Sourcetypes

Once your Apps are built, mapping Apps to Roles is easy:


Other User Control Methods

Edit Permissions by App in the App Manager

Remove options for users in the AccountBar

<module name="AccountBar" layoutPanel="appHeader">
<param name="mode">lite</param>
</module>


Splunk By Example
Basic UI Customization

*Disclaimer
The App and Splunk Web customizations that we will show in this
presentation are by no means comprehensive
Please see the Developer Manual for more detail

http://www.splunk.com/base/Documentation/latest/Developer/


Launch Page
In the demo we saw a
slide as a landing page
instead of a dashboard.
How did you do that?
Screenshot here


Where Stuff Lives
$SPLUNK_HOME/etc/apps/your_app_name
$SPLUNK_HOME/etc/apps/your_app_name/default
– Put all Splunk configuration files for you App here
$SPLUNK_HOME/etc/apps/your_app_name/default/data/ui/nav
– Contains default.xml, defines navigation menus
$SPLUNK_HOME/etc/apps/your_app_name/default/data/ui/views
– All dashboard and view xml files live here
$SPLUNK_HOME/etc/apps/your_app_name/default/appserver
– Add images, CSS, or HTML


Where Stuff Lives, Cont.
$SPLUNK_HOME/etc/apps/your_app_name/bin
– Custom scripts or executables for your App
$SPLUNK_HOME/etc/apps/your_app_name/local
– For users and admins to edit default configurations
$SPLUNK_HOME/etc/apps/your_app_name/metadata
– Stores object permissions


Working Backward - Navs
default/data/ui/nav/default.xml
<nav>
<collection label="About">
<view name="arch" default="true" />
<divider />
<a href="mailto:feedback@mycompany.com">Send Feedback...</a>
</collection>
<collection label="IT Operations">
<view name="operational_visibility" />
<view name="env_state" />…


The “arch.xml” View
default/data/ui/views/arch.xml
<?xml version='1.0' encoding='utf-8'?>
<dashboard>
<label>Architecture</label>
<row>
<html>
<h1>Welcome to the Application Management Demo</h1>
<h3>
Link when clicked
This Splunk instance is capturing data from a variety of applications, operating systems and network
devices.
</h3> Image to display
<center>
<a href="operational_visibility">
<img src="/static/app/appmgmt/arch.gif"/>
</a> …

Manage/Edit Your Navs and Views in Splunk Web


Other Customization Options
App Icon – create your own icon to show on Home screen
– $SPLUNK_HOME/etc/apps/your_app_name/appserver/static/appIcon.png
Custom Cascading Style Sheets (CSS) - background colors, fonts,
logos, buttons, navigation, menus, etc
– Default:
$SPLUNK_HOME/share/splunk/search_mrsparkle/exposed/css/skins/defau
lt/default.css
– Create your application.css in
$SPLUNK_HOME/etc/app/your_app_name/appserver/static
– See Splunk Apps (Splunk for *NIX, Splunk for Windows, etc) for examples


Dashboards and Views
Simplified XML
– Use the Visual Dashboard editor to
auto-generate simplified XML
– Adjust panel layouts
– Modify underlying searches and
format charts via Report Builder or
edit by hand
<dashboard>
<label>My dashboard</label>
<row>
<chart>
<searchName>My saved report</searchName>
<option name="charting.chart">line</option>
<option name="height">200</option>
</chart>
</row>
</dashboard>


Dashboards and Views
Advanced XML
– Provides extended customization and control over simplified XML
– All pages and dashboards in Splunk are composed of one or many
modules
– Each module contains params that control module specific
configurations
Check out available modules
– http://localhost:8000/modules

View any Splunk page in Advanced XML
– http://localhost:8000/en-US/app/<app_name>/<dashboard_name>?showsource=true


Operational Visibility – Current Capacity

Uses a real-time search
Sets thresholds with a filler
gauge
Includes some text
Rotated horizontally
Displays as a percentage


Building a Real-Time Search/Report

1.
From the CPU
sourcetype Extract column
headings to fields, only Send pctSystem to
Take the most return the “pctSystem” gauge command setting
recent event field ranges

2.
Set time range, Real- 3.
time, 30 second
window


Command Reference
sourcetype=cpu | head 1 | multikv fields pctSystem | gauge pctSystem 0 25 50 75 100

Command Usage Example
head head <N> - return N events (default is 10) | head 500
head <eval-expression> - a valid eval | head (foo>80)
expression

multikv multikv fields <field-list> - extracts field | multikv fields foo
values from table-formatted events and filters
out from extracted events fields not in the
given field list

gauge gauge <field> - transforms results into a | gauge foo 0 25 50 75 100
format for display in Gauge chart types and
sets gauge regions


Building a Real-Time Search/Report

4. 5.


Edit Using Advanced XML
<module name="StaticContentSample" layoutPanel="panel_row1_col3"> Dashboard position
<param name="text">Calculated based on maximum architecture throughput and capacity.</param>
</module>
<module name="HiddenSavedSearch" layoutPanel="panel_row1_col3" group=”Current Capacity" autoRun="True"> Text
<param name="savedSearch">AppMgmt - Capacity - Filler Gauge</param>
<param name="groupLabel">Current Capacity</param>
<module name="ViewstateAdapter"> Panel Title
<param name="savedSearch">AppMgmt - Capacity - Filler Gauge</param>
<module name="HiddenFieldPicker">
<param name="strictMode">True</param> Saved Search Name
<module name="JobProgressIndicator">
<module name="EnablePreview">
<param name="enable">True</param>
<param name="display">False</param>
Display as percentage
<module name="HiddenChartFormatter">
<param name="charting.chart.usePercentageValue">true</param>
<param name="charting.chart.orientation">x</param>
<module name="FlashChart">
Rotate Horizontally
<param name="width">100%</param>
<module name="ConvertToDrilldownSearch">
<module name="ViewRedirector">
<param name="viewTarget">flashtimeline</param>…..


Add a Single Value Button

Construct your search On our web logs, pull
sourcetype="access_combined" action="purchase" out the “purchase
Use the “stats” command to events
| stats avg(price) as price
take an average of a field
| eval basketvalue = "$" + tostring(round(price,2), "commas")
| rangemap field=basketvalue severe=0-50 elevated=50-75 default=low
Use “rangemap” to set
thresholds Format for display in button


Command Reference
sourcetype="access_combined" action="purchase” | stats avg(price) as price | eval basketvalue = "$" + tostring(round(price,2),
"commas") | rangemap field=basketvalue severe=0-50 elevated=50-75 default=low


stats stats (stats-function(field) [as field]) + [by | stats avg(foo) as foobar by bar
field-list] – provides statistics grouped
optionally by field

stats functions - avg() , c() , count() , dc() , distinct_count() , first() , last() , list() , max() , median() , min() , mode() , p<in>() , perc<int>() ,
per_day() , per_hour() , per_minute() , per_second() , range() , stdev() , stdevp() , sum() , sumsq() , values() , var() , varp()


Command Reference


eval eval eval-field=eval-expression – calculates | eval foo = if(bar==200, “OK”,
an expression and put the result in a field “Error”)

eval functions - abs(), case(), ceil() , ceiling(), cidrmatch(), coalesce(), commands(), exact(), exp(), floor(), if(), ifnull(), isbool(), isint(),
isnotnull(), isnull(), isnum(), isstr(), len(), like(), ln(), log(), lower(), ltrim(), match(), max(), md5(), min(), mvappend(), mvcount(),
mvindex(), mvfilter(), mvjoin(), now(), null(), nullif(), pi(), pow(), random(), relative_time(), replace(), round(), rtrim(), searchmatch(),
split(), sqrt(), strftime(), strptime(), substr(), time(), tonumber(), tostring(), trim(), typeof(), upper(), urldecode(), validate()


Command Reference


rangemap rangemap field=<string> | rangemap field=foo low=0-10
(<attrn>=<number>-<number>) + elevated=11-20 severe=21-30
[default=<string>] – sets range field to the default=low
name of the ranges that match

Note: Splunk ships with CSS that defines colors for low, elevated, and severe. You can customize CSS for
these values.


Embed Search and Time-range in View
<module name="StaticContentSample" layoutPanel="panel_row1_col2">
<param name="text">Real-time average shopping cart values.</param>
</module>
<module name="HiddenSearch" layoutPanel="panel_row1_col2" group="RT Shopping Cart Value" autoRun="True">
<param name="search">sourcetype="access_combined" action="purchase" | stats avg(price) as price | eval basketvalue = "$" +
tostring(round(price,2), "commas") | rangemap field=basketvalue severe=0-50 elevated=50-75 default=low</param>
<param name="groupLabel">RT Shopping Cart Value</param>
<param name="earliest">rt-1m</param>
<param name="latest">rt</param>
<module name="ViewstateAdapter">
Our Search
<module name="HiddenFieldPicker"> Real-time 1 minute
<param name="strictMode">True</param>
<module name="JobProgressIndicator"> window
<module name="SingleValue">
<param name="field">basketvalue</param>
<param name="classField">range</param>.....

Use the “basketvalue”
field


Build a Stacked Column Chart

Uses a lookup table to map a Product ID to a Product Name
Uses “timechart” command to display results over time


Use a Lookup Table

• In props.conf • In lookup directory, prod_lookup.csv
[access_combined] product_id,product_name,price,tdf_price,call_
LOOKUP-prod = prod_id_lookup flwrs_price
product_id OUTPUT product_name, RP-LI-02,Chocolate Dreams
price, tdf_price, call_flwrs_price Confections,379,299,319

• In transforms.conf
[prod_id_lookup]
filename = prod_lookup.csv

Construct Your Search and Format

• Either define in Report wizard • Or in view XML

<module name="HiddenChartFormatter">
<param name="chart">column</param>
<param name=“stackMode”>stacked</param>


Command Reference
sourcetype="access_combined” | timechart count by product_name


timechart timechart [agg=<stats-agg-term>] | timechart span=1m avg(foo) by
[<bucketing-option> ]* (<single-agg> [by bar
<split-by-clause>] )

timechart functions (same as stats) - avg() , c() , count() , dc() , distinct_count() , first() , last() , list() , max() , median() , min() , mode() ,
p<in>() , perc<int>() , per_day() , per_hour() , per_minute() , per_second() , range() , stdev() , stdevp() , sum() , sumsq() , values() , var() ,
varp()


Building a Correlation Search

Uses a simple Boolean search to get results from both sourcetypes
Uses a search macro to process results and display


Using a Search Macro
Define in Manager > Advanced search > Search macros
Reuse chunks of searches as part of your search string like a function

Invoke using the left quote character


Raw Event Output

Contain asynchronous entries
Possible duplicates


Step 1
sourcetype=mysql_config OR sourcetype=remedy_changeticket
| dedup _raw, User
| transaction TicketId, User
| eval hasTicket = if(eventcount > 1, "Yes", "No")
| rename PrevPropValue as "Original_Value", NewPropValue as "New_Value", hasTicket as
"Change_Ticket"
| fields _time, User, Property, "Original_Value", "New_Value", "Change_Ticket"

Get rid of duplicate events or duplicate users


Step 2
| dedup _raw, User
"Change_Ticket"

Combine two asynchronous events into one event using transaction
command
– Note: can also define boundaries for length of events (maxspan) or time between
events (maxpause)
Use the same “TicketId” and “User” fields between the two events to group
Will produce metadata field “duration” and “eventcount”


Step 3
| dedup _raw, User
"Change_Ticket"

Use eval to create a new field “hasTicket”
Use “eventcount” field generated by transaction command
In the results, if the uber-event has > 1 event then there is an change
with an associated change ticket, otherwise “hasTicket” is “No”


Step 4
| dedup _raw, User
"Change_Ticket"

Formatting and results
Rename a few fields for clarity in dashboard results
Use fields command to only display what we need to see


Support Through the Splunk Community
Splunkbase


Where to Go for Help
Documentation
– http://www.splunk.com/base/Documentation
Technical Support
– http://www.splunk.com/support
Videos
– http://www.splunk.com/videos
Education
– http://www.splunk.com/goto/education
Professional Services


Technical Workshops
Advanced User Training

Thank you

SplunkLive 2011 Advanced Session

More Related Content

What's hot (20)

Viewers also liked (18)

Similar to SplunkLive 2011 Advanced Session (20)

More from Splunk (20)

Recently uploaded (20)

SplunkLive 2011 Advanced Session

Editor's Notes