Visualizing HPCC Systems Log Data Using ELK

Innovation and
Reinvention Driving
Transformation
OCTOBER 9, 2018
2018 HPCC Systems® Community
Day
Rodrigo Pastrana & Miguel Vazquez
Visualizing HPCC Systems Log Data Using
ELK

Who are we?
Visualizing HPCC Systems Log Data Using ELK 3
Rodrigo Pastrana
Architect
HPCC Systems
Miguel Vazquez
Consulting SWE
HPCC Systems

Why visualize HPCC Systems log data?
• Component logs contain a
wealth of raw information
• This data can be used for
debugging, profiling, billing,
accounting, analyzing, etc.
• These actions are difficult to do
with raw log data
• Visualizing this data can help
you find the needle in the
haystack

A visualization is worth a thousand (or more) log entries…

Discussion Topics
• What is ELK (and why)
• HPCC Systems Log
Details
• ELK Topology and other
Considerations
• Sample ELK Topology
• Sample ESP
Transaction Info
Processing
• ELK Component
Configuration
• Demo Visualization of
ESP Transaction data

Elasticsearch Logstash Kibana (and Beats) – Elastic Stack
• Powerful, flexible Open-source stack for log analytics from Elastic
• Arguably the de-facto standard for log processing and analytics
Components
• Beats: Light-weight, single purpose log data shippers
• Logstash: Ingests data from a number of different sources,
parse, filter, mutate, and stashes it.
• Elasticsearch: Search and analytics engine - acts as a data store.
• Kibana: Visualization front end

HPCC Systems Component Logs
• Important to understand HPCC Systems log basics!
• Logs generated for major HPCC Systems components
• ESP, ROXIE, THOR, DALI, Sasha, DafileServ, DFUServer, ECLAgent, etc.
• Log files are time-stamped and rolled daily;
• Running log link provided
• Default location: /var/log/HPCCSystems/<componentname>
• Default log message format (can be edited in configuration):
• SequenceNumber, Date-TimeStamp, ProcessId, ThreadId, QuotedMessage
• QuotedMessage – Contains actual log message
• Aggregation of other fields – Uniquely identify log message instance

ELK Implementation - Design Considerations
• Several considerations when setting up ELK system to process log data
• What are we doing with the data
• Troubleshooting? Monitoring? Analytics? Reporting?
• Type and volume of data
• Data sensitivity
• Resources required
• Security
• Compliance
• Disaster recovery
• Data backup and others

Our approach (to illustrate the point)
• We will target ESP transactions and ROXIE completed queries
• ESP logs conveniently provide transaction summary log entries
• Roxie provides similar query complete summary log entries
• Focuses on most important info, minimize amount of data processed through
ELK
• This also prevents us from exposing sensitive data (PII, SPII nor business
data)
• We will target a remote ELK cluster
• Processing small subset of logs
• No sensitive data!
• Minimal resource contention with HPCC Systems components

ROXI
E
ROXI
E
ESP
…
Overall Topology

Metricbeats - Monitor cluster system health (optional)

Metricbeats - Monitor node system health (optional)

Setup Filebeat to forward HPCC Systems log entries
• Filebeat primarily responsible for tailing files, filtering, and multi-line stitching
• Filebeat configuration in filebeat.yml
• Declare filebeat “prospector” for each log message type to be forwarded
• Set prospector to target running component log file
• In case of HPCC component ESP labeled “myesp”
• /var/log/HPCCSystems/myesp/esp.log
• Define custom field “component” set to component type (ESP, ROXIE,etc.).
• “component” field subsequently used by Logstash
• Declare regex patterns to include or exclude message types
• Include_lines: standard component log columns + “TxSummary …”
• Declare regex patterns to handle multi-line TxSummary messages as single
entity

FileBeat ESP prospector
filebeat.prospectors:
type: log
enabled: true
paths:
- /var/log/HPCCSystems/myesp/esp.log
fields:
component: esp
fields_under_root: true
encoding: plain
include_lines:
['([A-Z-0-9]{8})s+(d{4}-d{2}-d{2}sd{2}:d{2}:d{2}.d{3})s+(d+)s+(d+)s+("TxSummary)']
multiline.pattern:
([A-Z-0-9]{8})s+(d{4}-d{2}-d{2}sd{2}:d{2}:d{2}.d{3})s+(d+)s+(d+)s+(")
multiline.negate: true
multiline.match: after
Targeting current log file
Custom field declaration
Rule treats multi-line messages as single
entity
RegEx describes default log entry with
Quoted message with a leading
“TxSummary”
Default ‘myesp’ ESP log
location
FileBeat.yml

FileBeat.yml details continued…
• Prospector definitions for other components very similar
• Target appropriate component log file
• Create custom “component” field, populate with appropriate label
• Regex to include or exclude entry types of interest
• Configure Filebeat to forward the filtered log entries to remote Logstash instance
• Logstash assumed to listen for these messages on particular address(es)
and ports, (X.Y.Z.W:5044 for this example):
#----------------------------- Logstash output -----------------
----
output.logstash:
# The Logstash hosts
hosts: ["X.Y.Z.W:5044"]
FileBeat.yml

Process HPCC Systems log messages through Logstash
Capture
HPCC
Systems
log
messages
Parse
Structure
Filter
Mutate
Create and
populate
ElasticSearch
indexes
ROXI
E
ROXI
E
ESP
…

Logstash Input setup
• Set up the Logstash input mechanism
• Many options (stdin, s3, redis, pipe, file, log4j, kafka, etc.)
• Let’s enable input for filebeats messages via port 5044
input
{
beats { port => 5044
}
}
Logstash.con
f

Setup Logstash to process ESP transactions
Filter{
if [component] == "esp"
{
grok {match => { "message" =>
"%{BASE16NUM:sequence}
%{TIMESTAMP_ISO8601:logtimestamp}
+%{INT:processID} +%{INT:threadID}
%{QUOTEDSTRING:logmessage}" }}
kv { source => "logmessage" field_split => "[;" value_split => "=" }
Logstash.con
f
Rule for ESP based
messages
Capture messages with known log
format
Expected sequence of space delimited fields
(fieldtype:fieldname)
Parses string field “logmessage” into key value pairs
"TxSummary[activeReqs=4;rcv=1ms;user=ausr@127.0.0.1;req=POST wssmc.ACTIVITY v1.2;total=45ms;]"

Setup Logstash to process ESP transactions (continued…)
mutate {
remove_field => [ "message", "@version" ]
rename =>
{
"total" => "TotalTrxmS”
"rcv" => "TimeReceived”
}
convert =>
{
"threadID" => "integer”
"processID" => "integer”
}}}
else if [component] == “roxie“
Logstash.con
f
Important to filter out noise
Assign meaningful field names
Assign field type for aggregation
purpose
Create similar rules for other log message
types

Setup Logstash to process ESP transactions (output to ES)
output
{
if [component] == "esp"
{
elasticsearch
{
hosts => ["yourelasticsearchaddress:9200"]
index => “esp-log-%{+YYYY.MM.dd}"
}
}
else if [component] == “roxie"
{ …}
}
Logstash.con
f
Define output Logstash output rules
Forward processed messages to
ES
Important to establish appropriate indexing mechanism
Similar rules for ROXIE based
messages

Confirm EL indexes are created
http://yourkibanaip:5601/app/kibana#/dev_tools/conso
le

Let’s create some
visualizations

Discover your newly created log events

Some of Kibana’s visualization toolset

Visualization creation
• Visualize > > Select a visualization type > Enter some metrics > Save

Deeper dive into what our users are doing

Visualizations using a search query

Lets tie it up all together with a Dashboard
• Dashboard > > > Save

Available in 7.0
• Ability to embed into the ECL Watch U/I

Questions?
Useful Links
• https://hpccsystems.com/blog/ELK_visualizations
• https://www.elastic.co/guide/index.html
• https://github.com/rpastrana/hpcc-elk
• http://cdn.hpccsystems.com/releases/CE-Candidate-7.0.0/docs/EN_US/HPCCSystemAdministratorsGuide_EN_US-7.0.0-
rc2.pdf#page=26
Contact Us
• Rodrigo.Pastrana@lexisnexisrisk.com
• Miguel.Vazquez@lexisnexisrisk.com

Thank you

Visualizing HPCC Systems Log Data Using ELK

More Related Content

What's hot (20)

Similar to Visualizing HPCC Systems Log Data Using ELK (20)

More from HPCC Systems (20)

Recently uploaded (20)

Visualizing HPCC Systems Log Data Using ELK