Log Analytics for Distributed Microservices

Kai Wähner
Technology Evangelist
kontakt@kai-waehner.de
LinkedIn
@KaiWaehner
www.kai-waehner.de
O’Reilly Software Architecture Conference 2016 (London, UK)
Log Analytics for Distributed Microservices

© Copyright 2000-2016 TIBCO Software Inc.
Can you answer to these questions?
• Are you storing all of your logs for enough time to answer the question “What
happened?” a week from now? How about a year from now?
• Can you issue a single search across all your machine data - regardless of source or
type?
• Can you set an alert that would trigger from any source in your enterprise?
• Do you analyze and correlate all events in your distributed microservice architecture?
• What about predictive monitoring?

Key Takeaways
• Log Analytics is needed to monitor distributed microservice architectures
• Consolidation of broad range of events is key to enabling business insights
• Log Analytics is complementary to other Big Data components

Agenda
• Distributed Microservice Log Events
• Introduction to Log Analytics
• Market Overview
• Relation to other Big Data Components

Scenarios for Distributed Log Events
Infrastructure
• Log Management
– Applications
– SOA
– Microservices
– Cloud Platforms
– SaaS
• Transaction Tracing
• Root Cause Analysis
• Visual Analytics on Machine Data
Competitive Undermining
• Filtering / Cost Avoidance Solution
IT Operations
• Troubleshooting Connectivity
• Outage Troubleshooting
• Application Monitoring / Tracking
• Service Level Confirmation for IT Outsourcing
Security
• Centralized Log / Event Management Platform
• Security
• Fraud Detection
Compliance
• PCI Compliance
• Retention Compliance
• Service Level Confirmation for IT Outsourcing

Distributed Microservice Architecture
http://blogs.gartner.com/gary-olliffe/2015/01/30/microservices-guts-on-the-outside/
”That complexity
has moved and […]
increased [to] the
outer architecture.”

12 Factor Apps for Cloud Native Microservices
Codebase
One codebase
tracked in
revision control,
many deploys.
Dependencies
Explicitly declare
and isolate
dependencies.
Config
Store config in
the environment.
Backing
Services
Treat backing
services as
attached
resources.
Build, Release,
Run
Strictly separate
build and run
stages.
Processes
Execute the app
as one or more
stateless
processes.
Port Binding
Export services
via port binding.
Concurrency
Scale out via the
process model.
Disposability
Maximize
robustness with
fast startup and
graceful
shutdown.
Dev / Prod
Parity
Keep dev,
staging, and
prod as similar as
possible.
Logs
Treat logs as
event streams.
Admin
Processes
Run
admin/mgmt
tasks as one-off
processes.
https://12factor.net/

Some Cloud Platforms (PaaS) with Support for 12 Factor Apps
With or without such a cloud platform,
you need a way to aggregate and analyze
distributed microservice logs.
… to treat logs as event streams.
! !

Distributed Microservice Architecture
http://blogs.gartner.com/gary-olliffe/2015/01/30/microservices-guts-on-the-outside/
Microservices means…
- distributed services
- distributed infrastructure
- different technologies
- containers and cloud platforms
- distributed log messages
- unstructured / semi-structured data
Log Analytics

Operational Intelligence Platform for Log Analytics
Log Analytics Platform
ü Centralize and Store of Record
ü Search, Auto-id, Parsing, Correlation
ü Forensics and Alerts
ü Reports
EngineLogs
ApplicationLogs
Microservices
Monitoring
Configuration
Messaging
Web
UI
API
Analysis
Tools
DataDiscovery
StreamingAnalytics
LiveVisualization

How an Operation Intelligence Platform Works
INGEST OPERATIONALIZE ANALYZE
Collect Data from
Any Source
Device Logs
Web Logs
Application & DB Logs
Configuration Files
OS Metrics
Sensor Data
Microservice Events
Make Unstructured
Data Usable
Normalize
Enrich
Transform
Index
Aggregate
Gain Actionable
Insight
Search
Report
Alert
Correlate
Visualize

Log Analytics Example
• May 2 23:06:14 app-1 login[5130]: pam_unix(login:auth): authentication failure;
logname=LOGIN uid=0 euid=0 tty=tty1 ruser= rhost= user=timothy
• "<13>Feb 5 08:34:55 10.92.2.188 MSWinEventLog 0 Security 106236353 Fri
Feb 05 08:33:15 2010 529 Security SYSTEM User Failure Audit
OHAEPHQDC009 Logon/Logoff Logon Failure: Reason: Unknown
user name or bad password User Name: timothy Domain: Logon Type: 3
Logon Process: CISCO Authentication Package:
MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Workstation Name: CISCO Caller User
Name: portal Caller Domain: CORP Caller Logon ID: (0x0,0x63194519)
Caller Process ID: 2972 Transited Services: - Source Network Address: -
Source Port: - 1679136992"
• Jun 11 10:51:04 10.0.0.244 Jun 11 10: 51:42 1,06/11
10:51:42,0001a100200,TRAFFIC,start,24,06/11
10:51:15,10.0.0.101,10.0.0.246,0.0.0.0,0.0.0.0,timothy,,,dns,vsys1,l2-lan-trust,l2-
lan-untrust,ethernet1/12,ethernet1/11,Forward to Timothy,06/11
10:51:42,2074963,1,54604,53,0,0,0x0,udp,allow,80,80,80,1,06/11 10:51:16,0,any,0
Source
Type
User User Name SRC User
Unix timothy
Windows timothy
Firewall timothy
Source Type User
Unix timothy
Windows timothy
Firewall timothy
• Unix
• Windows
• Firewall

Characteristics of Log Management Solutions
Data Sources
• Log information (standard protocols like TCP, UDP, File, Syslog, JMS)
• All events (logs, messaging, streams, ...)
• Extendable plugins (connectors, SDK, API)
Features
• Collect, parse, correlate, search, report, forward, etc.
• Store and index
• Query Language (SQL, Custom) à sliding windows, correlations, etc.
• Retention
• Compliance Templates
Frequency
• Historical data
• Near Real Time Processing (seconds or minutes)
Deployment Options
• On-premise vs. Cloud (SaaS)
• Open Source vs. Commercial
• Software vs. Hardware Appliance
Pricing
• Free (open source) vs. CPU-based vs. Volume-based
à Be careful here: IoT... Data grows exponentially

Market Analysis
Segment CAGR Incumbents Challengers
Log Management
15%
Splunk, TIBCO LogLogic,
etc.
Open Source (Graylog, “ELK Stack”)
SIEM RSA, ArcSight, LogRhythm Splunk, MSSPs (Managed Security Service Provider)
ITOA (1.6B) 100%
TIBCO LogLogic, Splunk, SumoLogic, AppDynamics,
NewRelic
APM (2.9B) 10% AppDynamics, NewRelic
ITOM (19B) 4% IBM, CA, BMC, MS, HP
AppDynamics, NewRelic, Chef, Puppet, Docker,
CloudFoundry
Rapidly Emerging and Evolving, Encompasses Many Segments
Traditional: Log Management, IT Operations Monitoring (ITOM), Security (SIEM)
Current: IT Operations Analytics (ITOA), Application Performance Management (APM)
Future: DevOps and Continuous Improvement
(2.9B)

Security information and event management (SIEM)
SIEM is a specific part of Log Analytics
focusing on Security.
• Threat management: Early detection of
targeted attacks and data breaches
• Compliance: Collect, store, analyze and
report on log data for incident response,
forensics and regulatory compliance
• Aggregates event data produced by
security devices, network infrastructures,
systems and applications
Log Analytics handles all kinds of use cases,
not focusing on security.
http://www.gartner.com/document/3097022
https://www-01.ibm.com/marketing/iwm/dre/signup?source=swg-WW_Security_Organic&S_PKG=ov37658&cm_mmc=Blog_SI-_-Sec_Int-_-Organic-_-IBM-is-a-leader-again-in-2015-gartner-magic-quadrant-for-SIEM
SIEM is out-of-scope for this presentation!

Alternatives for Log Analytics
Time
to
Market
Log Analytics
Product
Middleware Suite
(includes Log Analytics Product)
Slow Fast
Log Analytics
Framework
IncludesIncludes

Alternatives for Log Management (no complete list)
Open Source Closed Source
SaaS
On Premise

“Cloud washing is the purposeful and sometimes deceptive
attempt by a vendor to rebrand an old product or service by
associating the buzzword ‘cloud’ with it [and offering it via a
public cloud infrastructure].”
On Premise vs. Cloud Washing vs. Cloud Native / SaaS
http://searchcloudstorage.techtarget.com/definition/cloud-washing
!

SaaS
On Premise
Open Source Framework

Time
to
Market
Log Analytics
Product
Middleware Suite
(includes Log
Analytics Product)
Slow Fast
Log Analytics
Framework
Library (Java, .NET, Python)
Operators (Collect, Filter, Sort, Aggregate, Alert)
Scalability (Horizontal and Vertical, Fail Over)
Connectivity (Standards, Technologies, Products)
User Interface (Basic Monitoring and Reporting)

ELK Stack (Logstash, Elasticsearch, Kibana)
Characteristics
• Data Sources
• Features
• Frequency
• Deployment Options
• Pricing
Facts
Combination of Open Source Frameworks
• Quick getting started for developers with ”Hello World” example
• More complex Enterprise setup and usage (coding and configuration)
• AWS offering available for Elastic and Kibana, not Logstash)
Targeted for developers
• Mainly focused on helping developers detect and fix errors in their apps
• Entirely open source, i.e. free to use
• Commercial support available
• Combination of different mature frameworks
Less enterprise-focused
• Very basic user interface
• Based on ElasticSearch, Logstash and Kibana
• Plenty of connectors + easy to extend (with coding)
• Sufficient reporting (i.e. dashboards), but missing visual analytics

Live Demo
ELK Stack (Open Source) in Action…

graylog
Characteristics
• Data Sources
• Features
• Frequency
• Pricing
Facts
Combination of Open Source Frameworks
• Quick getting started for developers with ”Hello World” example
• More complex Enterprise setup and usage (coding and configuration)
Targeted for developers
• Mainly focused on helping developers detect and fix errors in their apps
• Entirely open source, i.e. free to use
• Commercial support available
• Young solution (1.0 GA in 2015) – not as mature as others yet
• Very basic user interface
• Based on MongoDB, ElasticSearch and Apache Kafka
• Marketplace for connectors + easy to extend (with coding)
• Missing extensive reporting and analytics

SaaS
On Premise
SaaS Cloud Service

papertrail
Facts
Easy setup and very simple to use
• Targeted for developers
• „Very small“ free version available (100MB/month)
• Cheap pricing, e.g. 1GB/month: 5 USD; 1000GB/month: 875 USD
• Stripped down and basic log analyzer
• Mostly text-based
• User interface is very similar to looking at a log on your machine
• No advanced integrations, predictive or reporting capabilities
SaaS
• Upload (masses of) data to the cloud
• Worse latency than on-premise solutions
• Efforts to anonymize sensitive data
Characteristics
• Data Sources
• Features
• Frequency
• Pricing

Live Demo
Papertrail (SaaS) in Action…

loggly
30
Facts
Easy setup and very simple to use
• Custom performance and DevOps dashboards
Targeted for developers and DevOps
• Pricing from 50 USD to some thousand USD
• Feature-limited free version available (200MB/day)
• Focus especially on logs from application servers
• Anything beyond that has to be built
• Find and fix operational problems
• Primary use cases are for troubleshooting / customer support scenarios
SaaS
• Upload (masses of) data to the cloud
• Worse latency than on-premise solutions
• Efforts to anonymize sensitive data
Characteristics
• Data Sources
• Features
• Frequency
• Pricing

Time
to
Market
Log Analytics
Product
Middleware Suite
(includes Log
Analytics Product)
Slow Fast
Log Analytics
Framework
Library
Operators
Scalability
Connectivity
User Interface
Visual Configuration (Analysis, Correlation, Alerting)
Simulation (Feed Testing, Test Generation)
User Interface (Advanced Monitoring, Reporting, Analytics)
Maturity (product, 24h support, consulting)

sumologic
Characteristics
• Data Sources
• Features
• Frequency
• Pricing
Facts
• Easy setup and simple to use
• Targeted for developer, security teams, business
– Pricing from 90 USD to some thousand USD
– Feature-limited free version available (500MB/day)
• Most enterprise-focused SaaS product
– Founded as „Splunk for the Cloud“
– Most feature-rich SaaS solution
– Many features of „enterprise grade solutions“
• SaaS
– Upload (masses of) data to the cloud
– Worse latency than on-premise solutions
– Efforts to anonymize sensitive data

SaaS
On Premise
Enterprise Product

Splunk
Characteristics
• Data Sources
• Features
• Frequency
• Pricing
Facts
• Complex setup (especially for larger scale)
– SaaS Offering for getting started quickly in the public cloud
• Simple to use for the end user
• Targeted for all use cases (including SIEM)
– Not just for log files, but also other events / messaging
– „Enterprise Pricing“ - Very High pricing (for medium and high volume)
– No access to your data if limit is reached! (contrary to other vendors)
• Enterprise Class
– Market leader
– Most feature-rich solution
– Moving into ITOA market
– No hardware appliance (just via partner „SBOX“)
– Just log analytics, no complete middleware suite

Time
to
Market
Log Analytics
Product
Middleware Suite
(includes Log
Analytics Product)
Slow Fast
Log Analytics
Framework
Library
Operators
Scalability
Connectivity
User Interface
Visual Configuration
Simulation
Advanced User Interface
Maturity
Out-of-the-Box Integration and Support
(Messaging, ESB, MDM, etc.)

IBM QRadar
Characteristics
• Data Sources
• Features
• Frequency
• Pricing
Facts
• Complex setup
• Targeted for all use cases (including SIEM)
– „Enterprise Pricing“ - High pricing (for medium and high volume)
– Part of a complete middleware suite
– Very feature-rich solution
– Available as SaaS offering
– Available as hardware appliance
– Moving into ITOA market

TIBCO LogLogic
37
© Copyright 2000-2015 TIBCO
Characteristics
• Data Sources
• Features
• Frequency
• Pricing
Facts
• Easy setup (small and large scale)
– Powerful user interface
– Not as powerful as Splunk or IBM QRadar
• Targeted for all use cases
– „Enterprise Pricing“ - Low costs compared to competitors
– „Always on“ – even after limit is reached
– Part of a complete middleware suite
– Most advanced analytics (via TIBCO Spotfire add-on)
– Available as hardware appliance

Live Demo
TIBCO LogLogic (Enterprise) in Action…

Message Pattern Generation with TIBCO LogLogic Web UI
Discover Unstructured Data à Generate Pattern à Validate à Apply Pattern for Structured Data

Spoilt for Choice
Does it make sense
to combine different
Log Analytics
solutions?

Example: TIBCO LogLogic à „A Splunk Management Solution“
http://www.tibco.de/assets/blt0da0bc2ea7d5b9b7/solution-brief-tibco-loglogic-splunk-management-solution.pdf

Conclusion - Market Analysis
Log Management
• SaaS à Easy to setup and use, but cloud cons (not flexible, public cloud)
• Open Source à Free and extendable, but coding / config instead of tooling
• Enterprise à Most feature-rich and powerful tooling, but more expensive
IT Operations Analytics (ITOA)
• Enterprise vendors entering this market these days
à Extending existing solutions
• Focus on more complex correlations, real time processing, predictive monitoring

Market Analysis
Segment
CAG
R
Incumbents Challengers
Log Management
15%
Splunk, TIBCO LogLogic,
etc.
Open Source (Graylog, “ELK Stack”)
SIEM RSA, ArcSight, LogRhythm Splunk, MSSPs (Managed Security Service Provider)
ITOA (1.6B) 100%
TIBCO LogLogic, Splunk, SumoLogic, AppDynamics,
NewRelic
APM (2.9B) 10% AppDynamics, NewRelic
ITOM (19B) 4% IBM, CA, BMC, MS, HP
AppDynamics, NewRelic, Chef, Puppet, Docker,
CloudFoundry
(2.9B)
Rapidly Emerging and Evolving, Encompasses Many Segments
Traditional: Log Management, IT Operations Monitoring (ITOM), Security (SIEM)
Current: IT Operations Analytics (ITOA), Application Performance Management (APM)
Future: DevOps & Continuous Improvement

Log Analytics is a
very stable and established market.
ITOA enhances Log Analytics
to allow more powerful real time correlation.

When to use Log Analytics
Time of
Action
Historical
Data
Near
Real Time
Real Time Predictive
IT Operations Analytics (ITOA)
Log Management
Data Warehouse Streaming Analytics
Data Discovery
„Data Lake“ (Various Apache Hadoop Frameworks)
Log Analytics
Visual Real Time Analytics
(There is some overlapping!)

Streaming Analytics: Act on Critical Business Moments

Streaming Analytics
Voltage
Temperature
Vibration
Device
history
Temporal analytic: “If vibration spike is followed by temp spike then
voltage spike [within 4 hours] then flag high severity alert.”

Live Demo
Apache Flink (Open Source), StreamSets (SaaS) and TIBCO StreamBase / Live Datamart (Enterprise) in Action…

Log Management / ITOA vs. Hadoop and Log Collectors
Why not use just a Data Lake (Apache Hadoop)? You can also store and analyze all data on its cluster!
Why not just use Log Collectors and forward data directly without Log Analytics “in the middle”?
• In general: Fluentd, Logstash
• Apache Hadoop specific: Apache Flume or Apache Kafka
DIFFERENTIATORS OF LOG MANAGEMENT / IT OPERATIONS ANALYTICS
• Integrated solution for data analysis (tooling, consulting, support)
• Built exactly for these use cases (Log Management, ITOA)
• Involves data indexing, data processing (querying) and data visualization by means of dashboards and other tools
• Tooling for Ease-of-Use and Time-to-Market
• Graphical user interface for operational intelligence
• There is no “one size fits all” tool to solve all your problems

Relation to other Big Data Components
• Data Warehouse
– Historical data
– Only structured data
– Reporting
• Apache Hadoop
– Historical and near real time data
– All data
– Storage and Analytics (e.g. MapReduce, Spark)
• NoSQL
– Specific Storage (graph, document, key/value, ...)
– Search (e.g. ElasticSearch)
• Stream Processing
– Especially real time data
• Predictive Analytics
– R, Machine Learning, SAS, etc.
– Combined with the others!
Log
Analytics
Forward
Forward
Parse, Filter, Structure, Forward

Trend: Machine Learning applied to Log Analytics
“… when the log-data patterns cannot be precisely defined in advance, unsupervised and
reinforcement learning may be appropriate [to find outliers or anomalies].”
http://www.infoworld.com/article/2608064/big-data/big-data-log-analysis-thrives-on-machine-learning.html
“… They combined the aggregation of log data, the
metadata that is created any time IT systems are used,
along with high-level analytics and machine learning tools
…
… give context to the ’needle in a haystack’ problem …”
http://www.forbes.com/sites/benkepes/2015/03/27/using-log-data-and-machine-learning-to-weed-out-the-bad-
guys

Questions? Please contact me!
Kai Waehner
Technology Evangelist
kontakt@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
LinkedIn

Log Analytics for Distributed Microservices

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Log Analytics for Distributed Microservices (20)

More from Kai Wähner (20)

Recently uploaded (20)

Log Analytics for Distributed Microservices