SlideShare a Scribd company logo

From logging to monitoring to reactive insights - C Schneider

M
mfrancis

The document provides an overview of using Apache Karaf Decanter for centralized monitoring and logging with Elastic Search and Kibana. It discusses various components, including collectors, appenders, and custom reactive processing to enhance logging insights. The content also outlines event processing use cases and architecture for both single and multiple server setups.

Technology
1 of 19
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
From logging to monitoring to reactive insights Christian Schneider
Christian Schneider » For the last 6 years: Open Source Architect at Talend » Since October: Computer scientist at Adobe. » Apache member and committer » Twitter @schneider_chris » Website liquid-reality.de
What you will learn » Apache Karaf Decanter + Elastic Search + Kibana » Approaches for central monitoring and logging » Custom reactive processing to produce higher level informations
Basic logging in OSGi » Pax logging » Logback » Log4j2 Files
Monitoring Hawtio JVisualVM ● Great tools ● Developer focused ● Production?
Production logging Central History Drill down Custom processing
ELK Stack Karaf Elastic Search + Kibana Logstash Server File Beat Metric Beat
Apache Decanter Karaf + Decanter Elastic Search + Kibana
Decanter architecture Collector Event Admin Appender
Apache Decanter Collectors ● Pax-Logging ● JMX ● EventAdmin ● Camel ● ActiveMQ ● ... Events Appenders ● File ● DB ● Elastic Search ● Kafka ● JMS ● ...
Decanter setup for single machine JMX Collector Log Collector Camel Collector Event Admin Elastic Search Appender
Decanter for multiple servers Server 1 + Decanter Server 2 + Decanter Kafka Cluster Kafka Cluster Elastic Search + Kibana Decanter
Decanter + logstash Server 1 Karaf Decanter Server 2 Spring boot Beats Kafka Cluster Kafka Cluster Elastic Search + Kibana Logstash Server Server 3 Tomcat Beats
CXF Message logging Client Server Request Out Request In Response Out Response Inslf4j slf4j ● Meta data as MDC properties ● Message id and correlation id ● Service name as logger name
Extending Decanter » Collectors » Appenders » Filter, Enrich Events » “Complex” Event Processing using Reactive Components
Event Processing use case » Compute “up time” of bundles by noting start and stop events and sending an event with the bundle runtime on stop » Alert when we we have more than x percent errors in logged CXF messages » Correlate request and response into one message sent through decanter » Only send aggregated events when your source produces Events very fast
Demo time Decanter in action
Thanks for your attention Questions?
Further reading Demo code Apache Karaf decanter CXF message logging Reactive Components

More Related Content

PDF
Kibana + timelion: time series with the elastic stack
Sylvain Wallez
 
PDF
Black friday logs - Scaling Elasticsearch
Sylvain Wallez
 
PDF
Keynote -- Percona Live Europe 2018
Kevin Xu
 
PPTX
A (XPages) developers guide to Cloudant
Frank van der Linden
 
PDF
Logging in The World of DevOps
DevOps Indonesia
 
PPTX
Kibana overview
Rinat Tainov
 
PDF
Zentral QueryCon 2018
Henry Stamerjohann
 
PDF
Lambda architecture
Ivan Kosianenko
 
Kibana + timelion: time series with the elastic stack
Sylvain Wallez
 
Black friday logs - Scaling Elasticsearch
Sylvain Wallez
 
Keynote -- Percona Live Europe 2018
Kevin Xu
 
A (XPages) developers guide to Cloudant
Frank van der Linden
 
Logging in The World of DevOps
DevOps Indonesia
 
Kibana overview
Rinat Tainov
 
Zentral QueryCon 2018
Henry Stamerjohann
 
Lambda architecture
Ivan Kosianenko
 

What's hot (20)

PDF
Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elasti...
Elasticsearch
 
PDF
Data / Streaming / Microservices Platform with Devops
Kidong Lee
 
PPTX
Open source big data landscape and possible ITS applications
SoftwareMill
 
PPTX
Realtime Business Platform Architecture Review
Anant Corporation
 
PDF
Presto Summit 2018 - 08 - FINRA
kbajda
 
PPTX
Logstash, Elasticsearch and Kibana
Saroj Panyasrivanit
 
PDF
Zentrales logging mit dem Elastic Stack
SimonSchneider24
 
PPTX
Visualizing large datasets with elasticsearch and kibana
Dan Fey
 
PDF
What’s Evolving in the Elastic Stack
Elasticsearch
 
PDF
Productionize spark structured streaming
Ivan Kosianenko
 
PPTX
Elk meetup boston - logz.io
tomerlevy9
 
PPTX
Serverless
Natan Deitch
 
PDF
TiDB Introduction - San Francisco MySQL Meetup
Morgan Tocker
 
PDF
Fluentd and Docker - running fluentd within a docker container
Treasure Data, Inc.
 
PDF
Graph Processing with Apache TinkerPop and Gremlin
Jason Plurad
 
PPTX
4Developers 2018: Przetwarzanie Big Data w oparciu o architekturę Lambda na p...
PROIDEA
 
PPTX
AWS Finland meetup 2018 August
Rolf Koski
 
PDF
Logging and Monitoring your container-based infrastructures
Mohammed Aboullaite
 
PDF
New Features in MongoDB Atlas
MongoDB
 
PDF
TiDB Introduction
Morgan Tocker
 
Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elasti...
Elasticsearch
 
Data / Streaming / Microservices Platform with Devops
Kidong Lee
 
Open source big data landscape and possible ITS applications
SoftwareMill
 
Realtime Business Platform Architecture Review
Anant Corporation
 
Presto Summit 2018 - 08 - FINRA
kbajda
 
Logstash, Elasticsearch and Kibana
Saroj Panyasrivanit
 
Zentrales logging mit dem Elastic Stack
SimonSchneider24
 
Visualizing large datasets with elasticsearch and kibana
Dan Fey
 
What’s Evolving in the Elastic Stack
Elasticsearch
 
Productionize spark structured streaming
Ivan Kosianenko
 
Elk meetup boston - logz.io
tomerlevy9
 
Serverless
Natan Deitch
 
TiDB Introduction - San Francisco MySQL Meetup
Morgan Tocker
 
Fluentd and Docker - running fluentd within a docker container
Treasure Data, Inc.
 
Graph Processing with Apache TinkerPop and Gremlin
Jason Plurad
 
4Developers 2018: Przetwarzanie Big Data w oparciu o architekturę Lambda na p...
PROIDEA
 
AWS Finland meetup 2018 August
Rolf Koski
 
Logging and Monitoring your container-based infrastructures
Mohammed Aboullaite
 
New Features in MongoDB Atlas
MongoDB
 
TiDB Introduction
Morgan Tocker
 
Ad

Similar to From logging to monitoring to reactive insights - C Schneider (20)

PDF
Akka, Spark or Kafka? Selecting The Right Streaming Engine For the Job
Lightbend
 
PDF
Logging for Production Systems in The Container Era
Sadayuki Furuhashi
 
PDF
Monitoring docker, k8s and your applications with the elastic stack
SmartWave
 
PPTX
Flink September 2015 Community Update
Robert Metzger
 
PDF
Logs aggregation and analysis
Divante
 
PDF
Concepts and Patterns for Streaming Services with Kafka
QAware GmbH
 
PDF
Managing your Black Friday Logs - Antonio Bonuccelli - Codemotion Rome 2018
Codemotion
 
PDF
Analyzing Data at Scale with Apache Spark
Nicola Ferraro
 
PDF
Introduction to apache kafka, confluent and why they matter
Paolo Castagna
 
PDF
Down the event-driven road: Experiences of integrating streaming into analyti...
inovex GmbH
 
PPTX
Data Streaming with Apache Kafka & MongoDB - EMEA
Andrew Morgan
 
PPTX
Webinar: Data Streaming with Apache Kafka & MongoDB
MongoDB
 
PDF
Full stack visibility with elastic, KubeCon 2017
Carlos Pérez-Aradros
 
PPTX
Data Analysis on AWS
Paolo latella
 
PDF
Akka Streams And Kafka Streams: Where Microservices Meet Fast Data
Lightbend
 
PDF
DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK
Timothy Spann
 
PPTX
Supporting Digital Media Workflows in the Cloud with Perforce Helix
Perforce
 
PDF
Introduction to apache kafka
Samuel Kerrien
 
PDF
Fluentd Overview, Now and Then
SATOSHI TAGOMORI
 
PDF
16 months @ SoundCloud
Tobias Schmidt
 
Akka, Spark or Kafka? Selecting The Right Streaming Engine For the Job
Lightbend
 
Logging for Production Systems in The Container Era
Sadayuki Furuhashi
 
Monitoring docker, k8s and your applications with the elastic stack
SmartWave
 
Flink September 2015 Community Update
Robert Metzger
 
Logs aggregation and analysis
Divante
 
Concepts and Patterns for Streaming Services with Kafka
QAware GmbH
 
Managing your Black Friday Logs - Antonio Bonuccelli - Codemotion Rome 2018
Codemotion
 
Analyzing Data at Scale with Apache Spark
Nicola Ferraro
 
Introduction to apache kafka, confluent and why they matter
Paolo Castagna
 
Down the event-driven road: Experiences of integrating streaming into analyti...
inovex GmbH
 
Data Streaming with Apache Kafka & MongoDB - EMEA
Andrew Morgan
 
Webinar: Data Streaming with Apache Kafka & MongoDB
MongoDB
 
Full stack visibility with elastic, KubeCon 2017
Carlos Pérez-Aradros
 
Data Analysis on AWS
Paolo latella
 
Akka Streams And Kafka Streams: Where Microservices Meet Fast Data
Lightbend
 
DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK
Timothy Spann
 
Supporting Digital Media Workflows in the Cloud with Perforce Helix
Perforce
 
Introduction to apache kafka
Samuel Kerrien
 
Fluentd Overview, Now and Then
SATOSHI TAGOMORI
 
16 months @ SoundCloud
Tobias Schmidt
 
Ad

More from mfrancis (20)

PDF
Eclipse Modeling Framework and plain OSGi the easy way - Mark Hoffman (Data I...
mfrancis
 
PDF
OSGi and Java 9+ - BJ Hargrave (IBM)
mfrancis
 
PDF
Simplify Web UX Coding using OSGi Modularity Magic - Paul Fraser (A2Z Living)
mfrancis
 
PDF
OSGi for the data centre - Connecting OSGi to Kubernetes - Frank Lyaruu
mfrancis
 
PDF
Remote Management and Monitoring of Distributed OSGi Applications - Tim Verbe...
mfrancis
 
PDF
OSGi with Docker - a powerful way to develop Java systems - Udo Hafermann (So...
mfrancis
 
PDF
A real world use case with OSGi R7 - Jurgen Albert (Data In Motion Consulting...
mfrancis
 
PDF
OSGi Feature Model - Where Art Thou - David Bosschaert (Adobe)
mfrancis
 
PDF
Migrating from PDE to Bndtools in Practice - Amit Kumar Mondal (Deutsche Tele...
mfrancis
 
PDF
OSGi CDI Integration Specification - Ray Augé (Liferay)
mfrancis
 
PDF
How OSGi drives cross-sector energy management - Jörn Tümmler (SMA Solar Tech...
mfrancis
 
PDF
Improved developer productivity thanks to Maven and OSGi - Lukasz Dywicki (Co...
mfrancis
 
PDF
It Was Twenty Years Ago Today - Building an OSGi based Smart Home System - Ch...
mfrancis
 
PDF
Popular patterns revisited on OSGi - Christian Schneider (Adobe)
mfrancis
 
PDF
Integrating SLF4J and the new OSGi LogService 1.4 - BJ Hargrave (IBM)
mfrancis
 
PDF
OSG(a)i: because AI needs a runtime - Tim Verbelen (imec)
mfrancis
 
PDF
Flying to Jupiter with OSGi - Tony Walsh (ESA) & Hristo Indzhov (Telespazio V...
mfrancis
 
PDF
MicroProfile, OSGi was meant for this - Ray Auge (Liferay)
mfrancis
 
PDF
Prototyping IoT systems with a hybrid OSGi & Node-RED platform - Bruce Jackso...
mfrancis
 
PDF
How to connect your OSGi application - Dirk Fauth (Bosch)
mfrancis
 
Eclipse Modeling Framework and plain OSGi the easy way - Mark Hoffman (Data I...
mfrancis
 
OSGi and Java 9+ - BJ Hargrave (IBM)
mfrancis
 
Simplify Web UX Coding using OSGi Modularity Magic - Paul Fraser (A2Z Living)
mfrancis
 
OSGi for the data centre - Connecting OSGi to Kubernetes - Frank Lyaruu
mfrancis
 
Remote Management and Monitoring of Distributed OSGi Applications - Tim Verbe...
mfrancis
 
OSGi with Docker - a powerful way to develop Java systems - Udo Hafermann (So...
mfrancis
 
A real world use case with OSGi R7 - Jurgen Albert (Data In Motion Consulting...
mfrancis
 
OSGi Feature Model - Where Art Thou - David Bosschaert (Adobe)
mfrancis
 
Migrating from PDE to Bndtools in Practice - Amit Kumar Mondal (Deutsche Tele...
mfrancis
 
OSGi CDI Integration Specification - Ray Augé (Liferay)
mfrancis
 
How OSGi drives cross-sector energy management - Jörn Tümmler (SMA Solar Tech...
mfrancis
 
Improved developer productivity thanks to Maven and OSGi - Lukasz Dywicki (Co...
mfrancis
 
It Was Twenty Years Ago Today - Building an OSGi based Smart Home System - Ch...
mfrancis
 
Popular patterns revisited on OSGi - Christian Schneider (Adobe)
mfrancis
 
Integrating SLF4J and the new OSGi LogService 1.4 - BJ Hargrave (IBM)
mfrancis
 
OSG(a)i: because AI needs a runtime - Tim Verbelen (imec)
mfrancis
 
Flying to Jupiter with OSGi - Tony Walsh (ESA) & Hristo Indzhov (Telespazio V...
mfrancis
 
MicroProfile, OSGi was meant for this - Ray Auge (Liferay)
mfrancis
 
Prototyping IoT systems with a hybrid OSGi & Node-RED platform - Bruce Jackso...
mfrancis
 
How to connect your OSGi application - Dirk Fauth (Bosch)
mfrancis
 

Recently uploaded (20)

PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PDF
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
PPT
Teaching material agriculture food technology
LiaRayya
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PPTX
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
AVTRON Technologies LLC
 
PDF
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
PDF
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
PDF
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
DOCX
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
PDF
Encapsulation theory and applications.pdf
gurumoop
 
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
PDF
Modernizing your data center with Dell and AMD
Principled Technologies
 
PDF
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PPTX
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
PDF
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
PDF
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
Teaching material agriculture food technology
LiaRayya
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
AVTRON Technologies LLC
 
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
Encapsulation theory and applications.pdf
gurumoop
 
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
Modernizing your data center with Dell and AMD
Principled Technologies
 
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
Machine learning based COVID-19 study performance prediction
IAESIJAI
 

From logging to monitoring to reactive insights - C Schneider