@omerlh
The Dark Side of Monitoring
Omer Levi Hevroni
May 2019
@omerlh
Observability
https://www.navantis.com/managed-monitoring/
@omerlh
Can Hackers Use It Too?
@omerlh
I’m a builder
@omerlh
@omerlh
DevSecOps @
@omerlh
Threat Modeling?
@omerlh
Threat Modeling?
● What are we building?
● What can go wrong?
● What are we doing about it?
● Did we do a good job?
@omerlh
What is Monitoring?
@omerlh
@omerlh
@omerlh
Black Box
Monitoring
White Box
Monitoring
https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/
@omerlh
Let’s Dive In!
@omerlh
White Box
Monitoring
@omerlh
Threat Modeling
● What are we building?
● What can go wrong?
● What are we doing about it?
● Did we do a good job?
@omerlh
White-box Monitoring
Monitoring based on metrics exposed by the internals of the system, including
logs, interfaces like the Java Virtual Machine Profiling Interface, or an HTTP
handler that emits internal statistics.
@omerlh
White-box Monitoring for Kubernetes
● Probes (liveness/readiness)
● Prometheus
@omerlh
Probes
@omerlh
What are we building?
@omerlh
Probes
@omerlh
Why?
●Check dependencies
○ Databases
○ Other micro-services
●Kill services if it misfunction before users experience it
@omerlh
Code Example
https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks
@omerlh
What Can go wrong?
@omerlh
Exploiting liveness - information disclosure
@omerlh
Exploiting liveness – Denial of Service
@omerlh
@omerlh
What are we doing about it?
@omerlh
Block access
https://www.edureka.
co/community/19277/
access-some-
specific-paths-while-
using-kubernetes-
ingress?show=19278
#a19278
@omerlh
Caching
Source code - Kamus
@omerlh
Prometheus
@omerlh
What are we building?
@omerlh
Prometheus
● Monitoring system
● Time-Series database
● Alerting system
● Auto-discovery
https://prometheus.io/
@omerlh
Prometheus Scraping Model
@omerlh
What Can go wrong?
@omerlh
Exploiting Prometheus - information disclosure
@omerlh
@omerlh
Some interesting metrics...
@omerlh
What about this metric?
@omerlh
Here is the code behind it…
@omerlh
What Can We Do About It?
@omerlh
Block access
https://www.edureka.
co/community/19277/
access-some-
specific-paths-while-
using-kubernetes-
ingress?show=19278
#a19278
@omerlh
Prometheus Metrics Limit
https://www.omerlh.info/2019/03/04/keeping-prometheus-in-shape/
@omerlh
Did we do a good job?
@omerlh
Threat Modeling
● What are we building?
● What can go wrong?
● What are we doing about it?
● Did we do a good job?
@omerlh
Black Box
Monitoring
White Box
Monitoring
https://landing.google.com/sre/sre-
book/chapters/monitoring-distributed-systems/
@omerlh
Movie Streaming
@omerlh
What are we building?
@omerlh
Black-box Monitoring
Testing externally visible behavior as a user would see it.
@omerlh
Our Monitoring System
@omerlh
What Can go wrong?
@omerlh
STRIDE
S – Spoofing
T – Tampering
R – Repudiation
I – Information Disclosure
D – Denial of Service
E – Elevation of Privileges
@omerlh
Spoofing
@omerlh
Information Disclosure
@omerlh
Repudiation
@omerlh
Denial of Service
@omerlh
What are we doing about it?
@omerlh
Potential Mitigations
● Least Privilege
● Block access
● Tracing
● Limit to test data only
@omerlh
Did we do a good job?
@omerlh
http://www.applestory.biz/hermione-hand-raise-gif.html
Questions?
@omerlh
Can Hackers Use It Too?
@omerlh
Key Take Away
● Monitoring is just code
● Careful when exposed to the internet
● Use cache when possible
● Conduct threat model for black-box
monitoring
@omerlh
Feedback appreciated
@omerlh
Observability
https://www.navantis.com/managed-monitoring/
@omerlh
Thank You!
Omer Levi Hevroni
April 2019

More Related Content

PDF
Learn How To Embrace Conflicts
PPTX
STRUCTURE
PDF
Building Better Products By Building Better Teams
PPTX
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...
PPTX
An Introduction to Prometheus (GrafanaCon 2016)
PDF
An Introduction to Prometheus
PPTX
Evolution of Monitoring and Prometheus (Dublin 2018)
PPTX
Prometheus - Open Source Forum Japan
Learn How To Embrace Conflicts
STRUCTURE
Building Better Products By Building Better Teams
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...
An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus
Evolution of Monitoring and Prometheus (Dublin 2018)
Prometheus - Open Source Forum Japan

Similar to The Dark Side of Monitoring (20)

PPTX
What does "monitoring" mean? (FOSDEM 2017)
PDF
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
PDF
Hacking Kubernetes Threat Driven Analysis and Defense 1st Edition Andrew Martin
PDF
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)
PPTX
Prometheus (Prometheus London, 2016)
PDF
Microservices and Prometheus (Microservices NYC 2016)
PDF
Prometheus (Microsoft, 2016)
PDF
Monitoring in Big Data Platform - Albert Lewandowski, GetInData
PPTX
The Art of Container Monitoring
PDF
Monitoring - deeper dive
PPTX
Prometheus for Monitoring Metrics (Percona Live Europe 2017)
PPSX
Service Mesh - Observability
PDF
Prometheus - basics
PDF
Prometheus Introduction (InfraCoders Vienna)
PPTX
Prometheus for Monitoring Metrics (Fermilab 2018)
PDF
Building an Observability Platform in 389 Difficult Steps
PDF
Infrastructure & System Monitoring using Prometheus
PPTX
Prometheus Training
PDF
Monitoring kubernetes with prometheus
PPTX
Prometheus (Monitorama 2016)
What does "monitoring" mean? (FOSDEM 2017)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Hacking Kubernetes Threat Driven Analysis and Defense 1st Edition Andrew Martin
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)
Prometheus (Prometheus London, 2016)
Microservices and Prometheus (Microservices NYC 2016)
Prometheus (Microsoft, 2016)
Monitoring in Big Data Platform - Albert Lewandowski, GetInData
The Art of Container Monitoring
Monitoring - deeper dive
Prometheus for Monitoring Metrics (Percona Live Europe 2017)
Service Mesh - Observability
Prometheus - basics
Prometheus Introduction (InfraCoders Vienna)
Prometheus for Monitoring Metrics (Fermilab 2018)
Building an Observability Platform in 389 Difficult Steps
Infrastructure & System Monitoring using Prometheus
Prometheus Training
Monitoring kubernetes with prometheus
Prometheus (Monitorama 2016)
Ad

More from Soluto (20)

PPTX
Solving trust issues at scale - AppSec California
PPTX
Solving trust issues at scale
PPTX
Things I wish someone had told me about Istio, Omer Levi Hevroni
PPTX
Can Kubernetes Keep a Secret? - Women in AppSec Webinar
PPTX
FTRD - Can Kubernetes Keep a Secret?
PPTX
Hacking like a FED
PPTX
Monitoria@Icinga camp berlin
PPTX
Can Kubernetes Keep a Secret?
PPTX
Kamus intro
PPTX
Secure Your Pipeline
PDF
React new features and intro to Hooks
PPTX
Secure the Pipeline - OWASP Poland Day 2018
PDF
Monitoria@reversim
PPTX
Languages don't matter anymore!
PPTX
Security Testing for Containerized Applications
PPTX
Owasp glue
PPTX
Unify logz with fluentd
PPTX
Storing data in Redis like a pro
PPTX
Monitor all the thingz slideshare
PPTX
Authentication without Authentication - AppSec California
Solving trust issues at scale - AppSec California
Solving trust issues at scale
Things I wish someone had told me about Istio, Omer Levi Hevroni
Can Kubernetes Keep a Secret? - Women in AppSec Webinar
FTRD - Can Kubernetes Keep a Secret?
Hacking like a FED
Monitoria@Icinga camp berlin
Can Kubernetes Keep a Secret?
Kamus intro
Secure Your Pipeline
React new features and intro to Hooks
Secure the Pipeline - OWASP Poland Day 2018
Monitoria@reversim
Languages don't matter anymore!
Security Testing for Containerized Applications
Owasp glue
Unify logz with fluentd
Storing data in Redis like a pro
Monitor all the thingz slideshare
Authentication without Authentication - AppSec California
Ad

Recently uploaded (20)

PDF
From MVP to Full-Scale Product A Startup’s Software Journey.pdf
PPTX
observCloud-Native Containerability and monitoring.pptx
PDF
A contest of sentiment analysis: k-nearest neighbor versus neural network
PDF
A Late Bloomer's Guide to GenAI: Ethics, Bias, and Effective Prompting - Boha...
PDF
Getting Started with Data Integration: FME Form 101
PDF
Video forgery: An extensive analysis of inter-and intra-frame manipulation al...
PDF
Microsoft Solutions Partner Drive Digital Transformation with D365.pdf
PDF
Developing a website for English-speaking practice to English as a foreign la...
PDF
TrustArc Webinar - Click, Consent, Trust: Winning the Privacy Game
PDF
Univ-Connecticut-ChatGPT-Presentaion.pdf
PPTX
Tartificialntelligence_presentation.pptx
PPTX
The various Industrial Revolutions .pptx
PDF
Architecture types and enterprise applications.pdf
PDF
STKI Israel Market Study 2025 version august
PPTX
Group 1 Presentation -Planning and Decision Making .pptx
PDF
Assigned Numbers - 2025 - Bluetooth® Document
PDF
Hybrid horned lizard optimization algorithm-aquila optimizer for DC motor
PDF
A comparative study of natural language inference in Swahili using monolingua...
PDF
Taming the Chaos: How to Turn Unstructured Data into Decisions
PDF
1 - Historical Antecedents, Social Consideration.pdf
From MVP to Full-Scale Product A Startup’s Software Journey.pdf
observCloud-Native Containerability and monitoring.pptx
A contest of sentiment analysis: k-nearest neighbor versus neural network
A Late Bloomer's Guide to GenAI: Ethics, Bias, and Effective Prompting - Boha...
Getting Started with Data Integration: FME Form 101
Video forgery: An extensive analysis of inter-and intra-frame manipulation al...
Microsoft Solutions Partner Drive Digital Transformation with D365.pdf
Developing a website for English-speaking practice to English as a foreign la...
TrustArc Webinar - Click, Consent, Trust: Winning the Privacy Game
Univ-Connecticut-ChatGPT-Presentaion.pdf
Tartificialntelligence_presentation.pptx
The various Industrial Revolutions .pptx
Architecture types and enterprise applications.pdf
STKI Israel Market Study 2025 version august
Group 1 Presentation -Planning and Decision Making .pptx
Assigned Numbers - 2025 - Bluetooth® Document
Hybrid horned lizard optimization algorithm-aquila optimizer for DC motor
A comparative study of natural language inference in Swahili using monolingua...
Taming the Chaos: How to Turn Unstructured Data into Decisions
1 - Historical Antecedents, Social Consideration.pdf

The Dark Side of Monitoring