OSDC 2019 | Fast log management for your infrastructure by Nicolas Frankel

@nicolas_frankel
FAST LOG MANAGEMENT
For your infrastructure

@nicolas_frankel
ME, MYSELF AND I
● Developer advocate
○ Previously
developer/architect
consultant
● DevOps-minded

@nicolas_frankel
European alternative to the “big” cloud-computing
players
● Privacy-minded
● Great support

@nicolas_frankel
LOGGER.debug(
"Cart price is now {}", cart.getPrice())
The root of all evil

@nicolas_frankel
if (LOGGER.isDebugEnabled) {
LOGGER.debug(
}
An improvement, but for whom?

@nicolas_frankel
if (LOGGER.isDebugEnabled) {
LOGGER.warn(
}
Ooops?!

@nicolas_frankel
LOGGER.debug(formatter ->
formatter.format(
)
Lazy computation for the win!

@nicolas_frankel
● SSD vs. HDD vs. NFS
● It takes time to actually
write to a file
We are bound to the physical world...

@nicolas_frankel
Writing process
1. Open the stream
2. Write bytes
3. Close the stream

@nicolas_frankel
Synchronous vs. asynchronous logging
● By default, logging is
blocking
● Most frameworks allow
asynchronous logging

@nicolas_frankel
<configuration>
<appender name="FILE" class="c.q.l.core.FileAppender">
<file>myapp.log</file>
<encoder>
<pattern>%logger{35} - %msg%n</pattern>
</encoder>
</appender>
<appender name="ASYNC" class="c.q.l.classic.AsyncAppender">
<appender-ref ref="FILE" />
</appender>
<root level="DEBUG">
<appender-ref ref="ASYNC" />
</root>
</configuration>

@nicolas_frankel
● Queue size
● Discarding threshold
● Never blocks:
○ Drop messages vs block
Example: Logback configuration

@nicolas_frankel
Associated meta-data
● Timestamp
● Log level
● Thread name
● Class name
● Method name
● Line number in the file
● etc.

@nicolas_frankel
Some metadata is expensive to get
● e.g. line number
● Better not compute it
○ Then, writing it explicitly
might be wrong

@nicolas_frankel
Logs aggregation
● Logs by themselves are
useless
● Centralized Logging Pattern
○ Elasticsearch
○ Splunk
○ Graylog

@nicolas_frankel
Mandatory additional metadata
● File
● Host/IP
● Cloud zone
● Environment
○ e.g. “DEV” vs “PROD”

@nicolas_frankel
Searching in logs
● Logs are not the end!
● Searches are:
○ “Find me all the logs that
happened yesterday on
JVMs in PROD”

@nicolas_frankel
Schema
On read vs. on write

@nicolas_frankel
Example architecture: the Elastic Stack
● Filebeat
○ Read log files
● Logstash
○ Parsing of log messages

@nicolas_frankel
2018-12-17 13:56:54.906 INFO
1 --- [ restartedMain]
c.e.configmgmt.demo.DemoAppli
cation : Started
DemoApplication in 3.833
seconds (JVM running for
4.303)

@nicolas_frankel
{
"date": [[ "18-12-17" ]],
"MONTHDAY": [[ "18" ]],
"MONTHNUM": [[ "12" ]],
"YEAR": [[ "17" ]],
"time": [[ "13:56:54.906" ]],
"HOUR": [[ "13" ]],
"MINUTE": [[ "56" ]],
"SECOND": [[ "54.906" ]],
"level": [[ "INFO" ]],
"threadName": [[ "restartedMain" ]],
"class": [[ "c.e.configmgmt.demo.DemoApplication" ]],
"message":
[[ "Started DemoApplication in 3.833 seconds (JVM running for
4.303)" ]]
}

@nicolas_frankel
We don’t actually need
Logstash if we produce JSON
directly!
Why?!

@nicolas_frankel
{
"date": "18-12-17",
"time": "13:56:54.906",
"level": "INFO",
"thread": "restartedMain",
"class":
"c.e.configmgmt.demo.DemoApplication",
"message": "Started DemoApplication in
3.833 seconds (JVM running for 4.303)"
}

@nicolas_frankel
● Events vs. log files
● Configuration hot reload
Food for thoughts

@nicolas_frankel
● syslog-ng
● Kafka
Logging @ Exoscale

@nicolas_frankel
Summary
1. Pass computations instead of results to log statements
2. Consider the physical file system to log to
3. Go asynchronous if speed counts more than reliability
4. Don’t use expensive meta-data
• Consider hot reloading configuration just in case
5. Schema on write is slower, but the alternative is worse
6. Send JSON directly

@nicolas_frankel
Takeaway
It’s everyone’s responsibility
to have fast logs:
● Developers
● Ops
● Architects

@nicolas_frankel
Takeaway #2
It’s a matter of trade-offs:
● Speed
● Reliability
● Custom context

@nicolas_frankel
THANKS!
● https://blog.frankel.ch/
● https://exoscale.com/syslog/
● @nicolas_frankel

OSDC 2019 | Fast log management for your infrastructure by Nicolas Frankel

More Related Content

What's hot (20)

Similar to OSDC 2019 | Fast log management for your infrastructure by Nicolas Frankel (20)

Recently uploaded (20)

OSDC 2019 | Fast log management for your infrastructure by Nicolas Frankel