ContainerDays NYC 2016: "Observability and Manageability in a Container Environment" (Tim Gross)
0 likes512 views
- The document discusses observability in containerized systems and microservices. It provides examples of logging output from containerized applications like Nginx and how to parse the logs to extract useful fields. - The examples show how Docker log drivers can mangle structured logs by adding metadata, and suggest directly logging from applications to collectors. - Configurations for Logstash are shown to parse logs from Nginx and other applications running in containers in order to extract fields like timestamps, requests IDs, and HTTP information for further analysis.
![@0x74696dgithub.com/tgross/observability-workshop
<20>Nov 11 2016 14:52:01 nginx/edaac4e19616 [4996]: [2016-11-04T14:52:01+00:00]
5c18b677d628d9511818baab9f33ceb3 "GET / HTTP/1.1" 200 204 "-" 127.0.0.1 "curl/
7.47.0"](https://image.slidesharecdn.com/containerdays-nyc-2016-observability-and-manageability-in-a-container-environment-tim-gross-161109015527/85/ContainerDays-NYC-2016-Observability-and-Manageability-in-a-Container-Environment-Tim-Gross-23-320.jpg)
![@0x74696dgithub.com/tgross/observability-workshop
<20>Nov 11 2016 14:52:01 nginx/edaac4e19616 [4996]: [2016-11-04T14:52:01+00:00]
5c18b677d628d9511818baab9f33ceb3 "GET / HTTP/1.1" 200 204 "-" 127.0.0.1 "curl/
7.47.0"
syslog wrapper: added by log driver](https://image.slidesharecdn.com/containerdays-nyc-2016-observability-and-manageability-in-a-container-environment-tim-gross-161109015527/85/ContainerDays-NYC-2016-Observability-and-Manageability-in-a-Container-Environment-Tim-Gross-24-320.jpg)
![@0x74696dgithub.com/tgross/observability-workshop
<20>Nov 11 2016 14:52:01 nginx/edaac4e19616 [4996]: [2016-11-04T14:52:01+00:00]
5c18b677d628d9511818baab9f33ceb3 "GET / HTTP/1.1" 200 204 "-" 127.0.0.1 "curl/
7.47.0"
container identifier via Compose
hostname flag](https://image.slidesharecdn.com/containerdays-nyc-2016-observability-and-manageability-in-a-container-environment-tim-gross-161109015527/85/ContainerDays-NYC-2016-Observability-and-Manageability-in-a-Container-Environment-Tim-Gross-25-320.jpg)
![@0x74696dgithub.com/tgross/observability-workshop
<20>Nov 11 2016 14:52:01 nginx/edaac4e19616 [4996]: [2016-11-04T14:52:01+00:00]
5c18b677d628d9511818baab9f33ceb3 "GET / HTTP/1.1" 200 204 "-" 127.0.0.1 "curl/
7.47.0"
Nginx access log timestamp](https://image.slidesharecdn.com/containerdays-nyc-2016-observability-and-manageability-in-a-container-environment-tim-gross-161109015527/85/ContainerDays-NYC-2016-Observability-and-Manageability-in-a-Container-Environment-Tim-Gross-26-320.jpg)
![@0x74696dgithub.com/tgross/observability-workshop
<20>Nov 11 2016 14:52:01 nginx/edaac4e19616 [4996]: [2016-11-04T14:52:01+00:00]
5c18b677d628d9511818baab9f33ceb3 "GET / HTTP/1.1" 200 204 "-" 127.0.0.1 "curl/
7.47.0"
Nginx $request_id field](https://image.slidesharecdn.com/containerdays-nyc-2016-observability-and-manageability-in-a-container-environment-tim-gross-161109015527/85/ContainerDays-NYC-2016-Observability-and-Manageability-in-a-Container-Environment-Tim-Gross-27-320.jpg)
![@0x74696dgithub.com/tgross/observability-workshop
$ vi ~/workshop/setup/supporting/logstash/logstash.conf
...
filter {
# first parse out the body from the syslog format
# we've added tags via the Docker log drivers to identify the
# specific container and the service identifier. See:
# https://docs.docker.com/engine/admin/logging/log_tags/
grok {
match => {
"message" => '%{SYSLOG5424PRI:syslog5424_pri}+(?:%
{SYSLOGTIMESTAMP:syslog_timestamp}|-) %{WORD:serviceid}/+(?:%
{HOSTNAME:containerid}|-)[+(?:%{POSINT:pid}|-)]: %{GREEDYDATA:msg}'
}
}
syslog_pri { }
...](https://image.slidesharecdn.com/containerdays-nyc-2016-observability-and-manageability-in-a-container-environment-tim-gross-161109015527/85/ContainerDays-NYC-2016-Observability-and-Manageability-in-a-Container-Environment-Tim-Gross-28-320.jpg)
![@0x74696dgithub.com/tgross/observability-workshop
...
# failed to match, so parse as error
if "_grokparsefailure" in [tags] {
mutate {
add_tag => "parse_error"
}
} else {
mutate {
# the raw message is redundant data at this point
remove_field => [ "message", "@source_host" ]
}
mutate {
# once we've got a valid syslog parse we can discard all this rubbish
# because the Docker log driver stomped all over anything useful
remove_field => [
"syslog_hostname", "syslog_message", "syslog_timestamp",
"syslog_severity", "syslog_facility_code", "syslog_severity_code",
"syslog_facility", "syslog5424_pri"
]
}
}
...](https://image.slidesharecdn.com/containerdays-nyc-2016-observability-and-manageability-in-a-container-environment-tim-gross-161109015527/85/ContainerDays-NYC-2016-Observability-and-Manageability-in-a-Container-Environment-Tim-Gross-29-320.jpg)
![@0x74696dgithub.com/tgross/observability-workshop
...
# filter to get the application-specific log format. lots of these have
# their own timestamps, which we'll capture and overwrite the "outermost"
# timestamp with
grok {
# nginx access log
match => {
"msg" => '[%{TIMESTAMP_ISO8601:log_timestamp}] %{WORD:req_id} "%
{WORD:http_method} %{URIPATHPARAM:http_request} HTTP/%{NUMBER:http_version}" %
{NUMBER:http_code} %{NUMBER:http_bytes_sent} (?:%{QUOTEDSTRING:http_referer}|-) %
{IP:client} (?:%{QUOTEDSTRING:http_user_agent}|-)'
}
# nginx error msg
match => {
"msg" => '(?<log_timestamp>%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME}) (?
<http_timestamp>%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME})? ?%{GREEDYDATA:msg}'
}
...](https://image.slidesharecdn.com/containerdays-nyc-2016-observability-and-manageability-in-a-container-environment-tim-gross-161109015527/85/ContainerDays-NYC-2016-Observability-and-Manageability-in-a-Container-Environment-Tim-Gross-30-320.jpg)
![@0x74696dgithub.com/tgross/observability-workshop
# mysql log messages
match => {
"msg" => '%{TIMESTAMP_ISO8601:log_timestamp} %{NUMBER} [%{LOGLEVEL:level}] ?%
{GREEDYDATA:msg}'
}
# mysql manage.py log messages
match => {
"msg" => '(?<log_timestamp>%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME})? %
{LOGLEVEL:level} manage %{GREEDYDATA:msg}'
}
# ContainerPilot log format
match => {
"msg" => '(?<log_timestamp>%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME})? ?%
{GREEDYDATA:msg}'
}
# catchall
match => { "msg" => "%{GREEDYDATA:msg}" }
overwrite => [ "log_timestamp" ]
overwrite => [ "msg" ]
} # end grok](https://image.slidesharecdn.com/containerdays-nyc-2016-observability-and-manageability-in-a-container-environment-tim-gross-161109015527/85/ContainerDays-NYC-2016-Observability-and-Manageability-in-a-Container-Environment-Tim-Gross-31-320.jpg)
![@0x74696dgithub.com/tgross/observability-workshop
# fortunes application
json {
source => "msg"
add_field => [ "log_timestamp", "%{time}"] # overwrites w/ timestamp from app
remove_field => [ "time" ]
}
# anything other than our fortunes application will not be JSON
# so we'll ignore this error
if "_jsonparsefailure" in [tags] {
mutate { remove_tag => "_jsonparsefailure" }
}
}](https://image.slidesharecdn.com/containerdays-nyc-2016-observability-and-manageability-in-a-container-environment-tim-gross-161109015527/85/ContainerDays-NYC-2016-Observability-and-Manageability-in-a-Container-Environment-Tim-Gross-32-320.jpg)
ContainerDays NYC 2016: "Observability and Manageability in a Container Environment" (Tim Gross)
- 1. OBSERVABILITY IN A CONTAINERIZED WORLD Tim Gross @0x74696d
- 3. @0x74696dgithub.com/tgross/observability-workshop “We have built mind-bogglingly complicated systems that we cannot see, allowing glaring performance problems to hide in broad daylight in our systems.” Bryan Cantrill, Joyent CTO ACM Queue Vol 4, Issue 1, 2006 Feb 23 http://queue.acm.org/detail.cfm?id=1117401
- 4. @0x74696dgithub.com/tgross/observability-workshop “Get used to interacting with your observability tooling every day. As part of your release cycle, or just out of curiosity. Honestly, things are broken all the time — you don’t even know what normal looks like unless you’re also interacting with your observability tooling under “normal” circumstances.” Charity Majors CEO Honeycomb.io Building Badass Engineers and Badass Teams http://ow.ly/IDOs305uN7W
- 17. @0x74696dgithub.com/tgross/observability-workshop Nginx Container Fortunes Container (node.js) Consul Agent Container MySQL Container Docker Your KVM: ▸ Docker Engine ▸ Composed containers ▸ Host networking (shared IP) Virtual Machine
- 18. @0x74696dgithub.com/tgross/observability-workshop Customer KVM Customer KVM SmartOS Container Hypervisor Customer Container Customer Container Customer Container Customer Container Customer Container Customer Container Customer Container Customer Container Customer Container Customer Container Triton compute node: ▸ SmartOS ▸ Many customer containers ▸ VXLAN: 1 container =1+ IPs Bare-metal compute
- 19. @0x74696dgithub.com/tgross/observability-workshop Customer KVM SmartOS Container Hypervisor Customer Container Customer Container Customer Container Customer Container Customer Container Customer Container Customer Container Customer Container Customer Container Customer Container Customer Container Customer KVM Elasticsearch Prometheus Logstash Kibana Triton Cloud ▸ Many compute nodes ▸ Containers distributed transparently across DC Bare-metal compute
- 20. @0x74696dgithub.com/tgross/observability-workshop Docker Log drivers ‣ Captures stdout/stderr; easy for 12-Factor apps ‣ Mangles multi-line logs (stack traces) ‣ Wraps log line in log driver structure: makes parsing structured logs really messy
- 21. @0x74696dgithub.com/tgross/observability-workshop Logging to File ‣ Mounting volume on host for logging to file isn’t portable across platforms (e.g. PaaS) ‣ Log shippers as co-process in container can be arbitrarily smart
- 22. @0x74696dgithub.com/tgross/observability-workshop Structured Logging ‣ Require more storage, less ingest processing ‣ More metadata to search on ‣ Docker log drivers blow away structured logs =( ‣ Consider logging directly from app to collector @0x74696d
- 33. @0x74696dgithub.com/tgross/observability-workshop Metrics ‣ Ephemeral containers == higher dimensionality ‣ Use service discovery to find collector / targets ‣ Pulling metrics safer / more scalable in multi-tenant environments (tenants can’t DDOS collector)
- 34. @0x74696dgithub.com/tgross/observability-workshop Prometheus Container Pilot Application Consul I’m “application” at 192.168.1.100:3000 I’m “telemetry” at 192.168.1.100:9090 Application container Sensor
- 37. @0x74696dgithub.com/tgross/observability-workshop Container Design ‣ Large containers == slower deploys ‣ Mount common tooling read-only from the host
- 38. @0x74696dgithub.com/tgross/observability-workshop Distributed Request Tracing ‣ Zipkin / Open Tracing looks promising; little support outside app code (load balancers, DB) ‣ Nginx can inject request ID field ‣ Carry request ID in your logging
- 39. OBSERVABILITY IN A CONTAINERIZED WORLD Tim Gross @0x74696d