Dev Day 2019: Phillip Krenn – Aggregierte Logging Patterns
1 like371 views
The document outlines best practices for logging patterns in various programming languages, emphasizing the importance of structured logging, error handling, and using tools like Filebeat and Elasticsearch for log management. It discusses anti-patterns, configuration examples, and the integration of metadata within logs to enhance observability. Additionally, the document touches on the orchestration of logging solutions in containerized environments using Docker and Kubernetes, along with performance considerations for logging infrastructure.
![[2018-09-28 10:30:38.516] ERROR net.xeraa.logging.LogMe [main] -
user_experience= , session=46, loop=15 -
Wake me up at night
java.lang.RuntimeException: Bad runtime...
at net.xeraa.logging.LogMe.main(LogMe.java:30)
^[%{TIMESTAMP_ISO8601:@timestamp}]%{SPACE}%{LOGLEVEL:log.level}
%{SPACE}%{USERNAME:log.package}%{SPACE}[%{WORD:log.method}]
%{SPACE}-%{SPACE}%{GREEDYDATA:log.labels}%{SPACE}-%{SPACE}
%{GREEDYDATA:message}(?:n+(?<stacktrace>(?:.|r|n)+))?
@xeraa](https://image.slidesharecdn.com/deck-2cd80654af7be447-190524075936/85/Dev-Day-2019-Phillip-Krenn-Aggregierte-Logging-Patterns-33-320.jpg)
![Bonues: Multi-Index
output.elasticsearch:
hosts: ["http://localhost:9200"]
indices:
- index: "warning-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
message: "WARN"
- index: "error-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
message: "ERR"
@xeraa](https://image.slidesharecdn.com/deck-2cd80654af7be447-190524075936/85/Dev-Day-2019-Phillip-Krenn-Aggregierte-Logging-Patterns-47-320.jpg)
![Ingest Pipeline
output.elasticsearch:
hosts: ["http://elasticsearch:9200"]
index: "docker"
pipelines:
- pipeline: "parse_java"
when.contains:
container.name: "java_app"
@xeraa](https://image.slidesharecdn.com/deck-2cd80654af7be447-190524075936/85/Dev-Day-2019-Phillip-Krenn-Aggregierte-Logging-Patterns-60-320.jpg)
![Ingest Pipeline
{
"description" : "Parse Java log lines",
"processors": [
{
"grok": {
"field": "message",
"patterns": [ "^[%{TIMESTAMP_ISO8601:timestamp}]%{SPACE}%{LOGLEVEL:log.level}
%{SPACE}%{USERNAME:log.package}%{SPACE}[%{WORD:log.method}]%{SPACE}-
%{SPACE}%{GREEDYDATA:labels}%{SPACE}-%{SPACE}%{GREEDYDATA:message_parsed}
(?:n+(?<stacktrace>(?:.|r|n)+))?" ],
"ignore_failure": true
}
}
]
}
@xeraa](https://image.slidesharecdn.com/deck-2cd80654af7be447-190524075936/85/Dev-Day-2019-Phillip-Krenn-Aggregierte-Logging-Patterns-61-320.jpg)
![Configuration Templates
filebeat.autodiscover:
providers:
- type: docker
templates:
- condition:
equals:
docker.container.image: redis
config:
- type: docker
containers.ids:
- "${data.docker.container.id}"
exclude_lines: ["^s+[-`('.|_]"]
@xeraa](https://image.slidesharecdn.com/deck-2cd80654af7be447-190524075936/85/Dev-Day-2019-Phillip-Krenn-Aggregierte-Logging-Patterns-64-320.jpg)
![Configuration Templates
filebeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition:
equals:
kubernetes.namespace: redis
config:
- type: docker
containers.ids:
- "${data.kubernetes.container.id}"
exclude_lines: ["^s+[-`('.|_]"]
@xeraa](https://image.slidesharecdn.com/deck-2cd80654af7be447-190524075936/85/Dev-Day-2019-Phillip-Krenn-Aggregierte-Logging-Patterns-74-320.jpg)
![Customize Indices
output.elasticsearch:
index: "%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}"
@xeraa](https://image.slidesharecdn.com/deck-2cd80654af7be447-190524075936/85/Dev-Day-2019-Phillip-Krenn-Aggregierte-Logging-Patterns-75-320.jpg)
Dev Day 2019: Phillip Krenn – Aggregierte Logging Patterns
- 1. Aggregierte Logging Patterns Philipp Krenn @xeraa @xeraa
- 2. @xeraa
- 3. @xeraa
- 4. @xeraa
- 5. @xeraa
- 6. @xeraa
- 7. @xeraa
- 8. @xeraa
- 10. @xeraa
- 12. Developer @xeraa
- 13. @xeraa
- 14. @xeraa
- 15. @xeraa
- 16. @xeraa
- 17. @xeraa
- 19. Disclaimer I build highly monitored Hello World apps @xeraa
- 20. Example: Java SLF4J, Logback, MDC with logstash-logback-encoder Alternative https://github.com/vy/log4j2-logstash-layout @xeraa
- 21. And Everywhere Else .NET: NLog JavaScript: Winston Python: structlog PHP: Monolog @xeraa
- 24. Parse @xeraa
- 25. @xeraa
- 26. Bind Mount Logs java_app: volumes: - './logs-docker/:/logs/' ... filebeat_for_logstash: volumes: - './logs-docker/:/mnt/logs/:ro' ... @xeraa
- 27. Collect Log Lines filebeat.inputs: - type: log paths: - /mnt/logs/*.log @xeraa
- 33. [2018-09-28 10:30:38.516] ERROR net.xeraa.logging.LogMe [main] - user_experience= , session=46, loop=15 - Wake me up at night java.lang.RuntimeException: Bad runtime... at net.xeraa.logging.LogMe.main(LogMe.java:30) ^[%{TIMESTAMP_ISO8601:@timestamp}]%{SPACE}%{LOGLEVEL:log.level} %{SPACE}%{USERNAME:log.package}%{SPACE}[%{WORD:log.method}] %{SPACE}-%{SPACE}%{GREEDYDATA:log.labels}%{SPACE}-%{SPACE} %{GREEDYDATA:message}(?:n+(?<stacktrace>(?:.|r|n)+))? @xeraa
- 36. Logstash Key Value Filter for MDC kv { source => "labels" field_split => "," trim_key => " " } @xeraa
- 37. Monitoring: Logstash Pipeline Plus other components @xeraa
- 38. Pro: No change Con: Regular expression, multiline, format changes @xeraa
- 39. Send @xeraa
- 40. @xeraa
- 41. logback.xml <appender name="logstash" class="net.logstash.logback.appender.LogstashAccessTcpSocketAppender"> <destination>logstash:4560</destination> <encoder class="net.logstash.logback.encoder.LogstashEncoder"/> </appender> @xeraa
- 42. Pro: No files Con: Outages & coupling @xeraa
- 43. Structure @xeraa
- 44. @xeraa
- 45. Collect JSON filebeat.input: - type: log paths: - /mnt/logs/*.json json: message_key: message keys_under_root: true @xeraa
- 47. Bonues: Multi-Index output.elasticsearch: hosts: ["http://localhost:9200"] indices: - index: "warning-%{[agent.version]}-%{+yyyy.MM.dd}" when.contains: message: "WARN" - index: "error-%{[agent.version]}-%{+yyyy.MM.dd}" when.contains: message: "ERR" @xeraa
- 48. Pro: Right format Con: JSON serialization overhead @xeraa
- 50. @xeraa
- 51. Where to put Filebeat? Sidecar @xeraa
- 52. @xeraa
- 54. Docker Logs filebeat.autodiscover: providers: - type: docker hints.enabled: true processors: - add_docker_metadata: ~ @xeraa
- 55. Metadata No Docker metadata with the other methods @xeraa
- 56. "docker": { "container": { "labels": { "app": "fizzbuzz", "co_elastic_logs/multiline_match": "after", "com_docker_compose_config-hash": "41520c6cf2b6a1f3dae4f16d0a6fd76760cdfc38fbfe43a3a3be2e09bdd1b8b5", "environment": "production", "co_elastic_logs/multiline_pattern": "^[", "co_elastic_logs/multiline_negate": "true", "com_docker_compose_oneoff": "False", "com_docker_compose_project": "java-logging", "com_docker_compose_service": "java_app", "com_docker_compose_container-number": "1", "com_docker_compose_version": "1.23.2" } } } @xeraa
- 57. Missing the Last Line Waiting for the newline @xeraa
- 58. Hints labels: - "app=fizzbuzz" - "co.elastic.logs/multiline.pattern=^[" - "co.elastic.logs/multiline.negate=true" - "co.elastic.logs/multiline.match=after" @xeraa
- 60. Ingest Pipeline output.elasticsearch: hosts: ["http://elasticsearch:9200"] index: "docker" pipelines: - pipeline: "parse_java" when.contains: container.name: "java_app" @xeraa
- 61. Ingest Pipeline { "description" : "Parse Java log lines", "processors": [ { "grok": { "field": "message", "patterns": [ "^[%{TIMESTAMP_ISO8601:timestamp}]%{SPACE}%{LOGLEVEL:log.level} %{SPACE}%{USERNAME:log.package}%{SPACE}[%{WORD:log.method}]%{SPACE}- %{SPACE}%{GREEDYDATA:labels}%{SPACE}-%{SPACE}%{GREEDYDATA:message_parsed} (?:n+(?<stacktrace>(?:.|r|n)+))?" ], "ignore_failure": true } } ] } @xeraa
- 63. ASCII Art _._ _.-``__ ''-._ _.-`` `. `_. ''-._ Redis 4.0.9 (00000000/0) 64 bit .-`` .-```. ```/ _.,_ ''-._ ( ' , .-` | `, ) Running in stand alone mode |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379 | `-._ `._ / _.-' | PID: 55757 `-._ `-._ `-./ _.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | http://redis.io `-._ `-._`-.__.-'_.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | `-._ `-._`-.__.-'_.-' _.-' `-._ `-.__.-' _.-' `-._ _.-' `-.__.-' @xeraa
- 64. Configuration Templates filebeat.autodiscover: providers: - type: docker templates: - condition: equals: docker.container.image: redis config: - type: docker containers.ids: - "${data.docker.container.id}" exclude_lines: ["^s+[-`('.|_]"] @xeraa
- 65. Who Logs the Logger Avoid loops Process without -e filebeat.yml: logging.to_files: true @xeraa
- 68. @xeraa
- 69. Where to put Filebeat? DaemonSet @xeraa
- 71. Metadata Either in cluster or outside processors: - add_kubernetes_metadata: in_cluster: true - add_kubernetes_metadata: in_cluster: false host: <hostname> kube_config: ${HOME}/.kube/config @xeraa
- 72. { "host": "172.17.0.21", "port": 9090, "kubernetes": { "container": { "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51", "image": "my-java:1.0.0", "name": "my-java" }, "labels": { "app": "java", }, "namespace": "default", "node": { "name": "minikube" }, "pod": { "name": "java-2657348378-k1pnh" } }, } @xeraa
- 73. More Metadata Add: Cloud, local timezone, process Drop: Events, fields Rename: Fields Dissect, DNS reverse lookup @xeraa
- 74. Configuration Templates filebeat.autodiscover: providers: - type: kubernetes templates: - condition: equals: kubernetes.namespace: redis config: - type: docker containers.ids: - "${data.kubernetes.container.id}" exclude_lines: ["^s+[-`('.|_]"] @xeraa
- 77. Moar @xeraa
- 80. Index Patterns Time based (default: daily) Versioned @xeraa
- 81. Sizing Daily volume * Retention * Replication Number of shards @xeraa
- 82. Index Lifecycle Management ! " @xeraa
- 83. Order https://github.com/elastic/elasticsearch/blob/7.1/x-pack/plugin/core/src/main/java/org/elasticsearch/ xpack/core/indexlifecycle/TimeseriesLifecycleType.java static final List<String> ORDERED_VALID_HOT_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, RolloverAction.NAME ); static final List<String> ORDERED_VALID_WARM_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, ReadOnlyAction.NAME, AllocateAction.NAME, ShrinkAction.NAME, ForceMergeAction.NAME ); static final List<String> ORDERED_VALID_COLD_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, AllocateAction.NAME, FreezeAction.NAME ); static final List<String> ORDERED_VALID_DELETE_ACTIONS = Arrays.asList( DeleteAction.NAME ); @xeraa
- 85. Ratio Heap : Storage Index > Frozen Index > Closed Index Read-only @xeraa
- 86. Throttled Thread Pool 1 parallel search / node 100 in queue @xeraa