Introducing log analysis to your organization
4 likes3,660 views
The document discusses introducing log analysis to an organization. It covers log shipping architecture using file shippers, centralized buffers like Kafka and Redis, and storage and analysis using Elasticsearch, Kibana and Grafana. Specific topics covered include choosing the right shipper, buffer types, protocols, and optimizing Elasticsearch configuration, indices, and hardware for different node types like data, ingest and client nodes.
Introducing log analysis to your organization
- 3. Next 60 minutes… Log shipping - buffers - protocols - parsing Central buffering - Kafka - Redis Storage & Analysis - Elasticsearch - Kibana - Grafana Why & How? - Should I try? - Open source - Commercial
- 27. Log shipping architecture File Shipper File Shipper File Shipper Centralized Buffer
- 28. Log shipping architecture File Shipper File Shipper File Shipper Centralized Buffer data
- 29. Log shipping architecture File Shipper File Shipper File Shipper Centralized Buffer ES ES ES ES ES ES ES ES ES data
- 30. Focus: Shipper File Shipper File Shipper File Shipper Centralized Buffer ES ES ES ES ES ES ES ES ES data
- 32. Buffers performance & availability batches & threads when central buffer is gone
- 33. Buffer types Disk || memory || combined hybrid approach On source || centralized App Buffer App Buffer file or local log shipper easy scaling – fewer moving parts often with the use of lightweight shipper App App Kafka / Redis / Logstash / etc… one place for all changes extra features made easy (like TTL) ES ES
- 35. Protocols UDP – fast, cool for the application, not reliable TCP – reliable (almost) application gets ACK when written to buffer Application level ACKs may be needed HTTP RELP Beats Kafka Logstash, rsyslog, Fluentd Logstash, rsyslog Logstash, Filebeat Logstash, rsyslog, Filebeat, Fluentd
- 36. Choosing the shipper application rsyslog Elasticsearch http socket memory & disk assisted queues
- 39. Focus: Centralized Buffer File Shipper File Shipper File Shipper Centralized Buffer ES ES ES ES ES ES ES ES ES data
- 42. Kafka & topics security_logs access_logs app1_logs app2_logs Kafka stores data in topics written on disk
- 47. Things to remember when using Kafka Scales by adding more partitions not threads The more IOPS the better Keep the # of consumers equal to # of partitions Replicas used for HA and FT only Offsets stored per consumer – multiple destinations easily possible
- 48. Focus: Elasticsearch File Shipper File Shipper File Shipper Centralized Buffer ES ES ES ES ES ES ES ES ES data
- 51. Elasticsearch – Indices Index – logical place for data
- 52. Elasticsearch – Indices Index – logical place for data Index – can be compared to database in DB
- 53. Elasticsearch – Indices Index – logical place for data Index – can be compared to database in DB Index – built out of one or more shards
- 54. Elasticsearch – Indices Index – logical place for data Index – can be compared to database in DB Index – built out of one or more shards Shard – can be spread among multiple nodes
- 64. Daily indices are a good start 2017.11.16 2017.11.17 2017.11.20 2017.11.21. . . Indexing is faster for smaller indices Deletes are cheap Search can be performed on indices that are needed Static indices are cache friendly indexing most searches
- 65. Daily indices are a good start 2017.11.16 2017.11.17 2017.11.20 2017.11.21. . . Indexing is faster for smaller indices Deletes are cheap Search can be performed on indices that are needed Static indices are cache friendly indexing most searches We delete whole indices
- 72. Slice using size Predictable searching and indexing performance Better indices balancing Fewer shards Easier handling of spiky loads Less costs because of better hardware utilization
- 73. Proper Elasticsearch configuration Keep index.refresh_interval at maximum possible value 1 sec -> 100%, 5 sec -> 125%, 30 sec -> 175% You can loosen up merges - possible because of heavy aggregation use - segments_per_tier -> higher - max_merge_at_once-> higher - max_merged_segment -> lower All prefixed with index.merge.policy } higher indexing throughput
- 76. Hot – cold architecture ES hot ES cold ES cold -Dnode.attr.tag=hot -Dnode.attr.tag=cold -Dnode.attr.tag=cold
- 77. Hot – cold architecture logs_2017.11.22 ES hot ES cold ES cold -Dnode.attr.tag=hot -Dnode.attr.tag=cold -Dnode.attr.tag=cold curl -XPUT localhost:9200/logs_2017.11.22 -d '{ "settings" : { "index.routing.allocation.exclude.tag" : "cold", "index.routing.allocation.include.tag" : "hot" } }'
- 78. Hot – cold architecture logs_2017.11.22 ES hot ES cold ES cold indexing
- 79. Hot – cold architecture logs_2017.11.22 logs_2017.11.23 ES hot ES cold ES cold indexing
- 80. Hot – cold architecture logs_2017.11.22 logs_2017.11.23 ES hot ES cold ES cold indexing move index after day ends curl -XPUT localhost:9200/logs_2017.11.22/_settings -d '{ "index.routing.allocation.exclude.tag" : "hot", "index.routing.allocation.include.tag” : "cold" }'
- 81. Hot – cold architecture logs_2017.11.23 logs_2017.11.22 ES hot ES cold ES cold indexing
- 82. Hot – cold architecture logs_2017.11.23 logs_2017.11.24 logs_2017.11.22 ES hot ES cold ES cold indexing
- 83. Hot – cold architecture logs_2017.11.23 logs_2017.11.24 logs_2017.11.22 ES hot ES cold ES cold indexing move index after day ends
- 84. Hot – cold architecture logs_2017.11.24 logs_2017.11.22 logs_2017.11.23 ES hot ES cold ES cold indexing
- 86. Hot – cold architecture summary ES cold Optimize costs – different hardware for different tier Performance – use case optimized hardware Isolation – long running searches don’t affect indexing
- 93. What about OS? Say NO to swap Set the right disk scheduler CFQ for spinning disks deadline for SSD Use proper mount options for ext4 noatime nodirtime data=writeback, nobarier For bare metal check CPU governor disable transparent huge pages /proc/sys/vm/nr_hugepages=0
- 100. Analysis - Kibana
- 101. Analysis - Grafana
- 102. Analysis - Grafana
- 103. Analysis - Grafana