On Centralizing Logs
13 likes15,583 views
The document discusses the centralization of logs using tools like Elasticsearch, Logstash, and Kibana, emphasizing stability, performance, and configuration tips. It outlines best practices for log management, including preventing split-brain scenarios, optimizing indexing performance, and setting up syslog daemons. Additionally, it provides insights into the functionalities of tools like Rsyslog and the importance of different log formats.
![2009's RFC5424
<165>1 2003-10-11T22:14:15.003Z host program - - -
[origin ip="192.168.0.1"] hello world
[ structured=data ] octet-count* + LF =
* UDP (RFC5426), TCP (RFC6587), TLS (RFC5425)](https://image.slidesharecdn.com/elasticsearch-logging-kibana-radugheorghe-monitorama-eu-2013-130924215105-phpapp01/85/On-Centralizing-Logs-22-320.jpg)
On Centralizing Logs
- 1. On Centralizing Logs Radu Gheorghe @radu0gheorghe radu.gheorghe@sematext.com @sematext
- 5. Elasticsearch Reason #1: Quick Search No indexing But... =>
- 6. ...and other reasons good write speed lots of tools for logging scales easily
- 8. Stability 1/4: Discovery multicast unicast vs cluster name list of nodes + plugins: EC2, GCE
- 9. Stability 2/4: Preventing Split Brain minimum_master_nodes = N/2 + 1
- 10. Stability 3/4: No OOMs, pls! 1GB ½ total RAM Monitor the requirements SPM for Elasticsearch 20% off with MONEU2013
- 11. Stability 4/4: Field Cache can be changed to index.cache.field.type: soft indices.fielddata.cache.size: X%
- 12. Performance 1/4: Bulk Processing use Bulk API or Bulk UDP API ...translog.flush_threshold_ops
- 13. Performance 2/4: Refresh Interval http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/ default: every second => but every 5s +25% indexing* every 30s +70% indexing*
- 14. Performance 3/4: Timed Indices
- 15. Performance 4/4: Buffers ...index_buffer_size: 30% (YMMV) index.store.type: mmapfs (on 64-bit machines) http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
- 16. Setting Up Kibana as Frontend servers you
- 17. Kibana: Search
- 19. Meet Some Syslog Daemons syslogd traditional everywhere syslog-ng OSE, PE documentation++ config format++ rsyslog OSS only ES output* * http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/
- 20. X-ray of a Modern Syslog Daemon read+buffer file /dev/log … parse syslog formats JSON unstructured data assemble conditionals formatting ... buffer+write file syslog Elasticsearch ...
- 21. 2001's RFC3164: The Semi-Standard <10>Oct 11 22:14:15 host program:hello world TCP + LF = no year, ms, nor TZ little structure
- 22. 2009's RFC5424 <165>1 2003-10-11T22:14:15.003Z host program - - - [origin ip="192.168.0.1"] hello world [ structured=data ] octet-count* + LF = * UDP (RFC5426), TCP (RFC6587), TLS (RFC5425)
- 23. Teaching Old Dog New Tricks RSYSLOG_ForwardFormat (ISO8601 over RFC3164) $MaxMessageSize 2048k log_message_size(2097152) @cee: {"message": "hello world"} @@(o)192.168.0.1 octet-counted framing
- 24. Reliable Transport? Encryption? TCP + TLS (RFC5425) RLTP + TLS RELP + TLS
- 25. Logstash: The Swiss Army Knife inputs (+codecs) filters (parse, modify) outputs (+codecs) lots of plugins => lots of options
- 29. Back to the Beginning Lumberjack Lumberjack Lumberjack Lumberjack syslogd
- 35. rsyslog 1/4: Upgrade to 7.x RPMs or DEBs better performance nicer config format omelasticsearch
- 36. rsyslog 2/4: Faster Inputs UDP increase TimeRequery TCP use imptcp
- 37. rsyslog 3/4: Main Message Queue $MainMsgQueueType FixedArray $MainMsgQueueSize 1000000.... ...or LinkedList or Disk $...DequeueBatchSize 1000 $...WorkerThreads 3
- 38. rsyslog 4/4: Action Queue queue.type="linkedlist" queue.size="1000000" bulkmode="on" # ES specific queue.dequeuebatchsize="1000" queue.workerthreads="3"