Why proper logging is important
Download as PPTX, PDF1 like1,362 views
The document discusses the importance of proper logging during all phases of development. It describes what syslog is and introduces syslog-ng as the next generation syslog server. It emphasizes that logging should be an integral part of development and developers should consult with operators. The document also discusses logging best practices like using structured, name-value pair logging and centralized logging.
![Free form log messages
• most log messages are: date + hostname + text
Mar 11 13:37:56 linux-6965 sshd[4547]: Accepted
keyboard-interactive/pam for root from 127.0.0.1
port 46048 ssh2
• text = English sentence with some variable parts
• easy to read by a human
Copyright 2013 BalaBit IT Security Ltd.](https://image.slidesharecdn.com/balabitwhyproperloggingisimportant-130208070702-phpapp01/85/Why-proper-logging-is-important-10-320.jpg)
Why proper logging is important
- 1. Why proper logging is important... ...in all phases of development? Péter Czanik community manager Copyright 2013 BalaBit IT Security Ltd.
- 2. About me • Peter Czanik from Hungary • community manager at BalaBit: syslog-ng upstream • BalaBit is an IT security company with HQ in Budapest, Hungary with 100+ developers • part of the openSUSE testing team • openSUSE syslog-ng package maintainer Copyright 2013 BalaBit IT Security Ltd.
- 3. Topics • no, it is not about cutting trees :-) • what is syslog? and syslog-ng? • who uses syslog-ng? • what to log? • free-form messages against name-value pairs • the new buzzword: journal • standardization efforts: CEE/Lumberjack • name-value pairs at work: ELSA Copyright 2013 BalaBit IT Security Ltd.
- 4. What is syslog? • logging: recording events • syslog: - application: collecting events - data: the actual log messages - protocol: forwarding events • history: - originally developed as a logging tool for sendmail - quickly many other apps started to use it • syslog-ng: “next generation” syslog server - since 1997 - focus on central log collection Copyright 2013 BalaBit IT Security Ltd.
- 5. What is syslog-ng • “Swiss army knife” of logging • OSE vs. PE • high performance • more input sources (files, programs, and so on) • more destinations (databases, encrypted net, etc.) • better filtering (not only priority, facility) • processing (rewrite, parse, correlate, and so on) • JSON output and parser • AMQP Copyright 2013 BalaBit IT Security Ltd.
- 6. Who uses syslog-ng? • syslog-ng is the default logging solution in SLES since SLES 10 - Uses 2.0, an ancient version • syslog-ng is the default logging solution in Gentoo • syslog-ng is available in openSUSE (package is maintained by me :-) ) • ...and? Copyright 2013 BalaBit IT Security Ltd.
- 7. Who uses syslog-ng? Copyright 2013 BalaBit IT Security Ltd.
- 8. What to log? • what: everything :-) • in more detail: SANS Top5 log reports: - authentication, change, resource access, etc. • during development logging is often an afterthought - some/many of the above is missing - aids just coding - difficult to debug or audit in production • logging should be an integral part of development - think also about production :-) - consult with operators → DEVOPS!!! - use a similar logging environment, as in production Copyright 2013 BalaBit IT Security Ltd.
- 9. How to log? • short answer: centrally • long: centrally, because: - ease of use: one place to check instead of many - availability: even if the sender machine is down - security: logs are available even if sender machine is compromised Copyright 2013 BalaBit IT Security Ltd.
- 10. Free form log messages • most log messages are: date + hostname + text Mar 11 13:37:56 linux-6965 sshd[4547]: Accepted keyboard-interactive/pam for root from 127.0.0.1 port 46048 ssh2 • text = English sentence with some variable parts • easy to read by a human Copyright 2013 BalaBit IT Security Ltd.
- 11. Why it does not scale? • few logs (workstation) → easy to find information • many logs (server) → difficult to find information • information is presented differently by each application • difficult to process them with scripts • answer: structured logging - Events represented as name value pairs Copyright 2013 BalaBit IT Security Ltd.
- 12. Solution from syslog-ng: PatternDB • syslog-ng: name-value pairs inside - date, facility, priority, program name, pid, etc. • PatternDB parser: - can extract useful information into name-value pairs - add status fields based on message text - message classification • example: an ssh login failure: - user=root, action=login, status=failure - classified as “violation” Copyright 2013 BalaBit IT Security Ltd.
- 13. Journal • the logging component of system • name-value pairs inside: - message - trusted properties - sny additional name-value pairs • native support for name-value pair storage • persistent log storage can be disabled • logs can be forwarded to syslog-ng through a socket • syslog-ng can filter, process logs and forward them to central log server Copyright 2013 BalaBit IT Security Ltd.
- 14. Journal: the enemy? • FAQ: Q: is journal the enemy? A: No! • Journal is local only (syslog-ng: client – server) • Journal does not filter or process log messages • Journal is limited to Linux/systemd (syslog-ng: all Linux/BSD/UNIX) Copyright 2013 BalaBit IT Security Ltd.
- 15. CEE • Journal, syslog-ng, Windows eventlog, rsyslog, auditd, and so on are based on name-value pairs • All use different field names • Standardization is a must: CEE → Common Event Expression • Events: name-value pairs instead of free-form text - Taxonomy: name-value pairs to describe events (example: status) - Dictionary: name-value pairs for event parameters (example: user) • PatternDB can turn free-form messages into CEE Copyright 2013 BalaBit IT Security Ltd.
- 16. Name-value pairs in action: ELSA • ELSA: Enterprise Log Search and Archive • based on syslog-ng, PatternDB and MySQL • simple and powerful web GUI • extreme scalability • patterns focused on network security: - firewalls: Cisco, iptables - IDS: Snort, Suricata, Bro - HTTP, Windows logs, etc. Copyright 2013 BalaBit IT Security Ltd.
- 17. Search Copyright 2013 BalaBit IT Security Ltd.
- 18. Graph Copyright 2013 BalaBit IT Security Ltd.
- 19. Map Copyright 2013 BalaBit IT Security Ltd.
- 20. So, why syslog-ng? • 15 years of open source development • high performance log management • flexible configuration • excellent documentation • PatternDB message parsing Copyright 2013 BalaBit IT Security Ltd.
- 21. Questions? (and some answers) • Questions? • Some useful syslog-ng resources: - Syslog-ng: http://www.balabit.com/network-security/syslog-ng - SANS top5 essential log reports extended: http://chuvakin.blogspot.hu/2010/08/updated-with-community- feedback-sans_06.html - Many books at http://oreilly.com/ - ELSA: http://code.google.com/p/enterprise-log-search-and-archive/ - My blog: http://czanik.blogs.balabit.com/ Copyright 2013 BalaBit IT Security Ltd.
- 22. Thank You! Péter Czanik community manager peter.czanik@balabit.com Copyright 2013 BalaBit IT Security Ltd.