Application logging for fun and profit
Download as PPTX, PDF0 likes616 views
This document discusses logging in applications for debugging and monitoring purposes. It covers why logging is useful, how to implement logging using frameworks like Log4j and SLF4J, best practices for log messages, and how to analyze log data to learn about application usage and debug issues. Tools like grep, awk and tail can be used to query and analyze log files. Rotating logs and using a retention policy is also recommended.
![©2010 Improving Enterprises, Inc.
Logging [vs Debugging]
Set up once, always ready to use
Minimum disruption to the flow
Works locally and in the cloud
Faster data collection
Collected data is persisted](https://image.slidesharecdn.com/loggingforfun-aa-130820190236-phpapp01/85/Application-logging-for-fun-and-profit-5-320.jpg)
![©2010 Improving Enterprises, Inc.
Example layout
log4j.appender.stdout.layout.ConversionPattern=
%d %5p [%t] (%F:%L) - %m%n
- Date
- Log Level
- Thread name
- File and line (slow to retrieve)](https://image.slidesharecdn.com/loggingforfun-aa-130820190236-phpapp01/85/Application-logging-for-fun-and-profit-13-320.jpg)
Application logging for fun and profit
- 1. ©2010 Improving Enterprises, Inc. Logging For Fun and Profit Jane Prusakova Jane.Prusakova@ImprovingEnter prises.com http://softwareandotherthings.blog spot.com Improving Enterprises AA.com 2013
- 2. ©2010 Improving Enterprises, Inc. Why How-to Using the results
- 3. ©2010 Improving Enterprises, Inc. Why logging? Enquiring minds want to know! Avoid programming-by-coincidence Trace what the users do See effects of Integration Multithreading High load
- 4. ©2010 Improving Enterprises, Inc. When to use logging? Greenfield development Brownfield development what came before the effect of changes Maintenance and troubleshooting trace bugs Bonus Evolve apps based on users behavior
- 5. ©2010 Improving Enterprises, Inc. Logging [vs Debugging] Set up once, always ready to use Minimum disruption to the flow Works locally and in the cloud Faster data collection Collected data is persisted
- 6. ©2010 Improving Enterprises, Inc. How-to Using the results
- 7. ©2010 Improving Enterprises, Inc. Yes, I want to setup logging! Pick a framework Proper configuration Consider ways to parse logs Logging will change with the application
- 8. ©2010 Improving Enterprises, Inc. Log4J SLF4J Apache MIT license Logging framework Logging facade Configuration, log levels and categories, rotate log files, thread-safe logging. Support: online documentation, tutorials, online forums.
- 9. ©2010 Improving Enterprises, Inc. Where to write logs? Console File system DB Queue mechanisms Combination
- 10. ©2010 Improving Enterprises, Inc. Database Files Easy to parse, hard to evolve Very hard to parse, easy to change Uses DB connections Requires FS access Can cause app crash, loss of logs, slow down app Can fill up space
- 11. ©2010 Improving Enterprises, Inc. Configuring Log4J Java properties-style XML Programmatically Hard-coded Takes preference
- 12. ©2010 Improving Enterprises, Inc. Log Message What to log with the message Importance level When Where in the code Data being processed Thread information
- 13. ©2010 Improving Enterprises, Inc. Example layout log4j.appender.stdout.layout.ConversionPattern= %d %5p [%t] (%F:%L) - %m%n - Date - Log Level - Thread name - File and line (slow to retrieve)
- 14. ©2010 Improving Enterprises, Inc. Enough and but not too much Logging should support in code correctness readability performance Consider making logs easier to use
- 15. ©2010 Improving Enterprises, Inc. Best Practices Relate each message to generating point Code Data Format messages for readability Use .ToString on objects
- 16. ©2010 Improving Enterprises, Inc. More Best Practices Input and output Errors and exceptions Transitions Integration points Thread rendezvous
- 17. ©2010 Improving Enterprises, Inc. Log input and output public int calculateCost(String user, Flight flight) { log.debug(“calculateCost: user “ + user + “: “ + flight); int cost = … ; … // cost calculation log.debug((“calculateCost: cost for “ + user + “ on “ + flight + “: “ + cost); return cost; }
- 18. ©2010 Improving Enterprises, Inc. Log errors and exceptions // do dangerous work result = DoDangerousWork(user, flight); } catch (Exception e) { log.error(“methodName: exception in DoDangerousWork for user “ + user + “ on “ + flight, e); // recover or re-throw }
- 19. ©2010 Improving Enterprises, Inc. More Log destination, parameters, outcomes DB calls SOAP calls Wait time Thread joins Available memory Memory-intensive operations Intermediate calculation results For complicated calculations
- 20. ©2010 Improving Enterprises, Inc. Practices to avoid Little information … // step 1 Logger.info(“did something”); … // ste 2 Logger.info(“did more work”); foreach (…) { … // useful work Logger.info(“working hard”); }
- 21. ©2010 Improving Enterprises, Inc. Watch for logging-caused bugs Side effects Logger.info(count++ “: did something”); Errors and exceptions Logger.info(m.GetValue());
- 22. ©2010 Improving Enterprises, Inc. More practices to avoid DRY principle applies
- 23. ©2010 Improving Enterprises, Inc. Using the results
- 24. ©2010 Improving Enterprises, Inc. You’ve got logs! Rotate logs: by size, by time Retention policy 2-level storage Provide access - read only! Transfer and analysis Warning: large datasets
- 25. ©2010 Improving Enterprises, Inc. Security and performance Moving and processing can impact performance Absolutely no sensitive data
- 26. ©2010 Improving Enterprises, Inc. Learning from the logs Never rely on eye-balling WYS is not WYG Aggregate data Distinguish common and rare
- 27. ©2010 Improving Enterprises, Inc. Tools Process datasets: Line by line grep, Perl, awk, cut, tail, head Load entire file to eyeball vi, notepad++ Better way to eyeball more, less
- 28. ©2010 Improving Enterprises, Inc. Learning from the logs > grep userName MyApplication.log grep methodName MyApplication.log grep threadName MyApplication.log tail -2000 VeryImportant.log | grep Exception grep FATAL MyApplication.log
- 29. ©2010 Improving Enterprises, Inc. Querying logs using Awk GNU Awk www.gnu.org/software/gawk create histograms Wait times Number of calls Exceptions per time period or per user Compare and relate events Exception stacks traces Outcomes by caller or thread
- 30. ©2010 Improving Enterprises, Inc. Logging For Fun and Profit Jane Prusakova Jane.Prusakova@ImprovingEnter prises.com http://softwareandotherthings.blog spot.com Improving Enterprises AA.com 2013
Editor's Notes
- #3: Shilpa: Without proper planning and clear strategy, the log messages produced by the system may not be useful for problem resolution.Effective logging is a balance between logging enough data to debug problems, while not dumping so much it confuses the issue.
- #7: Shilpa: Without proper planning and clear strategy, the log messages produced by the system may not be useful for problem resolution.Effective logging is a balance between logging enough data to debug problems, while not dumping so much it confuses the issue.
- #9: Log4J is old, excellent, widely used framework.SLF4J is written by the same author as Log4J. Both licenses permit for-profit usage.
- #10: Both Log4net and Nlog are open source, free libraries. Well-used and well-loved, mature technologies, that are known to work very well. No option to buy corporate support, but there is plenty of free support from the user community available online.
- #11: Console – for debugging purposes onlyWhile I prefer to log to a file: More flexibility to change what is being loggedLogging doesn’t compete with the application for DB connectionsDB access is broken more often than file accessWhy you may choose to log to the DB: Easier to parse logsIt is possible to use Aspects to separate logging concerns from the application logic - for logs that are stable. If you use logging to trace problems as the application is being developed and maintained – putting logs with the code is simpler and helps to shorted the development cycle.
- #12: While I prefer to log to a file: More flexibility to change what is being loggedLogging doesn’t compete with the application for DB connectionsDB access is broken more often than file accessWhy you may choose to log to the DB: Easier to parse logsIt is possible to use Aspects to separate logging concerns from the application logic - for logs that are stable. If you use logging to trace problems as the application is being developed and maintained – putting logs with the code is simpler and helps to shorted the development cycle.
- #13: Java properties is older, XML format can be more convenient since lots of other configuration is also done in XML.Whichever configuration file is found first on the file path is loaded and used. Configuring logging in code is generally a bad idea – any changes will require a re-compile and re-install.
- #14: Where in the code can be specified approximately, by outputting class, or in details, with file name, full class path, line number of the log request. Depending on the situation, it maybe helpful to have: Method caller informationthread diagnostic context-additional information on the state of the calling codeEach logging event should generate 1 line in the log file – makes for easier log processing.
- #15: While I prefer to log to a file: More flexibility to change what is being loggedLogging doesn’t compete with the application for DB connectionsDB access is broken more often than file accessWhy you may choose to log to the DB: Easier to parse logsIt is possible to use Aspects to separate logging concerns from the application logic - for logs that are stable. If you use logging to trace problems as the application is being developed and maintained – putting logs with the code is simpler and helps to shorted the development cycle.
- #16: Go back to examples
- #17: Go back to examples
- #18: Go back to examples
- #19: Go back to examples
- #21: Unless step 1 or step 2 throw, we already know both will happen – no need to log the second.When putting loggin inside a loop, it’s helpful to record loop counter and the item being processed.
- #22: Make sure logging operation itself will not add errors.Wrapping logging code in try-catch adds lines, making code less readable, and overhead. So, test!
- #23: It is important that logs are dense with information. If logging an object with no ToString method, it’s full type name will be printed – bad.Frameworks provide a way to post file and line from where logging is done – no need to do it in text. It’s fine to reduce logging as the code becomes stable and appears to have no more bugs.
- #24: Shilpa: Without proper planning and clear strategy, the log messages produced by the system may not be useful for problem resolution.Effective logging is a balance between logging enough data to debug problems, while not dumping so much it confuses the issue.
- #25: How to handle large log files,e.g Roll the files.... (zip them up after certain days etc....)
- #26: Programs that experience issues tend to log more, causing higher log volume, more stress on the disk, and more frequent log rotation. Which could make the situation worse. Watching log’s rate of growth can provide information on the health of the application.
- #27: A rare bug is harder to trace – and usually presents less of a problem to the business. When arbitraging future work, use logs to learn what users want.
- #28: Cygwin-provided Unix utilities work great on Windows
- #29: Target your research: Trace by user, by what happens in a particular piece of code, or what is logged at a particular level.
- #30: A rare bug is harder to trace – and usually presents less of a problem to the business. When arbitraging future work, use logs to learn what users want.