Cleaning The Part: Reclaim your Logging
Download as PPTX, PDF0 likes446 views
The document details a talk by Matt Campbell regarding the challenges and lessons learned in improving logging practices at D2L. He discusses the initial chaotic state of logging, attempts to clean it up, and emphasizes the need for incremental change despite resistance. Key advice includes celebrating small victories and understanding the inevitability of inertia in such transformations.
Cleaning The Part: Reclaim your Logging
- 1. Cleaning the Park: Reclaim your Logging Matt Campbell matthew.campbell@d2l.com @beardedcoder
- 2. Matt Campbell Engineering Director with D2L Leading project to achieve proper web-scale in AWS Previously lead move to monthly deployments matthew.campbell@d2l.com @beardedcoder
- 3. What can I expect from this talk? • Who is D2L and what do they do? • How we ended up with a dirty park • How we tried (and failed) to clean up • How we tried (and failed less) to clean up • What I’ve learned along the way
- 6. Millions of logins daily at peak TBs of aggregate data PBs of aggregate content Clients with Multi- TB DBs
- 16. Dev adds code Code generates log
- 18. Logging • Structured information written out during operation of code. • Can be at various levels (info, debug, warn, error, fatal). • Typically used to debug exceptional code behavior. For the purposes of this talk, when I refer to logging, I mean Error/Fatal level logging (the really bad stuff)
- 23. Users Application Servers Database New Application Servers Migrated DatabasePartially Migrated Database
- 24. Made visible count of logs made each day
- 26. Users Application Servers Database New Application Servers Migrated Database
- 30. Dev and Ops were very separated at the time Tried to solve logging for all things at once Tried to solve for all regions and all instances at once
- 31. They didn’t have a strong why
- 50. Dev adds code Code generates log Dev reviews log and makes change
- 68. Top Tips
- 71. Go for something sticky
- 73. Everyone has inertia to overcome
- 74. It’s OK to acknowledge this will take awhile
- 75. Celebrate the small victories
- 76. Don’t be afraid to start
- 77. You don’t need to know how to finish
- 78. Cleaning the Park: Reclaim your Logging Matt Campbell matthew.campbell@d2l.com @beardedcoder
Editor's Notes
- #2: https://www.soswildlifecontrol.com/wp-content/uploads/2017/01/qtq80-vB3b6f.jpeg
- #5: Who is D2L? What is it that we do? Image Copyright 2016 D2L Inc.
- #6: We’re global https://upload.wikimedia.org/wikipedia/commons/0/09/BlankMap-World-v2.png
- #7: We’re data intensive <See if you can get stats on logins, amount of data, size of DBs>
- #8: In the early days of D2L, there weren’t fancy technologies like ElasticSearch and fancy third-party logging vendors. So we dumped the logs in the spot we had: SQLServer http://1.bp.blogspot.com/-M1MjcnIA7fU/Voz-EKMq5PI/AAAAAAAAL-w/xvywHnJugm8/s1600/rolling-logs.jpg
- #9: When the company was small this worked just fine. Every had access to everything and people worked close (as in physically) together.
- #10: However, as the company grew, people had to specialize and we had to start restricting access to client data. This meant that developers no longer had easy access to the logs that were being generated https://i.ytimg.com/vi/TeYaQGbD6Xc/maxresdefault.jpg
- #11: However, we knew logging was a good thing so we kept adding logging to the code. For a time I imagine devs still had insight (through back channels), but overtime that would have waned.
- #13: However, we knew logging was a good thing so we kept adding logging to the code. For a time I imagine devs still had insight (through back channels), but overtime that would have waned.
- #14: Eventually, we were adding logging because the code we saw had logging in it. And as much as we don’t want to admit it, we copy other code to help get our work done faster. And in copying that code we don’t also think if what we are copying should all remain. So, more and more logging was added. https://i.redd.it/tjltyl5m332z.jpg
- #15: This meant we had 100,000’s of logs monthly (just at an error level) that were rarely looked at. https://upload.wikimedia.org/wikipedia/commons/c/c7/Logs.jpg
- #16: However, all these logs were consuming disc and IOPS on our databases https://img.gawkerassets.com/img/19aanrziuk65tjpg/original.jpg
- #20: https://www.soswildlifecontrol.com/wp-content/uploads/2017/01/qtq80-vB3b6f.jpeg
- #21: https://www.soswildlifecontrol.com/wp-content/uploads/2017/01/qtq80-vB3b6f.jpeg
- #23: To my knowledge we’ve had 3 previous attempts to correct this problem https://i0.wp.com/mobcblog.wpengine.com/wp-content/uploads/2015/09/52.jpg
- #24: The first attempt spun off of my past project, moving D2L to continuous delivery. My manager at the time (and the architect of the CD solution) realized that the amount of noise in the logs meant we couldn’t use them to figure out if we had broken anything while moving the deployment model forward.
- #25: So we built a widget on a dashboard that showed the number of error logs generated that day.
- #26: And nothing happened. While, something happened, we all felt really bad about how many logs we made.
- #27: We hadn’t fixed the main problems blocking us: Simple access to the logs and an aggregated view across all instances.
- #28: This is when the idea of combining all the logs together into a Centralized solution first came about http://cdn1us.denofgeek.com/sites/denofgeekus/files/2016/10/captain-planet-movie.jpg
- #29: http://teamgaffney.com/wp-content/uploads/2016/02/Return-to-Sender.png
- #31: https://i.imgflip.com/y9sav.jpg
- #32: Focused on solving logging for the sake of logging. But no one cared about logging.
- #33: Full disclosure, we’re not actually done yet <gasp> https://fluffrick.files.wordpress.com/2012/01/richmond.jpg
- #34: So why is this time any different?
- #35: We have a stronger why behind the move. This time it is spurred by our large-scale migration into AWS. https://cdn.vox-cdn.com/thumbor/tlEpA9YH7R4FUtj2Bf347AoeP8I=/0x0:308x164/1200x800/filters:focal(95x65:143x113)/cdn.vox-cdn.com/uploads/chorus_image/image/54087325/images.0.jpg
- #36: Moving into AWS provided us with access to technologies that we didn’t have before
- #37: With our move to AWS we also are rearchitecting our application for multi-tenancy. The way we currently log is a blocker to that.
- #38: All of this gave us a stronger vision for the project than just fix logging.
- #50: https://nbchardballtalk.files.wordpress.com/2017/02/field-of-dreams-e1486999847754.jpg?w=1200
- #58: The goal was to still get to 0 logs. With error logs under control we can actually start to treat new error logs as critical events. However there were far too many logs for one team to handle. Even with the new people who came on board once Kibana was out, it wasn’t enough to start making progress. http://i2.cdn.turner.com/money/dam/assets/150109042203-line-graph-ground-breaking-1024x576.png
- #59: Recall that we want to get to 0 error logs so that any logs can be treated as an Incident
- #62: Enter LogDropper. We started an initiative to
- #65: The emotional cycle of change
- #74: http://www.fitcrazy.tv/wp-content/uploads/2016/09/blog-lesson-from-barbell-main.jpg
- #75: http://static.tabatatimes.com/wp-content/uploads/2013/10/Sisyphus-Image-01C.jpg
- #76: https://greatperformersacademy.com/images/images/Articles_images/small-wins-happiness-success.jpg
- #77: https://upload.wikimedia.org/wikipedia/commons/6/63/African_elephant_warning_raised_trunk.jpg
- #79: https://www.soswildlifecontrol.com/wp-content/uploads/2017/01/qtq80-vB3b6f.jpeg