Developing Highly Instrumented Applications with Minimal Effort
Download as PPTX, PDF0 likes8,023 views
The document discusses strategies for developing instrumented applications that provide valuable operational and business insights with minimal effort. It emphasizes the importance of quality data, application instrumentation, and logging best practices across various programming environments, including .NET, Java, and Node.js. The presentation aims to equip developers with the necessary tools and methodologies to enhance application monitoring and data capture effectively.
![Supporting Frameworks
The Front
Door
The Inner
Sanctum
Annotations
& Attributes
• ASP.Net
Modules/ActionFi
lters
• Java Servlet
Filters/Spring
Interceptors
• Node.js
interceptors
• Unity Dynamic
Proxies
• Spring
@AspectJ
Pointcuts
• JavaScript
Mixins
• @Instrumentable
• [Instrumentable]
17](https://image.slidesharecdn.com/timhobson-codecamp2013final-131016093240-phpapp01/85/Developing-Highly-Instrumented-Applications-with-Minimal-Effort-17-320.jpg)
Developing Highly Instrumented Applications with Minimal Effort
- 1. Developing highly instrumented applications with minimal effort Tim Hobson Principal Engineer, Chief Caffeine Officer - Intuit
- 2. About Me 2
- 3. Agenda Concepts • Quality Data • Application Instrumentation • Application Logging Theory • Patterns • Supporting Frameworks • Supporting Components Practice • .Net Example • Java Example • Node.js Example GOAL: You should be able to apply this to your application today 3
- 4. Concepts
- 5. If You Do Nothing… OR 5
- 6. There’s No Free Lunch Garbage In, Garbage Out Insight! 6
- 7. Quality In, Insight Out Most apps start with only the framework or app server logging (or nothing!) None of the above is interesting to the business or the developer. You can’t get if you don’t give – there are many ways to give, and many classes of data to provide. 7
- 8. Classes of System Output System Instrumentation • JMX/WMI/SNMP monitoring • Apache/IIS/nginx access logs App Logging App Instrumentation • • • • • • • • Intentional Business Transactions Overtly triggers alerts Source of business metrics • Aids in troubleshooting failures, bugs Cross-cutting (free) App Activity Passively triggers alerts Source of performance data • Source of usage data 8
- 10. Best Practices (© splunk>) Create human readable events Clearly timestamp events Use key-value pairs Be aware of multi-value fields Log unique identifiers 10
- 11. Best Practices (© Tim) Global timestamps (UTC – 2013-08-21 22:43:31,990) Context setting (who/what/where/when/how) Categories/taxonomy (what tier, what component) Timing (time everything!) Security (never log sensitive data: password=***) Consistency in naming – (action=purchase; sale=oct13; productId=123123) – (action=buy; promo=oct13; sku=123123) 11
- 12. Example Output Context 2013-08-21 22:55:36,504; LogLevel=INFO; sid=q3prv41kt511vzojytnx1d42; rid=6500583; userLogin=(null); ipAddress=0.0.0.0; thread=249; category=Web.Controllers.BaseWebController; msg=RequestInfo; server=ws001prod; url=https://myapp.com/account/logon; method=GET; languages=en-US,en;q=0.8; referrer=https://myapp.com/members/dashboard; userAgent=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36; userId=123456; controller=Account; action=Logon; Metrics 2013-08-21 22:55:36,519; LogLevel=INFO; sid=q3prv41kt511vzojytnx1d42; rid=6500583; userLogin=(null); ipAddress=0.0.0.0; thread=249; category=TraceInterceptor; timeTracing=Service.Users.GetUser, time=3; 12
- 13. Patterns Pipeline – Filter – Interceptor Dependency injection – Proxy Pointcuts/aspects 13
- 14. HTTP Pipelines var app = express(); app.use(func1); app.use(func2); app.use(func3); app.use(func4); app.use(func5); app.use(func6); app.use(func7); app.use(func8); app.listen(8080); Java (from Oracle Docs) ASP.Net (from MSFT Docs) 14 express.js
- 15. Dependency Injection • Enforces interface-based programming • Forbids circular references • Lends itself to testability • Flexibility in changing implementations • For our purposes: cross-cutting capabilities 15
- 16. Intercepting Calls With Dynamic Proxies Interceptor Dependency A Implementation DI Container 16 Interceptor Interceptor Proxy Interceptor Interceptor Interceptor Caller Interceptor Proxy Interceptor Interceptor Proxy Dependency B
- 17. Supporting Frameworks The Front Door The Inner Sanctum Annotations & Attributes • ASP.Net Modules/ActionFi lters • Java Servlet Filters/Spring Interceptors • Node.js interceptors • Unity Dynamic Proxies • Spring @AspectJ Pointcuts • JavaScript Mixins • @Instrumentable • [Instrumentable] 17
- 18. Supporting Components .Net • Log4Net • NLog Java node.js • LogBack • SLF4J • Log4J 18 • SenchaLabs Connect • Winston
- 19. Intervention!
- 20. Sample App Intervention 3 Platforms 3 Sample Apps 20 3 Interventions
- 21. Goals Leverage an HTTP pipeline for context Leverage aspects for interception Apply best practices Minimize impact on existing code 21
- 22. MVC Music Store Intervention (Demo) Missing dependency injection/interception framework: Unity Missing logging library: Log4Net, Buche 1. 2. 3. 4. 5. 6. 7. 8. Add DI framework (Unity) Configure aspects (Interception) Configure controller factory, container locator Set up logger (Log4Net) Configure log pattern and targets (Log4Net.xml) Wire up logging interceptor Wire up LogActionFilter and BaseLoggingController Deploy! 22
- 23. Spring Pet Clinic Intervention (Demo) 1. Set up logger a. Configure log pattern (to support context data) b. Configure targets (rotating file) 2. Create LogAspect.java and @Instrumentable attribute a. Capture calling context b. Start/stop timing 3. Create LogInterceptor.java a. Capture request metadata b. Set MDC c. Start/stop timing 4. Configure aspect bean 1. Configure interceptor bean 1. Annotate the methods we care about 23
- 24. NodeCellar Intervention (Demo) Missing Interception Framework: Scarlet Missing Logging Library: Winston 1. 2. 3. 4. 5. 6. Set up Scarlet Configure log pattern Configure method interception Create logging interceptor Create LoggingFilter Deploy! 24
- 25. Closing Thoughts
- 26. Got Log? Now Get Intimate With Your App Ops Dashboards Business Dashboards Quality Assurance Pro-Active Service Degradation Alerting SLA Tracking Security Alerting 2 6 Customer Support Performance Metrics Pre-Release Sanity Testing
- 27. Key Takeaways • It is YOUR responsibility as a developer to provide useful operational and business data. • It is not hard, and most of it is for free once you have the patterns in place. • The same patterns and practices can be applied to practically any platform, and any type of application or service. • When you provide consistent and predictable data others can build on your greatness 27
- 28. Get the Code @hoserdude Spring Pet Clinic Intervention: https://github.com/hoserdude/spring-petclinic-instrumented MVC Music Store Intervention: https://github.com/hoserdude/mvcmusicstore-instrumented NodeCellar Intervention: https://github.com/hoserdude/nodecellar-instrumented 28
- 29. Intuit Speakers @ Silicon Code Camp 2013: SATURDAY 9:45 a.m. - Ramakrishna Kollipara – “Complete Automation of Performance Testing” 1:45 p.m. - Joe Wells - “QBO: Journey From legacy Java app to a Client-side HTML5 app” 3:30 p.m. - Naga Addagadde & Sangeeta Narang – “Intuit APIs for Financial Transaction Aggregation” 5:00 p.m. Ted Drake –“Hitting the Accessibility High Notes with ARIA” SUNDAY 9:15 a.m. - Eugene Krivopaltsev –“Building Native Mobile Apps with Custom Views” 1:15 p.m. - Tim Hobson – “Developing Highly Instrumental Applications with Minimal Effort” For more information about joining our organization visit our booth or connect with our onsite recruiter: Chriscox_recruiter@intuit.com You don't want to miss out on a chance to win this cool headset. Stop by our booth to enter!
- 30. THANK YOU Want to talk more? I’ll be at the Intuit booth today 3-5 PM. 30
Editor's Notes
- #7: Many products promise to provide insights and value from whatever data you throw at them. There’s no doubt they can provide something, but you can’t escape the law of GIGO.
- #8: As a result, it’s on you as a developer, product manager and business to care about the data you are creating. This presentation will demonstrate some best practices on how to go about this, regardless of your platform.
- #9: Now for some definitions. There is a lot of stuff generated by the system – let’s classify the stuff broadly and narrow on what I am going to be talking about.
- #14: Quick review of patterns we’ll use. Interception is the secret sauce here. We want to be able to get in front of every call to every object or component in our system. That lets us see what data is being passed around, who is doing it, and how long it took.
- #19: Indent the bullets