#Fstoco - Monitoring and Instrumentation, why Tracing is Key
0 likes564 views
The document presents a comprehensive overview of monitoring and instrumentation in IT, highlighting the importance of telemetry and various methods for frontend and backend instrumentation. It discusses key metrics, use cases, and best practices for effective logging and log correlation, emphasizing the need for shared responsibility between business and IT. The document also explores current and future trends in transaction correlation and open source solutions for application performance monitoring (APM).
![@jkowall #fstoco
Monkey patching
Wikipedia relevant definition:
In Ruby,[3] Python,[4] and many other dynamic
programming languages... dynamic modifications of a
class or module at runtime, motivated by the intent to
patch existing third-party code as a workaround to a
bug or feature which does not act as desired
Disclaimer : Can be very dangerous, hard to maintain
• Replace methods / attributes / functions at runtime
• Apply a patch at runtime to the objects in memory,
instead of the source code on disk;](https://image.slidesharecdn.com/fstoco-monitoringandinstrumentationstrategiestipsandbestpractices-171020195633/85/Fstoco-Monitoring-and-Instrumentation-why-Tracing-is-Key-24-320.jpg)
![@jkowall #fstoco
Correlation in logs
• Log every transaction segment
• Persist a GUID or transaction ID
• This is very difficult in large teams
• Inefficient to analyze and pull metrics from logs
• Doesn’t work unless you own the code
[code]
PERF,2013-04-03 11:29:52.640,external,0x123456,NA,service1,MyAPP,jimmy,NA,336,NA,NA
INFO,2013-04-03 11:29:53.189,internal,789012,0x123456,service2,TheirApp,jimmy,NA,174,NA,NA
INFO,2013-04-03 11:29:52.892,internal,345678,789012,service3,TheirApp,jimmy,NA,163,NA,NA
[/code]](https://image.slidesharecdn.com/fstoco-monitoringandinstrumentationstrategiestipsandbestpractices-171020195633/85/Fstoco-Monitoring-and-Instrumentation-why-Tracing-is-Key-37-320.jpg)
![@jkowall #fstoco
Transaction correlation and logs!
• Many integrations across APM
and Log vendors
• Can add correlation in code
and use any log tool
– ex: [%X{AD.requestGUID}]
• We auto inject and correlate
(one platform)](https://image.slidesharecdn.com/fstoco-monitoringandinstrumentationstrategiestipsandbestpractices-171020195633/85/Fstoco-Monitoring-and-Instrumentation-why-Tracing-is-Key-38-320.jpg)
#Fstoco - Monitoring and Instrumentation, why Tracing is Key
- 1. @jkowall #fstoco Monitoring and Instrumentation, why Tracing is Key Jonah Kowall VP Market Development and Insights Twitter : @jkowall
- 2. @jkowall #fstoco Jonah Kowall’s background • Over 20 years in IT • Over 15 years working with Infrastructure and Operations enterprises and startups – Security - CISSP, CISA, PCI – Started one of the first content filtering companies • Head of global monitoring at Thomson Reuters • Head of IT Operations at MFG.com – Bezos Expeditions • Gartner Research VP 4 years • Strategy AppDynamics 3 years, acquired by Cisco in March 2017
- 3. @jkowall #fstoco Agenda • Monitoring Fundamentals • Frontend Instrumentation • Backend Instrumentation (Java and .NET) • Backend Instrumentation of interpreted languages (PHP, Python, Node.js) • Transaction Correlation (Tracing) –How it’s done, and where it’s going in commercial and OSS • Logging and Log Correlation
- 5. @jkowall #fstoco Definitions Instrumentation “The design, construction, and provision of instruments for measurement, control, etc; the state of being equipped with or controlled by such instruments collectively.” https://en.wikipedia.org/wiki/Instrumentation Telemetry “Automated communications process by which measurements are made and other data collected at remote or inaccessible points and transmitted to receiving equipment for monitoring.” https://en.wikipedia.org/wiki/Telemetry
- 6. @jkowall #fstoco Software instrumentation types • Metrics – Key value pairs (number/tag) – Can run maths on data • Paths or Topologies – Service – Transaction • Events – Text, such as logs – Can parse and extract metrics… and other values
- 7. @jkowall #fstoco Collecting Telemetry • Pull Collection – Polling APIs - HTTP, SNMP, WMI Server Monitoring System Browser Mobile Server ServerCloudWatch Server Payment API Both methods are scalable and useful for different reasons • Push Collection – Manual code changes – Software agent to attach and extract • Code level or OS Level
- 8. @jkowall #fstoco Priorities of instrumentation…. • Infrastructure • Services • Application • Business by Technologists by the Business • Business • Application • Services • Infrastructure
- 9. @jkowall #fstoco Uplevel the conversation • Understand the customer – Internally and externally • Requirements should be gathered across business and IT teams • Responsibility for definition of monitoring should be shared
- 10. @jkowall #fstoco Business metrics and KPIs • Customer metrics – Conversion between products – Loyalty and retention (churn) – Usage metrics (feature and product) • Sales / marketing metrics – Revenue – Cost of customer acquisition – User flows through applications
- 11. @jkowall #fstoco Technical metrics and KPIs • End to end performance – User through transaction hops – Error isolation • End user experience – Client side errors – Latency per element (page or app) + 3rd party – Client side DNS • Application component performance – Metrics from app server – Metrics from code – Queries – Errors • Intra application component performance
- 12. @jkowall #fstoco Use cases for business and technical data • Usage • Problem identification - MTTI • Problem resolution - MTTR • User satisfaction • Usability • Performance • Change analysis – Code release – A/B testing – data center moves – technology changes
- 14. @jkowall #fstoco Browser Performance APIs 14
- 15. @jkowall #fstoco Resource Timing API 15
- 16. @jkowall #fstoco Navigation Timing API 16
- 18. @jkowall #fstoco Simulating users • Synthetic transactions for: –SLAs –Availability –Baseline performance –DNS –SSL If you try to use it as a barometer for performance you will fail
- 19. @jkowall #fstoco Stop using Synthetic for performance 19
- 20. @jkowall #fstoco Backend instrumentation of Java and .NET
- 21. @jkowall #fstoco Java instrumentation JSR-163 (JavaTM Platform Profiling Architecture) added in Java 1.5 Overloads the default behavior of Java to allow hooks into code for many use cases Since JDK 1.6, for the Oracle HotSpot JVM, a javaagent may be dynamically attached to a running JVM by specifying the process-id (pid).
- 22. @jkowall #fstoco .NET Instrumentation ● Profiling API loaded into the same process as the application process that is being profiled. ● Callback interface (ICorProfilerCallback in the .NET Framework version 1.0 and 1.1, ICorProfilerCallback2 in version 2.0 and later) ● CLR calls the methods in that interface to notify the .NET agent of events in the profiled process ● Profiler can also call into the CLR by using the methods in the ICorProfilerInfo and ICorProfilerInfo2 interfaces to obtain information about the state of the profiled application ● Callbacks are used to inject MSIL (Microsoft Intermediate Language) bytecode into existing application code for instrumentation.
- 23. @jkowall #fstoco Backend instrumentation of interpreted languages
- 24. @jkowall #fstoco Monkey patching Wikipedia relevant definition: In Ruby,[3] Python,[4] and many other dynamic programming languages... dynamic modifications of a class or module at runtime, motivated by the intent to patch existing third-party code as a workaround to a bug or feature which does not act as desired Disclaimer : Can be very dangerous, hard to maintain • Replace methods / attributes / functions at runtime • Apply a patch at runtime to the objects in memory, instead of the source code on disk;
- 25. @jkowall #fstoco PHP instrumentation Zend callback methods zend_execute(…), zend_execute_internal(…) and zend_compile_file(…) so that it can wrap the original implementations with instrumentation code. Handles state changes and new web server initialization (which are PHP instances)
- 26. @jkowall #fstoco Node.js instrumentation ● Wrap methods using before, after and around aspect interceptors. ● Callback along with after, before and around aspect interceptor. ● Notifications when asynchronous calls are complete.
- 29. @jkowall #fstoco Correlation in end to end APM
- 30. @jkowall #fstoco Correlation in asynchronous calls (headache)
- 31. @jkowall #fstoco Correlation in Open Source • OpenZipkin -Automated instrumentation – Brave for Java – JavaScript, .NET Core, Go, Ruby, and many others – Integrated into Pivotal Cloud Foundry for automated tracing (!) 31 But... – Manual instrumentation library for many other languages – No async support, no overhead controls (dependant on how implemented in code)
- 32. @jkowall #fstoco Future of correlation in Open Source • OpenTracing an API (non-standard) for instrumentation – Vendors must implement specific code per instrumentation library :( – Must add code manually – No overhead controls – Few vendors support it, created/backed by Lightstep (former Googlers in stealth) • Stagemonitor (APM for Java and Browser) – Backed by German consulting company – Uses Kibana and Zipkin for split-brain UI • Pinpoint (APM for Java) – Backed by Korean consulting company – Most similar to a commercial APM product • InspectIT (APM for Java and Browser) – Backed by German consulting company – Never spoken to customer/user
- 33. @jkowall #fstoco Graveyard of Open Source • Examples of dead projects – PivotTracing – Spigo • Issues with current OSS APM – Complex – High overhead or cause of performance issues – Primitive sampling for tracing
- 34. @jkowall #fstoco Betting on OpenContext and OpenCensus • OpenCensus https://github.com/census-instrumentation – Created by Google, starting to involve Huawei, Microsoft, Pivotal, Uber, and others – Set of libraries for automated instrumentation • Go, Python, Java, Erlang, PHP – Allows tool standardization (Support for Brave, ZipKin, Google StackDriver) • TraceContext - https://github.com/TraceContext – Plans to create a new standard for headers and tracing – Involving similar people above, but also APM vendors
- 35. @jkowall #fstoco Logging and Log Correlation
- 36. @jkowall #fstoco Logging best practices • Logs are not transaction records • Log errors and exceptions – Easily parsed (JSON) – Time (long), Source – Write your own identifiers for each statement logged (or instrument and inject) • Think about security implications (plain text, on disk, syslog are all insecure) • Keep log statements small (thanks Java, .NET…) • Every log statement introduces overhead, so don’t overdo it • Exception logs create even more overhead, so fix them • Do not try to use log tools as metric stores
- 37. @jkowall #fstoco Correlation in logs • Log every transaction segment • Persist a GUID or transaction ID • This is very difficult in large teams • Inefficient to analyze and pull metrics from logs • Doesn’t work unless you own the code [code] PERF,2013-04-03 11:29:52.640,external,0x123456,NA,service1,MyAPP,jimmy,NA,336,NA,NA INFO,2013-04-03 11:29:53.189,internal,789012,0x123456,service2,TheirApp,jimmy,NA,174,NA,NA INFO,2013-04-03 11:29:52.892,internal,345678,789012,service3,TheirApp,jimmy,NA,163,NA,NA [/code]
- 38. @jkowall #fstoco Transaction correlation and logs! • Many integrations across APM and Log vendors • Can add correlation in code and use any log tool – ex: [%X{AD.requestGUID}] • We auto inject and correlate (one platform)