What uses for observing operations of Configuration Management?
0 likes357 views
The document discusses the complexities and importance of observability in configuration management, focusing on the various roles and needs of operators and experts. It emphasizes the use of event sourcing and tracing to understand and monitor infrastructure states, as well as the need for integrating logging and expert insights to improve diagnostics and compliance tracking. Additionally, it outlines how Rudder centralizes information for better decision-making while addressing security challenges related to data visibility and permissions.
![Event sourcing & Tracing
51
Temporal relationships between Spans in a single Trace
––|–––––––|–––––––|–––––––|–––––––|–––––––|–––––––|–––––––|–> time
[Span A···················································]
[Span B··············································]
[Span D··········································]
[Span C········································]
[Span E·······] [Span F··] [Span G··] [Span H··]
https://opentracing.io/specification/](https://image.slidesharecdn.com/nicolas-190205180242/85/What-uses-for-observing-operations-of-Configuration-Management-51-320.jpg)
What uses for observing operations of Configuration Management?
- 1. rudder.io What uses for observing operations of Configuration Management? Nicolas CHARLES nicolas@rudder.io - @nico_charles 1
- 2. Are we really looking at logs? 2 I’m sure everyone here does, but...
- 3. No error nor change in logs means success? 3 Aren’t we missing something?
- 4. Getting and understanding the info is complex 4 Operators, Managers, Experts, APIs have differents needs Frustration if we need a third party to get data We mistrust what we don’t understand
- 5. Getting and understanding the info is complex Putting errors into perspective Errors can be expected Errors in production can have catastrophic consequences Errors in a Vagrant VM is much less critical
- 6. Getting and understanding the info is complex Strong reliance on Expert(s) SPOF Fatigue
- 7. Knowing the exact infrastructure state monitoring observabilityVS
- 8. Observability adoption Databases Built in facilities Tooling ecosystem to extract knowledge
- 9. Observability adoption Software Legacy: embedding agent (often proprietary solutions) New developments: Best practices Open standards Architectural bricks
- 10. These concepts are core to Rudder Everyone/thing can be an actor of configuration management
- 11. These concepts are core to Rudder Technique A set of operations & configurations to reach a state With variables for configuration Created by experts
- 12. These concepts are core to Rudder
- 13. These concepts are core to Rudder Directive Technique + Parameters Defines how services must be managed Driven by business needs, managed by admins or APIs
- 14. These concepts are core to Rudder Rule The application of Directive(s) to Group(s) Defines the targets of the Directive(s) Higher approach of services, managed by admins or APIs
- 15. Each can focus on what is relevant 15 Operators Security Experts
- 16. Each can focus on what is relevant 16 Managers APIs "rules": [ { "id": "32377fd7-02fd-43d0-aab7-28460a91347b", "name": "Security rules - baseline", "compliance": 100, "mode": "full-compliance", "complianceDetails": { "successAlreadyOK": 87.47, "successNotApplicable": 12.53 }, "directives": [ { "id": "c16e3a90-b9d7-427d-83c1-d80e33124e4c", "name": "CIS Benchmark 2.1.6 - rsh", "compliance": 100.0, "complianceDetails": { "successAlreadyOK": 100.00 }
- 17. What is this compliance? PARAM RULE ● Id DIRECTIVE ● Id ● (Components) GROUP ● Id RUDDER config (global) ● Policy Mode ● Schedule NODE ● Properties ● Policy Mode ● Schedule Environmental context ● Id : . . . ● Generated : . . . Files Node configuration Change request Historisation Historization Event logs
- 18. What is this compliance? RUDDER config (global) ● Policy Mode ● Schedule NODE ● Properties ● Policy Mode ● Schedule Environmental context ● Id : . . . ● Generated : . . . Files Node configuration Change request Historisation Event logs PARAM RULE ● Id ● Groups + Directives DIRECTIVE ● Id ● Components GROUP ● Id Historization
- 19. What is this compliance? PARAM RULE ● Id DIRECTIVE ● Id ● (Components) GROUP ● Id RUDDER config (global) ● Policy Mode ● Schedule NODE ● Properties ● Policy Mode ● Schedule Environmental context ● Id : . . . ● Generated : . . . Files Node configuration Change request Historisation Historization Event logs
- 20. What is this compliance? PARAM RULE ● Id DIRECTIVE ● Id ● (Components) GROUP ● Id RUDDER config (global) ● Policy Mode ● Schedule NODE ● Properties ● Policy Mode ● Schedule Environmental context ● Id : . . . ● Generated : . . . Files Node configuration Change request Historisation Historization Event logs
- 21. What is this compliance? PARAM RULE ● Id DIRECTIVE ● Id ● (Components) GROUP ● Id RUDDER config (global) ● Policy Mode ● Schedule NODE ● Properties ● Policy Mode ● Schedule Environmental context ● Id : . . . ● Generated : . . . Files Node configuration Change request Historisation Historization Event logs
- 22. What is this compliance? 22 ● Id : . . . ● Generated : . . . Files Node configuration RUN ● Reports ● Reports ● ... ● ... METADATA ● node id ● config id ● run timestamp RUN ● Reports ● Reports ● ... ● ... METADATA ● node id ● config id ● run timestamp ● Signature Get Policy Send configuration reports Expected reports (node id, config id, timestamp) Run reports Historization Compliance historized Store expected reports Metadata ● Integrity ● Signature Config ● Id ● For Rule R, Directive D1, Component C
- 23. What is this compliance? 23 ● Id : . . . ● Generated : . . . Files Node configuration Run reports RUN ● Reports ● Reports ● ... ● ... METADATA ● node id ● config id ● run timestamp RUN ● Reports ● Reports ● ... ● ... METADATA ● node id ● config id ● run timestamp ● Signature Get Policy Send configuration reports Expected reports node id config id timestamp end of validity Historization Compliance historized Store expected reports Metadata ● Integrity ● Signature Config ● Id ● For Rule R, Directive D1, Component C
- 24. What is this compliance? 24 ● Id : . . . ● Generated : . . . Files Node configuration RUN ● Reports ● Reports ● ... ● ... METADATA ● node id ● config id ● run timestamp RUN ● Reports ● Reports ● ... ● ... METADATA ● node id ● config id ● run timestamp ● Signature Get Policy Send configuration reports Expected reports (node id, config id, timestamp) Run reports Historization Compliance historized Store expected reports Metadata ● Integrity ● Signature Config ● Id ● For Rule R, Directive D1, Component C
- 25. What is this compliance? 25 ● Id : . . . ● Generated : . . . Files Node configuration RUN ● Reports ● Reports ● ... ● ... METADATA ● node id ● config id ● run timestamp RUN ● Reports ● Reports ● ... ● ... METADATA ● node id ● config id ● run timestamp ● Signature Get Policy Send configuration reports Expected reports (node id, config id, timestamp) Run reports Historization Compliance historized Store expected reports Metadata ● Integrity ● Signature Config ● Id ● For Rule R, Directive D1, Component C
- 26. What is this compliance? 26 ● Id : . . . ● Generated : . . . Files Node configuration RUN ● Reports ● Reports ● ... ● ... METADATA ● node id ● config id ● run timestamp RUN ● Reports ● Reports ● ... ● ... METADATA ● node id ● config id ● run timestamp ● Signature Get Policy Send configuration reports Expected reports (node id, config id, timestamp) Run reports Historization Compliance historized Store expected reports Metadata ● Integrity ● Signature Config ● Id ● For Rule R, Directive D1, Component C
- 27. Make information available 27 A lot information from inside Rudder, usable in Rudder context Details of each run (timestamped info) Policy generation details Serialization of configurations Inventories ...
- 28. Causality and dependencies of events 28 Why would we need it? ● We have logs ● We have experts
- 29. Causality and dependencies of events 29
- 30. Causality and dependencies of events 30 Diagnostic on infrastructures is hard ● Many systems ● Dependencies across systems ● Many actors involved An issue on one component can impact hundred systems We need to separate the causes from the symptoms
- 31. Causality and dependencies of events 31 Monitoring can only correlate Causes and precedences help root cause analysis
- 32. Causality and dependencies of events 32 How can we do that ??!??
- 33. Event sourcing & Tracing 33 Events happen on the whole infrastructure Describe and analyze over systems Order events Contextualize
- 34. Event sourcing & Tracing 34 Terminology (Dapper & OpenTracing) Trace: Description of a “transaction” as it moves through systems Span: Named and timed operation, piece of workflow (+ tags and logs) Span context: Trace information that accompanies the transaction
- 35. Event sourcing & Tracing 35 What’s in a span? Operation name Start & end timestamps Tags: Set of key:value Logs: Set of key:value SpanContext
- 36. Event sourcing & Tracing 36 Temporal relationships between Spans in a single Trace https://www.jaegertracing.io/docs/1.9/architecture/
- 37. Event sourcing & Tracing 37 What would be the traces? Defining the infrastructure state is a trace Each changes before validation is a span Validating results in a change request closes the trace Computing the nodes configurations is a trace Computing targets, overrides and generating files are spans Closes with the serialization of the nodes configurations in database Each run on an node is a trace Each configuration check is a span
- 38. Event sourcing & Tracing 38 RULE ● Id DIRECTIVE ● Id GROUP ● Id Environmental context ● Id : . . . ● Generated : . . ● Commit id. Files Node configuration Change request RUN ● Reports ● Reports ● ... ● ... METADATA ● node id ● config id ● run timestamp RUN ● Reports ● Reports ● ... ● ... METADATA ● node id ● config id ● run timestamp ● Signature Get config Send configuration reports Expected reports (node id, config id, timestamp) Run reports Historisation Store expected reports Metadata ● Integrity ● CommitId ● Signature Config ● For Rule R, Directive D1, Component C Events Commit Id Defining state Trace + Spans Trace Run: Trace Each step: span Message bus
- 39. Event sourcing & Tracing 39 ● Id : . . . ● Generated : . . ● Commit id. Files Node configuration METADATA ● node id ● config id ● run timestamp RUN METADATA Signature Get config Send configuration reports Expected reports (node id, config id, timestamp) Run reports Store expected reports Metadata ● Integrity ● CommitId ● Signature Config ● For Rule R, Directive D1, Component C Trace Message bus Run: Trace Each step: span Compliance CMDB Hooks Monitoring
- 40. Event sourcing & Tracing 40 Store Traces & Events: ● Integrate with systems in place ● Many tools are compatible with OpenTracing Correlate with non-observable systems
- 41. Closing thoughts 41 With Rudder, information is centralized and made available in a relevant way for all actors/things
- 42. Closing thoughts 42 How can you benefit more of your configuration management?
- 43. Closing thoughts 43 What can we do of these billions events?
- 44. Closing thoughts 44 What can we do of these billions events? Reactive approach Query, search and analyze traces in case of problems
- 45. Closing thoughts 45 What can we do of these billions events? Proactive approach Process mining: Machine Learning on these events Detect unusual behaviours Outliers Inconsistencies across systems
- 46. Closing thoughts 46 Mark Burgess Founder of Configuration Management http://markburgess.org/anomalies.htm l
- 47. rudder.io Questions ? Nicolas CHARLES nicolas@rudder.io - @nico_charles 47
- 48. Security? 48 Events, trace and logs hold critical data Within a unique system, security can be built-in AuthN/AuthZ For distributed system, it’s much harder Who can see what? Who defines and enforces the authorizations? Tags on events for authorizations
- 49. Security? 49 Events, trace and logs hold critical data Cipher information vs partial visibility?
- 50. rudder.io What uses for observing operations of Configuration Management? Nicolas CHARLES nicolas@rudder.io - @nico_charles 50
- 51. Event sourcing & Tracing 51 Temporal relationships between Spans in a single Trace ––|–––––––|–––––––|–––––––|–––––––|–––––––|–––––––|–––––––|–> time [Span A···················································] [Span B··············································] [Span D··········································] [Span C········································] [Span E·······] [Span F··] [Span G··] [Span H··] https://opentracing.io/specification/
- 52. Event sourcing & Tracing 52 Every components need to know the context ● Carry the Span Context along each events Add some information for each events ● Save on logging thanks to context Send these traces on message bus