Real time monitoring-alerting: storing 2Tb of logs a day in Elasticsearch
2 likes925 views
The document discusses the implementation of real-time monitoring and log storage using Elasticsearch in a microservices architecture, detailing various elements of observability, data ingestion processes, and performance metrics. It includes technical details about Elasticsearch's features, cluster architecture, data hierarchy, index management, and alerting mechanisms. Additionally, it highlights best practices and lessons learned in maintaining a highly available and efficient logging system.
![@aliostad
/// bulk api
âą Always use Bulk API to index documents
âą Batches of 1K-5K documents
âą Watch-out for error 429 and back-oïŹ pattern
âą Check bulk rejects [change bulk queue length]](https://image.slidesharecdn.com/real-timemonitoring-alertingstoring1tboflogsadayinelasticsearch-170523143229/85/Real-time-monitoring-alerting-storing-2Tb-of-logs-a-day-in-Elasticsearch-40-320.jpg)
Real time monitoring-alerting: storing 2Tb of logs a day in Elasticsearch
- 1. >>> Realtime Monitoring Storing 2TB of logs a day in Elasticsearch @aliostad Ali Kheyrollahi, ASOS
- 2. @aliostad
- 3. @aliostad The joy of hitting F5
- 4. @aliostad The joy of a single process
- 5. @aliostad The joy of a having a production-size database locally
- 6. @aliostad The joy of having a dev machine build running all services
- 7. @aliostad /// What if your systems is âMicroservicesâ?
- 8. @aliostad
- 9. @aliostad - +40 platform teams - 1oos of microservices - some services >10k rps
- 10. @aliostad > stackoverflow > ÂŁ1.5 bln global fashion destination > 35% year-on-year
- 12. @aliostad /// observability >>> Control Theory âa measure for how well internal states of a system can be inferred by knowledge of its external outputsâ
- 14. @aliostad Tracing Logging Credit: Peter Bourgon Events Aggregations Request Scope Telemetry Alerting /// Scope
- 15. @aliostad Logging Telemetry Tracing Alerting Log4Net â Time-series DBs â â â Zipkin â â â â Prometheus â â â Elasticsearch â â â â New Relic* â â â Circonus* â â â * paid services /// comparison
- 16. @aliostad 1 2 3 At source (perf counters) At the storage (Circonus) In the visualisation tool (Kibana) /// aggregations 4 In the pipeline (Riemann)
- 18. @aliostad /// use cases âą Metrics (Visualisation) âą CPU, number of errors âą Response time percentiles âą Full-text search capability (logs and errors) âą Correlating across services âą Alerting when there is an SLO breach
- 19. @aliostad /// azure logs âą Azure Diagnostics (WADLogs table) âą IIS logs âą VM Windows Event Logs âą Performance Counters (standard + custom)
- 20. @aliostad /// application logs Microservice ETW SLAB Azure Table Sink ETW Application Logs EC e.g. CRIT_ORD_API_DatabaseDown
- 21. @aliostad /// instrumentation logs Microservice Perf Counters SLAB Azure Table Sink ETW Instrumentation Logs Azure Performance Counter Logs PerfIt Azure Agent
- 23. @aliostad /// pull vs push
- 25. @aliostad /// ConveyorBelt Performance Counters ConveyorBelt Azure WAD logs ETW Logs Elasticsearch Instrumentation Logs IIS Logs Woodpecker Outputs (Pull Logs) Sources Config Up to 2TB/day
- 26. @aliostad /// ConveyorBelt Source Source Source ConïŹg Scheduler Parser units of work To Elasticsearch Source Actor Actor Actor
- 29. @aliostad /// elasticsearch âą Linearly-scalable and HA* search (and visualisation) âą ELK Stack âą Open Source (enterprise features require license) âą Speaks JSON âą REST API and very developer-friendly
- 30. @aliostad /// cluster âą Cluster: No ZK âą Gossip / discovery âą Node type: âą master - leader election âą data âą client
- 31. @aliostad /// data hierarchy âą Index âą Shard âą Replica âą Type/Mapping âą Document: JSON, immutable,âš versioned INDEX MAPPING MAPPING MAPPING ⊠Document Document Document âŠ
- 32. @aliostad /// data types âą JSON data types: bool, long, ïŹoat, string*, datetime âą Array? âą String tokenisation/analysers âą Best of both world? âą Object âą nested { âaâ: { âbâ : { âcâ: 42 } } }
- 33. @aliostad /// doc operations âą Upsert âą Delete âą Partial Update âą Search (JSON-based query DSL)
- 36. @aliostad /// index shard/replica Write Read Master Shard Shard Replica Replica Index Index
- 37. @aliostad /// index âą Daily indices âą Hot/Cold with index alias âą Creation => templates âą Settings: âą refresh_interval
- 38. @aliostad /// mapping/type âą Schema âą How many mappings per index? âą Dynamic mapping âą Operations âą Upsert âą Delete
- 39. @aliostad /// templates PUT https://es_cluster:9200/_template/my_template { âtemplateâ: âmy_index_*â, âsettingsâ: {âŠ} âmappingsâ: { âmapping_1â: {âŠ}, âmapping_2â: {âŠ} } }
- 40. @aliostad /// bulk api âą Always use Bulk API to index documents âą Batches of 1K-5K documents âą Watch-out for error 429 and back-oïŹ pattern âą Check bulk rejects [change bulk queue length]
- 43. @aliostad /// resources node type âą Data: Disk, RAM, CPU, Network âą Master: CPU, Network, (RAM) âą Client: Network, CPU, (RAM) âą Kibana: CPU, Network, RAM
- 44. @aliostad /// simple data/master/kibana âą CPU âą RAM âą Disk âą Network
- 45. @aliostad /// next level d a t a / m a s t e r kibana
- 46. @aliostad c l i e n t k i b a n a m a s t e r d a t a trafïŹc trafïŹc
- 47. @aliostad 3x client 2x kibana 20x data (hot) trafïŹc trafïŹc 10x data (warm) /// our setup 3x master ARM Template Desire State ConïŹguration
- 48. @aliostad /// hot/warm âą Hot => CPU, Warm => Memory âą Index Allocation/Routing âą At the index:âš "index.routing.allocation.require.box_type" : "warm" âą At the node (elasticsearch.yml)âš box_type: warm
- 49. @aliostad /// security âą x-pack: SSL + username/password security (basic, Kerberos) âą No Federated Authentication âą Proxy (nginx, apache, etc) âą IP-whitelisting
- 50. @aliostad /// administration âą Like all: logs, slow query logs, etc âą top, htop, iostat âą collectd + local logstash âą two clusters, each watching the other âą curator for hot/cold and deleting old indices
- 52. @aliostad /// ActivityId Microservice Id IdId Thread Local Storage Id To Other APIs Id Event
- 54. @aliostad /// watcher âą Trigger âą Input âą Condition âą Action
- 55. @aliostad /// watcher notes âą All watches get executed on the active master âą Use Action Throttling to limit alerts âą Use watch templates when you see common patterns âą Use transforms and metadata to include context in actions/emails
- 57. @aliostad /// Do you speak CAP? âą Consistency? Treat all data dispensable. Back up data that gets mastered in Elasticsearch. Not a document db. âą Highly-Available? For >99.9% availability use redundancy âą Partition-Intolerance? Node intercommunication highly chatty, ideally keep in the same data centre and even in the same VPC (aws)/VNet (azure)
- 58. @aliostad /// Beware âą Split brain common âą Data corruption possible âą Backup data that gets mastered in ES (kibana indices) âą It seems safest High Availability is redundancy (expensive)
- 60. @aliostad Credits âą Picture: Embroidery thread macro - https://www.ïŹickr.com/photos/39908901@N06/ âą Picture: Calculate Red - https://www.ïŹickr.com/photos/93277085@N08/10398245145 âą Picture: 1950's wristwatch workings - https://www.ïŹickr.com/photos/ 134832191@N08/27301612554/ âą Picture: Tokyo Tower_58 - https://www.ïŹickr.com/photos/ajari/2756645901 âą Picture: 1Bamboo and Rust - https://www.ïŹickr.com/photos/hammershaug/5816522126/ âą Picture: IMG_1899 - https://www.ïŹickr.com/photos/johnas/9650255412/ âą Picture: fan2 https://www.ïŹickr.com/photos/sidelong/444054290/ âą Picture: Glass jar ïŹlled with pasta https://www.ïŹickr.com/photos/76588981@N02/16766079567/ âą Picture: Rusty cogs https://www.ïŹickr.com/photos/paperpariah/25375888671/ âą Picture: Do you see the world in different colours? https://www.ïŹickr.com/photos/luopl/6012467435/ âą Picture: danger https://www.ïŹickr.com/photos/armydre2008/9650951334/ âą Link: ETW equivalent for Linux http://blogs.microsoft.co.il/sasha/2017/04/02/tracing-net-core-on-linux- with-usdt-and-bcc/