Briefing Room analyst comments - streaming analytics
3 likes893 views
The document discusses the evolving landscape of data management and streaming analytics, emphasizing the need for adaptable data architectures to handle diverse event cycles and sources. It critiques traditional ETL processes and their limitations in real-time data ingestion and management. Third Nature, the author, aims to assist organizations in leveraging data-driven strategies and emerging technologies for enhanced analytics and performance management.
Briefing Room analyst comments - streaming analytics
- 8. Copyright Third Nature, Inc. We have a model for the persisted portion only The DW can’t handle real time ingest ▪ One of the original DW design assumptions: solve for conflicting workloads by using a different database ▪ Workload management has limits ▪ Scalability problem for event streams ▪ Spiky flow patterns and dynamic scaling Static schema: ▪ Reaction time ‐ shapes, holes, dropped packets ▪ What happens first, upstream change or data model change? Polling architectures do not work well for streaming ▪ Introduces latency ▪ Polling creates performance and scaling problems
- 10. Copyright Third Nature, Inc. Flowing Persisted Sliding window of “now” Persisted but not yet loaded into a platform Queryable history Managed history Streaming isn’t either‐or, it’s part of core architecture A DB or ETL can get you to within minutes (at large scale) but it won’t be easy or cheap; mainly lives in the realm of history Event streams, in-mem stores, CEP streaming SQL can be used for these Real time monitoring doesn’t use only real time data: windows, restarts, detecting deviation, so the above boundaries are crossed. ESB Cache/Queue Database / platform
- 11. Copyright Third Nature, Inc. Stream If you want to do realtime and still manage your data effectively then you need to think about data architecture Collect Refine Manage Deliver Flowing Managed historyPersisted Microservices Metadata Metadata & reuse? Flow, persisted, managed define different access, processing, storage and retrieval requirements