Briefing room: An alternative for streaming data collection
1 like841 views
The document discusses the complexities of streaming analytics and data collection systems within organizations, highlighting the reliance on both live and persisted data. It emphasizes the need for streaming infrastructure that enables the reuse of data across various applications and demonstrates the challenges posed by existing information systems. Third Nature, the firm behind the commentary, aims to assist companies in leveraging analytics and information strategies to improve their operational effectiveness.
Briefing room: An alternative for streaming data collection
- 1. 1 Live Wire: The Next Dimension for Streaming Analytics Alternatives for collecting and using streaming data Analyst commentary December, 2015 Mark Madsen Third Nature @markmadsen
- 2. Copyright Third Nature, Inc. Almost all data today flows on networks, mostly used after collection, much of it never stored 2 Source: Noumenal Disconnected Milliseconds Minutes Hours+
- 3. Copyright Third Nature, Inc. Source: Noumenal Milliseconds Minutes Hours+ Events and monitoring occur in different time frames, follow different cycles of use 3 Disconnected Data lives in multiple places, at multiple levels of detail, for differing durations. Unlikely to all be in one place. Nor should it be.
- 4. Copyright Third Nature, Inc. Streaming infrastructure, collection systems, big data, tend to be viewed as standalone systems Most talk about this topic focuses on streams and programs living on the network. Usually viewed as independent from the rest of data use in the organization. The problem is that the live and persisted data are used together, or have dependencies of use.
- 5. Copyright Third Nature, Inc. Monthly Production plans Weekly pre- orders for bulk cheese Availability confirmation and location In store system Store Stock Management Store EPOS data Category Supervis or Stock adjustments/ order interventions Order adjustment Stock/order interventions * * Orders (based on 6 day forecast) Dallas Distrib Centre WMS Picking/load teams Pos/Pick lists/Load sheets Confirmed Deliveries/ Confirmed picks + loads Farmers Milk intake/ silos Cheese plant Plant Processor In-house Cheese store Contract Cheese store Processor Packing plant Processor National Distribution Centre Retailer RDC Retailer Stores (550) Retailer HQ Consolidated Demand Ordering Processor NDC Customer Services Daily order - SKU/Depot/ Vol Sent @ 12.30-13:00 Delivery orders Processor HQ Sales Team/ Account Manager Processor HQ Forecasting Team Processor HQ Bulk Planning Team Cheese plant Planner/Stock office Processor HQ Milk Purchasing Team Cheese plant Transport Manager Actual daily delivery figures Daily collection planning Weekly order for delivery to Packing plant Daily & weekly Call- off Daily Call-off 15/day 22 pallet loads 15/day A80 Shortages/ Allocation instructions Annual Buying plan Milk Availability Forecast Annual prediction of milk production Shortages/ Allocation instructions Daily milk intake Weekly milk shortages shortages Spot mkt or Processor ingredients Packing plant Planning Team Processor HQ JBA Invoicing and Sales Monitor FGI and Last 5 weeks sales Expedite Changes to existing forecast - exceptions Retailer HQ Retailer Buyer Meeting every 6 weeks Packing plant Cheese ordering 10 day stock plan On line stock info 7 day order plan for bulk cheese Arrange daily delivery schedule Emergency call-off Daily optimisation of loads Service Monitor Despatch and delivery confirmations Processor NDC Transport Planning Transport Plan Processor NDC Inventory MonitoringStock and delivery monitoring Processor NDC Warehouse management syatem Operation Instructions Key Shaded Boxes = Product flow system Un-shaded boxes = Information flow system RetailerCheese ProcessorFarms Schedule weekly & Daily 10 Day plan(wed) and daily plan 15/day Changes to existing forecast - exceptions Stock availability Monthly review Annual f/cast Source: IGD Food Chain Centre, February 2008 What about all the systems that already exist, built up over years? A typical value chain diagram, showing the supply chain for cheese. Notice the extreme complexity of information exchanges (in red)
- 6. Copyright Third Nature, Inc. Even within one company it’s complicated Off-the-shelf software, custom applications, SaaS applications, logs, event streams, 3rd party datasets…
- 7. Copyright Third Nature, Inc. Order Entry Order Database Customer Service Integration Program Inventory Database Distribution Integration Program Receivables Database Accounts Receivable Data Warehouse Analysts & users Data is already moving between many applications but it isn’t streaming
- 8. Copyright Third Nature, Inc. The shortcomings of the log-centered approach
- 9. Copyright Third Nature, Inc. Supplement log-based models with non-intrusive collection
- 10. Copyright Third Nature, Inc. Streaming infrastructure needs to enable reuse
- 11. Copyright Third Nature, Inc. Image Attributions Thanks to the people who supplied the images used in this presentation: ignore_no_parking.jpg - http://www.flickr.com/photos/vermininc/2352068440 Streaming environment - http://www.noumenal.com/ phone booth red snow - http://www.flickr.com/photos/wwarby/5907617001/ lots of screens to watch – Cprbis stock photo Slide 11
- 12. Copyright Third Nature, Inc. About Third Nature Third Nature is a research and consulting firm focused on new and emerging technology and practices in analytics, business intelligence, and performance management. If your question is related to data, analytics, information strategy and technology infrastructure then you‘re at the right place. Our goal is to help companies take advantage of information-driven management practices and applications. We offer education, consulting and research services to support business and IT organizations as well as technology vendors. We fill the gap between what the industry analyst firms cover and what IT needs. We specialize in product and technology analysis, so we look at emerging technologies and markets, evaluating technology and hw it is applied rather than vendor market positions.