Continuous Intelligence - Streaming Apps That Are Always In Sync | Simon Crosby, Swim
Download as PPTX, PDF0 likes337 views
Swim is an Apache 2.0 licensed platform designed for building applications that deliver continuous intelligence from streaming data at scale, automating the operation of distributed applications. It utilizes stateful, real-time stream processing through a graph of concurrent web agents, allowing real-time predictions and analytics on live data with reduced infrastructure needs. The platform supports applications that analyze complex relationships and continuously update insights, making it efficient for real-time traffic predictions and similar use cases.
Continuous Intelligence - Streaming Apps That Are Always In Sync | Simon Crosby, Swim
- 2. Swim is an Apache 2.0 licensed platform that makes it easy to build applications that deliver continuous intelligence from streaming data, at massive scale swimos.org
- 3. Swim is a Stateful, Real-time Stream Processor • Builds and scales apps from event data, creating a graph of stateful, concurrent actors that continuously compute – driven by data • Automates distributed application infrastructure operation • Load balances, secures, persists and auto-scales the application App: Distributed, stateful, graph of concurrent Web Agents (actors), streaming APIs & real-time UIs Infrastructure: Distributed, p2p mesh of instances on k8s using WebSockets Fabric
- 5. Predict real-time traffic flow traffic.swim.ai Deliver a continuous stream of predictions
- 7. Analyze, learn & predict, driven by streaming data Respond in real-time – in sync with the real-world Continuously evaluate parametric functions and causal relationships eg: “near to” or “predicted to be within” Build the app from streaming data by linking sources into a fluid graph of real-world relationships Continuous Intelligence
- 8. Continuously evaluate complex parametric relationships • Find trucks that are in or approaching a banned area • Find trucks with bad braking behavior and alert the nearest inspector
- 9. 9 Benefits Use 90% less infrastructure Apps are easy to develop and operate Do data science on live data Deliver answers a million times faster…
- 10. Client Client Client Client Streaming analytics (solved!) ☞ polling not real-time (a, b) < 290 Ω(b, t) > 188 Ψ(c,t-1) < ¾ Complements Kafka Streaming & kSQL Entity relationships are often parametric in time and space Evaluations are at huge scale, stateful and interdependent Application
- 11. Swim – Continuous Intelligence Developer builds an object- oriented Java application & deploys with standard tools Web Agents link to represent relationships and share state Swim uses streaming data to build a stateful graph of concurrent actors – Web Agents- one per object (source) .. to respond in sync with the real world, & deliver continuously updated insights to UIs / apps They continuously analyze, learn & predict from their state and the states of linked Agents … and stream their analysis and predictions in real-time over their links …
- 12. Swim is like for the real world • Creates a stateful, concurrent Web Agent for each data source, that continuously and statefully analyzes data from its real-world sibling (actors – like Akka without messaging) • Each Agent dynamically links to related Web Agents, builiding a fluid in-memory graph that tracks complex relationships eg: • Containment, proximity - “neighbor” or ”is approaching” • Computed - “correlated to” or ”predicted to be within” • Linked Agents use each others’ state to continuously analyze, learn and predict • And stream their insights to apps, real-time UIs, storage
- 13. Web Agents are stateful, concurrent, distributed (Java) actors • Lanes are object members – Properties and methods eg: “average” – Hold state, have logic eg “reduce” – Streamed as updates over links • Links are relationships between web agents • Schema derived: contains, average • Computed: maps, joins, correlations, membership, “predicted to be”, “near”, “correlated” – Build a graph – A link to a lane lets the linker continuously observe lane state Data Lanes Links (data, old_state) new_state Developer View: Web Agents
- 14. Streamed Inputs λ(i1, i2,.. in, ) → (o1,…oj, ) St-1 St Streamed Outputs Continuously Current Materialized Views λ(ỉ, ●) → (ỏ, ◙) • A Web Agent can analyze millions of (concurrent) updates to Agents it is linked to • “Debounced” to allow bursts of state changes • Changed input invalidates dependent outputs • Does not immediately trigger recomputation • Use timers to enforce latency bounds • Informed of partitions millions of links
- 15. 1 5 Agents Continuously Analyze, Learn & Predict MapReduce Graph Analysis Learn & Predict Spark / Flink / Jupyter etc Relational Queries
- 16. A Swim Application is an Active Graph
- 17. 1000m A Graph of Linked Web Agents
- 18. A Computational Graph That Computes as Data Flows
- 19. Edge Fog Cloud k8s P2p Mesh Where?
- 20. Edge Fog Cloud k8s P2p Mesh Web Agents Distribute Over Instances
- 21. React in real-time Easy to develop, deploy & run Enable data science on live data Continuous Intelligence for Apache Kafka Use 90% less infrastructure Always have the answer Analyze, learn & predict on-the-fly Deliver answers a million times faster…