Real Time UI with Apache Kafka Streaming Analytics of Fast Data and Server Push
Download as PPTX, PDF5 likes3,488 views
The document discusses the use of Apache Kafka for real-time UI updates and streaming analytics, focusing on handling live data from various sources, such as social media and IoT devices. It details the integration of Kafka within a system architecture that allows for instant analysis and user interfaces that reflect real-time data without needing page refreshes. The presentation includes practical examples of streaming analyses, such as live tweet feeds and user interactions, along with Kafka's functionalities like ksql and Kafka Streams for effective data processing.
![FAST DATA AND ACTIVE UI
• Handle [data | event] influx
• Analyze in real time
• Publish findings instantaneously
• Update UI & notify end user immediately
• Convince end user that the UI is (still) active (and no F5 is required)
• Decoupled components
• No data loss when a component is temporarily down
• Scalable with volume of events and of number of clients](https://image.slidesharecdn.com/realtimeuiwithapachekafkastreams-devoxxuk-may2018-180511140530/85/Real-Time-UI-with-Apache-Kafka-Streaming-Analytics-of-Fast-Data-and-Server-Push-8-320.jpg)
![INTRODUCING APACHE KAFKA
• ..- 2010 – creation at Linkedin
• Message Bus | Event Broker | Streaming Data Platform
• High volume, low latency, highly reliable, cross technology Commit Log (ledger)
• Scalable (#messages & #consumers), distributed, durable, strict message ordering, ….
• 2011/2012 – open source under the Apache Incubator/ Top Project
• Backed by Confluent – Confluent Open Source & Confluent Enterprise
• Kafka is used by most many [large] corporations:
• And embraced by [almost] all many software vendors & cloud providers
• Client libraries available for NodeJS, Java, C/C++, Python, Ruby, PHP, Scala,
Clojure, Rust, .NET, go (aka golang) and many more
• Apache Kafka includes Connect, Mirror Maker, Streams
• KSQL is Open Source, part of the Confluent Platform](https://image.slidesharecdn.com/realtimeuiwithapachekafkastreams-devoxxuk-may2018-180511140530/85/Real-Time-UI-with-Apache-Kafka-Streaming-Analytics-of-Fast-Data-and-Server-Push-14-320.jpg)
![KAFKA STREAMS
• Real Time Event [Stream] Processing integrated into Kafka
• Aggregations & Top-N
• Time Windows
• Continuous Queries
• Latest State (event sourcing)
• Turn Stream (of changes) into Table
(of most recent or current state)
• Part of the state can be quite old
• Exactly-once processing in Kafka Streams (as of 0.11.0)
• Note: Kafka Streams is relatively new
• Only support for Java clients; Scala client is planned too](https://image.slidesharecdn.com/realtimeuiwithapachekafkastreams-devoxxuk-may2018-180511140530/85/Real-Time-UI-with-Apache-Kafka-Streaming-Analytics-of-Fast-Data-and-Server-Push-38-320.jpg)
![THE THREE FLAVORS OF
STREAM[ING] ANALYTICS WITH KAFKA
Kafka Cluster
KTable
KStream
Topic
Topic
Topic
KTable
Topic
Topic
KStream](https://image.slidesharecdn.com/realtimeuiwithapachekafkastreams-devoxxuk-may2018-180511140530/85/Real-Time-UI-with-Apache-Kafka-Streaming-Analytics-of-Fast-Data-and-Server-Push-51-320.jpg)
![CONCLUSION
• Fast data – Active UI & Active API leveraging live data streams
• Proactively informing consumers with [results from streaming analysis of] events
• Decoupled processing
• Unentangled, separated in space and time
• Distributed across clouds and on premises
• Kafka
• Scalable, reliable, historic Events & Data Store
• Streaming Analysis –Kafka Streams and KSQL
• Modern browser – push capability & bi-directional communication
• SSE, WebSocket, HTTP/2, WebWorker Notifications
• Active UI with happy users, fresh data without burden on back end systems
• “Step away from that F5 key”](https://image.slidesharecdn.com/realtimeuiwithapachekafkastreams-devoxxuk-may2018-180511140530/85/Real-Time-UI-with-Apache-Kafka-Streaming-Analytics-of-Fast-Data-and-Server-Push-59-320.jpg)
Real Time UI with Apache Kafka Streaming Analytics of Fast Data and Server Push
- 1. #ActiveUIDevoxxUK @lucasjellema Real Time UI with Apache Kafka Streaming Analytics of Fast Data and Server Push Lucas Jellema (CTO AMIS) @lucasjellema http://technology.amis.nl www.amis.nl
- 8. FAST DATA AND ACTIVE UI • Handle [data | event] influx • Analyze in real time • Publish findings instantaneously • Update UI & notify end user immediately • Convince end user that the UI is (still) active (and no F5 is required) • Decoupled components • No data loss when a component is temporarily down • Scalable with volume of events and of number of clients
- 9. API µ µ µ µ
- 10. THE CASE AT HAND Client Client Client Client Tweets on #devoxxUK #java #oraclecode Show live tweet feed for tags Show live tweet aggregates per tag Allow users to like tweets – and show live list of liked tweets Show a live list of top 3 liked tweets per tag Standin for all Hot Sources of Live Data: IoT Physical World Reports, IT Ops, Web Click Statistics, Business Process execution, open data feeds (traffic, weather, stocks, …), Points of Sales, Social Media, microservices chatter
- 12. FOLLOWING ALONG (AND HELP WITH) THE DEMO http://bit.ly/ActiveUIDevoxxUK
- 14. INTRODUCING APACHE KAFKA • ..- 2010 – creation at Linkedin • Message Bus | Event Broker | Streaming Data Platform • High volume, low latency, highly reliable, cross technology Commit Log (ledger) • Scalable (#messages & #consumers), distributed, durable, strict message ordering, …. • 2011/2012 – open source under the Apache Incubator/ Top Project • Backed by Confluent – Confluent Open Source & Confluent Enterprise • Kafka is used by most many [large] corporations: • And embraced by [almost] all many software vendors & cloud providers • Client libraries available for NodeJS, Java, C/C++, Python, Ruby, PHP, Scala, Clojure, Rust, .NET, go (aka golang) and many more • Apache Kafka includes Connect, Mirror Maker, Streams • KSQL is Open Source, part of the Confluent Platform
- 16. KAFKA TERMINOLOGY • Topic • partition • Message • == ByteArray • Broker • replicated • Producer • Consumer • Working together in Consumer Groups Producer Consumer Topic Broker Key Value Time Message
- 18. THE CASE AT HAND – STEP ONE Client Client Client Client Tweets on #devoxxUK #java #oraclecode Show live tweet feed for each tag in each client Tweets Topic
- 19. THE CASE AT HAND – STEP ONE AND A HALF Client Client Client Client Tweets on #devoxxUK #java #oraclecode Show live tweet feed for each tag in each client Tweets Topic
- 20. THE CASE AT HAND – STEP ONE AND TWO Client Client Client Client Tweets on #devoxxUK #java #oraclecode Show live tweet feed for each tag Tweets Topic Oracle Cloud Event HubApplication Container
- 24. KAFKA CONSUMER IN NODE GETS EVENTS PUSHED INTO APPLICATION
- 25. WEB APP REGISTERS LISTENER WITH KAFKA EVENT CONSUMER
- 26. THE CASE AT HAND SERVER SENT EVENTS FOR PUSH BACK Client Client Client Client Tweets on #devoxxUK #java #oraclecode Show live tweet feed for each tag Tweets Topic Server Sent Event
- 27. SERVER SENT EVENT – SERVER SIDE
- 28. SERVER SENT EVENT – CLIENT SIDE
- 29. LIVE TWEET STREAM Server Sent Event
- 30. THE CASE AT HAND TWEET LIKES – CLIENT TO SERVER TO ALL CLIENTS Client Client Client Client Tweets on #devoxxUK #java #oraclecode Show live tweet feed for each tag Tweets Topic SSE Allow users to like tweets – and show live list of liked tweets
- 31. THE CASE AT HAND WEB SOCKETS – FOR BI DIRECTIONAL PUSH Client Client Client Client Tweets on #devoxxUK #java #oraclecode Show live tweet feed for each tag Tweets Topic SSE WebSockets Allow users to like tweets – and show live list of liked tweets
- 32. WEBSOCKETS – SERVER SIDE
- 33. WEBSOCKETS – CLIENT SIDE - PUBLISHING Client WebSockets
- 34. WEBSOCKETS – CLIENT SIDE - CONSUMING Client WebSockets
- 36. THE CASE AT HAND: RUNNING COUNT STREAMING ANALYSIS OF TWEET EVENTS Client Client Client Client Tweets on #devoxxUK #java #oraclecode Show live tweet feed for each tag Tweets Topic SSE WebSockets Allow users to like tweets – and show live list of liked tweets Show live tweet aggregates per tag
- 37. THE CASE AT HAND - STREAMING ANALYSIS OF TWEETS Client Client Client Client Tweets on #devoxxUK #java #oraclecode Show live tweet feed for each tag Tweets Topic WebSockets Allow users to like tweets – and show live list of liked tweets Show live tweet aggregates per tag tweetAnalytics Topic Streaming Tweets Aggregation µ SSE
- 38. KAFKA STREAMS • Real Time Event [Stream] Processing integrated into Kafka • Aggregations & Top-N • Time Windows • Continuous Queries • Latest State (event sourcing) • Turn Stream (of changes) into Table (of most recent or current state) • Part of the state can be quite old • Exactly-once processing in Kafka Streams (as of 0.11.0) • Note: Kafka Streams is relatively new • Only support for Java clients; Scala client is planned too
- 40. EXAMPLE OF KAFKA STREAMS Topic groupBy Aggregate Join Topic Map (Xform) Publish TweetMessage Tag Text Author Set Conference as key Sum/Avg/Top3 by key (==tagFilter) As JSON Round aggregate to nearest 100 e.g. Author Details Topic: CountTweetsPerTag and possibly per time window
- 41. ADD KAFKA STREAMS CAPABILITIES TO ANY JAVA APPLICATION • Add Maven Dependency • Connect to Kafka Cluster • Compose & Execute Kafka Streams logic: • Write Java code
- 42. KAFKA STREAMS – RUNNING COUNT TWEETS PER CONFERENCE
- 43. WOULDN’T IT BE NICE IF YOU COULD JUST DO select tagfilter , count(*) as tag_cnt from tweets window hopping ( size 5 minutes , advance by 30 seconds) group by tagfilter
- 44. KSQL – CONTINUOUS QUERIES – PROCESSING EVENT STREAMS LIKE TABLES • Transform, Filter, Join, Aggregate, (time) Window on Event Streams • Results are produced as regular Kafka Events
- 45. RUNNING TWEET COUNT THROUGH KSQL
- 46. RUNNING TWEET COUNT THROUGH KSQL • KSQL CLI • List topics • Print topic • Create stream • Create table • List Topics • Query from table
- 47. RUNNING TWEET COUNT THROUGH KSQL
- 48. KSQL RUNNING COUNT OF TWEETS PER TAG RESULTS PUBLISHED ON KAFKA TOPIC Client Client Client Client Tweets on #devoxxUK #java #oraclecode Show live tweet feed for each tag Tweets Topic Allow users to like tweets – and show live list of liked tweets Show live tweet aggregates per conference TWEET_COUNT Topic Running Tweets Aggregation SSE Show a live list of top 3 liked tweets per tag
- 49. STREAMING TWEET ANALYTICS - PUSHED TO CLIENT Server Sent Event
- 50. KSQL INTERFACES Kafka Cluster Kafka Streams API KSQL Server HTTP REST API CLI JDBC Driver KTable KStream Topic Topic Topic KTable Topic Topic Java Application KStream
- 51. THE THREE FLAVORS OF STREAM[ING] ANALYTICS WITH KAFKA Kafka Cluster KTable KStream Topic Topic Topic KTable Topic Topic KStream
- 52. THE CASE AT HAND – STREAMING ANALYSIS OF TWEET LIKES Client Client Client Client Tweets on #devoxxUK #java #oraclecode Show live tweet feed for each tag Tweets Topic WebSockets Allow users to like tweets – and show live list of liked tweets Show live tweet aggregates per conference TWEET_COUNT Topic Running Tweets Aggregation SSE Show a live list of top 3 liked tweets per tag
- 53. THE CASE AT HAND - STREAMING ANALYSIS OF TWEET LIKES Client Client Client Client Tweets on #devoxxUK #java #oraclecode Tweets Topic WebSockets TWEET_COUNT Topic Running Tweets Aggregation SSE Show a live list of top 3 liked tweets per tag Tweet-Like Topic Running Top3 Like Counts Aggregation Tweet_Like_ Counts Topic
- 54. WEBSOCKETS – SERVER SIDE PRODUCE TWEET LIKE EVENTS
- 55. CREATE STREAM FROM TOPIC & TABLE FROM STREAM create stream tweet_likes ( tweetid varchar) with (kafka_topic='tweet-like_topic' , value_format='json', key='tweetid'); create table like_counts as select count(*) likecount , tweetid , tagfilter from tweet_likes window hopping ( size 60 seconds , advance 10 seconds) group by tweetid, tagfilter;
- 56. CREATE STREAM FROM TABLE & ENRICHED STREAM FROM STREAM create stream top3_likes as select tweetid, tagfilter, topk( likecount, 3) from like_counts group by tagfilter; create stream enriched_top3_likes as select tl.tagfilter, tl.tweetid, t.text, t.rowtime from top3_likes tl left join tweets t on tl.tweetid = t.tweetid;
- 57. RUNNING TOP 3 OF BEST LIKED TWEETS PER CONFERENCE Server Sent Event
- 58. Tweets on #devoxxUK #java #oraclecode Tweets Topic Oracle Cloud Event HubApplication Container Tweets Topic TWEET_COUNT Topic Running Tweets Aggregation Tweet-Like Topic Running Top3 Like Counts Aggregation Tweet_Like_ Counts Topic Client Client Client Client IoT metrics from hundreds of devices User actions & click events from webshop Live Traffic EventsMicroservices chatter Social Media events (Facebook, Whatsapp, …) IT Operations – monitoring metrics µ µ µ µ
- 59. CONCLUSION • Fast data – Active UI & Active API leveraging live data streams • Proactively informing consumers with [results from streaming analysis of] events • Decoupled processing • Unentangled, separated in space and time • Distributed across clouds and on premises • Kafka • Scalable, reliable, historic Events & Data Store • Streaming Analysis –Kafka Streams and KSQL • Modern browser – push capability & bi-directional communication • SSE, WebSocket, HTTP/2, WebWorker Notifications • Active UI with happy users, fresh data without burden on back end systems • “Step away from that F5 key”
- 60. • Blog: technology.amis.nl • Email: lucas.jellema@amis.nl • : lucasjellema • : lucas-jellema • : www.amis.nl, info@amis.nl +31 306016000 Edisonbaan 15, Nieuwegein https://github.com/lucasjellema/devoxxUK18-active-ui-streaming-analysis
Editor's Notes
- #16: https://cwiki.apache.org/confluence/display/KAFKA/Clients