Real Time Analytics with Apache Cassandra - Cassandra Day Berlin
4 likes1,689 views
This document provides an overview of a presentation on real-time analytics using Apache Cassandra, delivered by Guido Schmutz from Trivadis. It discusses various aspects such as Cassandra data modeling for time series data, social media intelligence systems, and the integration of graph databases like Titan with Cassandra. The agenda includes customer use cases, architecture, and insights on big data technologies, emphasizing the importance of denormalization and understanding application-specific requirements.
Real Time Analytics with Apache Cassandra - Cassandra Day Berlin
- 1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH Real-Time Analytics with Apache Cassandra Cassandra Day Berlin, 11.2.2016 Guido Schmutz
- 2. Guido Schmutz Working for Trivadis for more than 19 years Oracle ACE Director for Fusion Middleware and SOA Co-Author of different books Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data Member of Trivadis Architecture Board Technology Manager @ Trivadis More than 25 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Slideshare: http://de.slideshare.net/gschmutz Twitter: gschmutz 2
- 3. Our company. © Trivadis – The Company3 2/11/16 Trivadis is a market leader in IT consulting, system integration, solution engineering and the provision of IT services focusing on and and Open Source technologies in Switzerland, Germany, Austria and Denmark. We offer our services in the following strategic business fields: Trivadis Services takes over the interacting operation of your IT systems. O P E R A T I O N
- 4. COPENHAGEN MUNICH LAUSANNE BERN ZURICH BRUGG GENEVA HAMBURG DÜSSELDORF FRANKFURT STUTTGART FREIBURG BASEL VIENNA With over 600 specialists and IT experts in your region. © Trivadis – The Company4 2/11/16 14 Trivadis branches and more than 600 employees 200 Service Level Agreements Over 4,000 training participants Research and development budget: CHF 5.0 million Financially self-supporting and sustainably profitable Experience from more than 1,900 projects per year at over 800 customers
- 5. Agenda 1. Customer Use Case and Architecture 2. Cassandra Data Modeling 3. Cassandra for Timeseries Data 4. Titan:db for Graph Data 5
- 6. Customer Use Case and Architecture 6
- 7. Data Science Lab @ Armasuisse W&T W+T flagship project, standing for innovation & tech transfer Building capabilities in the areas of: • Social Media Intelligence (SOCMINT) • Big Data Technologies & Architectures Invest into new, innovative and not widely-proven technology • Batch / Real-time analysis • NoSQL databases • Text analysis (NLP) • Graph Data • … 3 Phases: June 2013 – June 2015 7
- 8. SOCMINT System – Time Dimension Major data model: Time series (TS) TS reflect user behaviors over time Activities correlate with events Anomaly detection Event detection & prediction 8
- 9. SOCMINT System – Social Dimension User-user networks (social graphs); Twitter: follower, retweet and mention graphs Who is central in a social network? Who has retweeted a given tweet to whom? 9
- 10. SOCMINT System - “Lambda Architecture” for Big Data Data Collection (Analytical) Batch Data Processing Batch compute Batch Result StoreData Sources Channel Data Access Reports Service Analytic Tools Alerting Tools Social RDBMS Sensor ERP Logfiles Mobile Machine (Analytical) Real-Time Data Processing Stream/Event Processing Batch compute Real-Time Result Store Messaging Result Store Query Engine Result Store Computed Information Raw Data (Reservoir) = Data in Motion = Data at Rest 10
- 11. SOCMINT System – Frameworks & Components in Use Data Collection (Analytical) Batch Data Processing Batch compute Batch Result StoreData Sources Channel Data Access Reports Service Analytic Tools Alerting Tools Social (Analytical) Real-Time Data Processing Stream/Event Processing Batch compute Real-Time Result Store Messaging Result Store Query Engine Result Store Computed Information Raw Data (Reservoir) = Data in Motion = Data at Rest 11
- 12. Streaming Analytics Processing Pipeline Kafka provides reliable and efficient queuing Storm processes (rollups, counts) Cassandrastores results at same speed StoringProcessingQueuing 12 Twitter Sensor 1 Twitter Sensor 2 Twitter Sensor 3 Visualization Application Visualization Application
- 14. Cassandra Data Modelling 14 • Don’t think relational ! • Denormalize, Denormalize, Denormalize …. • Rows are gigantic and sorted = one row is stored on one node • Know your application/use cases => from query to model • Index is not an afterthought, anymore => “index” upfront • Control physical storage structure
- 15. “Static” Tables – “Skinny Row” 15 rowkey CREATE TABLE skinny (rowkey text, c1 text PRIMARY KEY, c2 text, c3 text, PRIMARY KEY (rowkey)); Grows up to Billion of Rows rowkey-1 c1 c2 c3 value-c1 value-c2 value-c3 rowkey-2 c1 c3 value-c1 value-c3 rowkey-3 c1 c2 c3 value-c1 value-c2 value-c3 c1 c2 c3 Partition Key
- 16. “Dynamic” Tables – “Wide Row” 16 rowkey Billion of Rows rowkey-1 ckey-1:c1 ckey-1:c2 value-c1 value-c2 rowkey-2 rowkey-3 CREATE TABLE wide (rowkey text, ckey text, c1 text, c2 text, PRIMARY KEY (rowkey, ckey) WITH CLUSTERING ORDER BY (ckey ASC); ckey-2:c1 ckey-2:c2 value-c1 value-c2 ckey-3:c1 ckey-3:c2 value-c1 value-c2 ckey-1:c1 ckey-1:c2 value-c1 value-c2 ckey-2:c1 ckey-2:c2 value-c1 value-c2 ckey-1:c1 ckey-1:c2 value-c1 value-c2 ckey-2:c1 ckey-2:c2 value-c1 value-c2 ckey-3:c1 ckey-3:c2 value-c1 value-c2 1 2 Billion Partition Key Clustering Key
- 17. Cassandra for Timeseries Data 17
- 18. Show Timeseries: Provide list of metrics 18 CREATE TABLE tweet_count ( sensor_id text, bucket_id text, key text, time_id timestamp, count counter, PRIMARY KEY((sensor_id, bucket_id), key, time_id)) WITH CLUSTERING ORDER BY (key ASC, time_id DESC); Use of “Static” Table bucket-id defines buckets of values • HOUR-2015-10 = values collected hourly in one partition for one month ABC-001:HOUR-2015-10 dse:10:00:count 1’550 ABC-001:DAY-2015-10 dse:14-OCT:count 105’999 dse:13-OCT:count 120’344 nosql:14-OCT:count 2’532 dse:09:00:count 2’299 nosql:10:00:count 25 30d * 24h * n keys = n * 720 cols Open Source TimeSeries DBs over Cassandra: Kairos DB: https://kairosdb.github.io/ Heroic: http://spotify.github.io/heroicPartition Key Clustering Key
- 19. Show Timeseries: Provide list of metrics 19 UPDATE tweet_count SET count = count + 1 WHERE sensor_id = 'ABC-001’ AND bucket_id = 'HOUR-2015-10' AND key = 'ALL’ AND time_id = '2015-10-14 10:00:00'; SELECT * from tweet_count WHERE sensor_id = 'ABC-001' AND bucket_id = 'HOUR-2015-10' AND key = 'ALL' AND time_id >= '2015-10-14 08:00:00’; sensor_id | bucket_id | key | time_id | count ----------+--------------+-----+--------------------------+------- ABC-001 | HOUR-2015-10 | ALL | 2015-10-14 12:00:00+0000 | 100230 ABC-001 | HOUR-2015-10 | ALL | 2015-10-14 11:00:00+0000 | 102230 ABC-001 | HOUR-2015-10 | ALL | 2015-10-14 10:00:00+0000 | 105430 ABC-001 | HOUR-2015-10 | ALL | 2015-10-14 09:00:00+0000 | 203240 ABC-001 | HOUR-2015-10 | ALL | 2015-10-14 08:00:00+0000 | 132230 Partition Key Clustering Key
- 20. Titan:db & Cassandra for Graph Data 20
- 21. Supporting Graph Data with Titan:db and Cassandra 21 http://thinkaurelius.github.io/titan/
- 22. Gremlin in Action – Creating the Graph 22
- 23. Gremlin in Action – Graph Traversal 23
- 24. Gremlin in Action – Graph Traversal (II) 24
- 25. Summary - Know your domain Connectedness of Datalow high Document Data Store Key-Value Stores Wide- Column Store Graph Databases Relational Databases
- 26. Guido Schmutz Email: guido.schmutz@trivadis.com +41 79 412 05 39 26