Designing & Optimizing Micro Batching Systems Using 100+ Nodes (Ananth Ram, Rumeel Kazi, Accenture / Rich Rein, DataStax) | C* Summit 2016

Designing & Optimizing micro batching systems with
Cassandra, Spark and
Kafka
Accenture

Copyright © 2014 Accenture. All rights reserved.
Designing & Optimizing
micro batching systems
with Cassandra, Spark and
Kafka

Copyright © 2012 Accenture All rights reserved. 3
• Ananth Ram
– Big Data & Oracle Solution Architect , Accenture
– (Accenture Enkitec Group)
– Ananth.ram@Accenture.com
• Rumeel Kazi
– Big Data Solution Architect , Accenture
– (Accenture Federal)
– rumeel.k.kazi@accenturefederal.com
• Rich Rein
– Solution Architect , Datastax
– Rich.rein@datastax.com
Speaker Details and Contact

• Data Acceleration and Micro Batching
• Big data Architecture
– Technical Architecture
– Application Architecture
– Data Supply Chain Approach & Framework
• Application Design & Operations
– Design Considerations
– Data Flow
– Optimizations and Operations
• Application Access Patterns
– The Problems and Physics
– Idempotency
– Partition per Read
• Takeaways
Agenda

• Data as Value Chain
• Data Acceleration
– Movement
– Processing
– Insights
• High throughput with Micro Batch
Data Acceleration & Micro Batch !

Big Data Architecture

IV Hardware Architecture
Oracle 12c
Technical Architecture – Sample
Cassandra
Spark
Solr
Hadoop
Kafka
Spark
Big Data
Interfaces
NAS
Clustered
MQ
Files
External
Databases
Prod
(A)
Prod
(B)
Prod
(C)
Prod
(D)
Oracle 12c
12 Blades
288 Cores
6TBRAM
12 Blades
288 Cores
6TBRAM
12 Blades
288 Cores
6TB RAM
12 Blades
288 Cores
6TB RAM
4 x 10G– RAC
Interconnect

Data
Enrichment
44 nodes
RAC
4 nodes
Data Ingest
16 Nodes 23 Nodes
112 Nodes
Interfaces
12 Nodes
Technical Architecture – Additional Details
• Separate Datacenters for Cassandra and Solr.
• Spark is running in the same node as Cassandra for data locality.
• Kafka , java spring batch and spark streaming are used to enrich billions of
records a day
Java

Application Architecture
• Data enriched using java spring batch and spark streaming using kafka as temporary
staging area.
• Cassandra is used for faster lookups, summary views and persistence storage.
Data Ingestion
&
Business Rules
Application Cache
External
System
Interfaces
TXN DATA
(MQ, FILES, DB LINK)
OPERATIONAL
EVENTS
(MQ)
REFERENCE DATA
(MQ, FILES)
Java spring batch
Enriched Data
Aggregated Views
Reference Data
DataStore
Events
Data
IN-MEMORY TABLES
Reporting
WEB PORTAL
(CANNEDREPORTS)
&
PUSH ALERTS
AD-HOCQUERIES
Spark
Streaming
& Kafka
Enrichment
Process 2
Enrichment
Process 1
Enriched
Data
EVENTSDATA
Aggregated Views
Cassandra , Solr
HDFS, HIVE, Spark,
Spark R

• Cassandra
– Cassandra 400K/sec read/writes
– Cassandra - 1ms – 3ms Read Latency, 0.2 – 0.3 ms write.
Spark
– Spark Streaming processes 200K events/sec.
– Spark Streaming runs in the same host as Cassandra for data locality
• Kafka
– 800K/second total messages processed through 30 brokers.
– Kafka broker throughput is 30k/messages per broker.
– Snappy Compression gives up to 5X throughput in Benchmark. Yet to be tested in our apps.
• Java Apps
– Java spring batch processes 400K records/sec using 1000’s of threads in apps server.
– 32GB JVM with GC1 garbage collection with application cache gives this throughput.
Cassandra, Spark and Kafka Metrics

Big Data Architecture Approach
Accenture-Data-Acceleration-Architecture-Modern-Data-Supply-Chain.pdf
*Accenture Labs Paper – Carl Dukatz

Big Data Architecture Design Considerations - Criteria
Sample

Big Data Design Considerations - Approach

Design Considerations &
UseCases
Big Data Design Considerations

Application Design and
Operations

High Level Design Pattern

Pipeline Stage 0 (Partial Data Enrichment)
Kafka Cluster
Topic A
Partition 0
Topic A
Partition 1
Topic A
Partition N
DSE Cassandra / Spark Cluster
Executor 0 Cache
Executor 1 Cache
Executor N Cache
Kafka Cluster
Topic B
Partition 0
Topic B
Partition 1
Topic B
Partition N
Executor 0 Cache
Executor 1 Cache
Executor N Cache
Kafka Cluster
Topic C
Partition 0
Topic C
Partition 1
Topic C
Partition N
Executor 0 Cache
Executor 1 Cache
Executor N Cache
Data Processing Pipeline

 Application Metric Collection / Diagnostic Logging
 Include application level operational metrics as part of design. Collect Cassandra and Kafka
processing metrics including response times at object level.
 Executors report application functionality specific throughput and backlog metric to the driver
that then keeps aggregated count of point in time metrics for the process.
 Kafka / Cassandra data partitioning strategies
 Distribute partitioning keys evenly across the nodes on Cassandra and Kafka brokers. For
scenarios where this can’t be done easily when data is skewed to certain data entities that
need to be part of partitioning keys, add time windows as part of partitioning keys to avoid
data skewed to few nodes.
 Time-based partitioning to avoid data skewed to few nodes.
 Spark Executor Configurations
 Define Spark number of executors to match the number of partitions on the topic. Can have
more than one partition per executor depending on the throughput/latency need - keep it low
for reduced latency.
 Web / Solr Interface Consideration
 For consistency requirement, write at consistency-level of ALL on Solr data centers, if it fails
write local quorum. Additional sub-second overhead to be considered based on functional
needs.
Application Design Considerations

 Compaction Strategies
 Date Tiered v/s Size Tiered Compactions – High resource utilization on over 50 TB sized
tables running size tiered compactions on high velocity data and need to consider Date
Tiered for time series data.
 “Hot Spots” monitoring and actions
 Partition keys are chosen to ensure hot data is distributed evenly over the nodes
 Application logs with query, keys, and duration for exceeded SLAs can make problems with
specific keys known.
 Instrument application to rerun the query with CQL trace enabled to see where time was
spent.
 OpsCenter table metrics can show which nodes contain hotspots
 Nodetool toppartitions also shows the hot partition keys on a node
Performance Considerations

 Spark batch window optimization and max messages per partitions
 Optimize batch duration to not have wasted batch processing time.
 Define max messages per partition when executor spans multiple partitions. Prevents OOM
exceptions as well as keeps batch processing rate balanced.
 Dynamically change max rate based on wasted batch processing time.
 DataStax Driver Settings
 Separate Transaction data and Searched data into right sized data centers
 Search data to be read and written to the same DC.
 Use local data center aware strategy in conjunction with token aware.
 In-memory tables and Local caching
 Limit the number of in-memory tables to constantly changing but smaller tables that are
accessed very frequently.
 Consider local application caching for frequently accessed data.
Performance Considerations

 Latency & Throughput Monitoring
 Application should drive the data instead of technology stacks
 Use Splunk or ELK to aggregate, correlate data across nodes
 Co-relating Errors
 Use tools like Splunk or ELK
 Build Custom tools
 For Cassandra ( Nodetool, data from Opscenter)
 JMX from kafka
 Aggregated data in metrics table
 Use Java profiler like Yourkit
 For Cassandra latency Debugging
 Java memory , CPU and contentions
 Identify bottlenecks causing by specific methods/calls
Application Operations

Access Patterns

High Speed, Never Stop
1. The pipeline should never stop or wait
2. No stopping to upgrade software or hardware
3. No time for rollback. Roll forward.
4. No delays that will disrupt the write pipeline or read
throughput.
5. No time for locks, slow reads, large reads, joins, or
read-modify-write.
6. All frequent operations are short.

• Cost prohibits the frequent unnecessary
• No unnecessary frequently read data.
• No unnecessary frequently written data.
Affordable

No
• Long operations – Use the correct access patterns
• Client congestion
– Threads, sockets, heap, CPU, Memory, NUMA Cache
• Node congestion
– Threads, sockets, heap, CPU, Memory, NUMA Cache
– Storage channels
– Un-tuned or inconsistently tuned Cassandra nodes
• Network and NIC congestion
Pipeline Delays

If 2 ms is your target
• Think about how many requests can a node process
in that time window without congesting the client or
node.
• Web and IoT tend to be evenly distributed over time,
avoiding timeslot contention.
• Batch size that can be processed in the time slot?
• Careful parallelization may be needed.
Physics of the SLA Time Slot

Hot Partitions
Physics of Partitions
Hot Batch or Traffic

• Correct table partition keys and access patterns
– Scale from 6 nodes to 1000’s
• Incorrect
– Does not scale by adding nodes
– Will not handle more load
Get the Partition Access Patterns Right

Physics of a single Partition
Microseconds Operation
0.1 Read and Write RAM
100 Write Partition
100 Read Partition from memory
2,000 Read Hash access to partition in memory and read SSD
20,000 Read Hash access to partition in memory and read Spindle

• Avoid
– Lists (collection)
– Read-modify-write Updates
– Counters
– GUIDs only identification of real world objects or actions
• Allows client retry (roll-forward)
• Allows pipelining of updates without waits
Idempotency

• Replace read-modify-write operations
– Counters
– Updated aggregates
– Lists (collection)
With
– data increment values which get aggregated in
microbatches
– Cassandra 3.0 Aggregates
– Sets (collection)
Replace Read-Modify-Write

• Reads must wait
• API Reads are 25-50x slower than writes
• Reads consume 5x the resource bandwidth of a write
• Disk is far cheaper than RAM, CPU, and Rack
• So
– Design writes for reads
– De-normalization the same as for relational
• Multiple materialized views and temp tables
• Summary tables
Denormalize

Nesting Rows in the Partitions – 1 of 3

Write nested data to further reduce the read to 1 partition

Cassandra allows 3 levels to be nested in a single partition

Summary / Takeaways

• Treat data pipeline as value chain and accelerate movement
using fit-for-purpose Bigdata stack.
• Design your apps to drive latency/throughput visibility
• Micro batch in every layer possible to get high throughput
• Enrich data in Kafka using spark/spark streaming as process
engine.
• Cache frequently accessed data closer to code to get best
throughput.
• Focus on datamodel and Access patterns
• Review distinct features of Bigdata technology platform for
data acceleration (Accenture Approach white paper).
Summary / Take Away

Designing & Optimizing Micro Batching Systems Using 100+ Nodes (Ananth Ram, Rumeel Kazi, Accenture / Rich Rein, DataStax) | C* Summit 2016

More Related Content

What's hot (20)

Viewers also liked (9)

Similar to Designing & Optimizing Micro Batching Systems Using 100+ Nodes (Ananth Ram, Rumeel Kazi, Accenture / Rich Rein, DataStax) | C* Summit 2016 (20)

More from DataStax (20)

Recently uploaded (20)

Designing & Optimizing Micro Batching Systems Using 100+ Nodes (Ananth Ram, Rumeel Kazi, Accenture / Rich Rein, DataStax) | C* Summit 2016

Editor's Notes