The Apache Cassandra ecosystem

Cassandra’s Ecosystem
Drivers, Hardware, Data Centres, Provisioning,
Configuration - how it all fits together.
Alex Thompson @ Datastax Sydney - Cassandra Meetup September 2015 @ Rozetta

Drivers (modern CQL, data-aware drivers)
● Java
● C#
● Node.js
● Python
● Ruby
● C/C++
● PHP (rc1)
● Spark Connector
● ODBC (drivers with CQL connectors, for Spark and third party apps)

How drivers probe a C* cluster
Cassandra has two levels of internal communication with other nodes:
1. The query messaging layer which moves INSERTS, UPDATES, DELETES and DATA around the cluster.
messaging
protocol

1. The query messaging layer which moves INSERTS, UPDATES, DELETES and DATA around the cluster
2. The Gossip protocol for discovering other nodes, learning about their location, token range and state.
(Think of the messaging protocol doing the heavy lifting and gossip doing only lightweight discovery)
messaging
protocol
gossip protocol

1. The query messaging layer which moves INSERTS, UPDATES, DELETES and DATA around the cluster
2. The Gossip protocol for discovering other nodes, learning about their location, token range and state.
(Think of the messaging protocol doing the heavy lifting and gossip doing only lightweight discovery)
messaging
protocol
gossip protocol
driver
The driver sends a probe into the cluster to:
1. discover nodes and their location
2. their token range
3. state
4. latency

Drivers are data aware
1
32
driver
ID IP Address Token range State Latency
1 10.0.0.1 1-3 UP 6ms
2 10.0.0.2 4-6 UP 5ms
3 10.0.0.3 7-9 DOWN -
When a query comes into the driver, the driver can use its
gossip table to lookup the token and identify the node
responsible for that partition_key.

Drivers maintain a connection pool to C* nodes
driverconnection
pool
A pool of reusable persistent connections that are taken from and put back
into the pool as required by the driver and application.

Drivers can fail over to other replicas*
1
32
driver
1 10.0.0.1 1-3 UP 6ms
2 10.0.0.2 4-6 UP 5ms
3 10.0.0.3 7-9 DOWN -
With node 3 down, the driver knows that the token range for a
particular keyspace is replicated to node 1 and will fail over to
that node to satisfy the query.
*configurable

Drivers can fail to lower consistency levels*
1
32
driver
1 10.0.0.1 1-3 UP 6ms
2 10.0.0.2 4-6 UP 5ms
3 10.0.0.3 7-9 UP 6ms
With a replication factor of 2 for a keyspace and querying at ConsistencyLevel=2 a query will fail if both replicas are
not available. The driver can be configured to try at CL=2 and if it fails (due to a node being down) accept CL=1
*configurable

Driver logging can be turned on in log4j
Here's a simple configuration that outputs to the console and to a file:
log4j.rootLogger=ERROR, console, file
log4j.logger.com.datastax.driver.core=DEBUG
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern= %-6r [%t] %-5p %c %x - %m%n
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=/tmp/driver.log
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern= %-6r [%t] %-5p %c %x - %m%n
Save the above section into a file called log4j.properties.
The log4j.properties file needs to be at the root of the Java classpath, so at the root of a directory or a JAR that is in the classpath.
I've used an absolute path for the target file, if you put a relative path it will be relative to the directory you run the program from.

What are Cassandra Data Centers?
1
32
DC1
1
32
DC2cluster
A cluster can have many Logical Cassandra Data Centres, data can be
replicated between DCs, DCs can have different workloads and exist in
different physical locations.

What are uses for DCs?
● Split workloads
● Split analytics from OLTP
● Geographic redundancy
● Availability zone redundancy (AMZN)
● Geographic Proximity serving
● DR
1
32
DC1
1
32
DC2cluster

DC2DC1 cluster
Each DC is its own complete token range
1-3
7-94-6
1-3
7-94-6
Because a DC is a complete token range, a DC can fail and the other one can stay up and operational.

DC2DC1
Replication is efficient across DCs:
1
32
1
32
multi node replication within a DC : 2x writes (RF=2)
single node replication between DCs 1x write
multi node replication within a DC : 3x writes (RF=3)
coordinator election
coordinator election

Example multi DC setup - OLTP and Analytics
1
32
DC1
1
32
DC2cluster
web facing OLTP DC Hadoop/Spark analytics DC
replication

Example multi DC setup - Geo redundancy
Europe DC US DC
replication
1
32
DC1
1
32
DC2cluster
Proximity serving and geographic failover

Configuring Cassandra - Memory
Memory is configured in the cassandra-env.sh file.
For local development on a laptop or workstation let the automatic script in the cassandra-env.sh file allocate memory to
the C* process, the script will allocate ¼ of the RAM on the machine to MAX_HEAP_SIZE up to a max of 8gb.
For all other environments you will want to start off on 8gb and work your way up to 12gb for the MAX_HEAP_SIZE, you
will need to explcity set values above 8gb in the cassandra-env.sh MAX_HEAP_SIZE setting.
If you are using an ‘older’ Cassandra you will probably be using the CMS/ParNew garbage collector, in this case you will
additionally need to set the HEAP_NEWSIZE setting, you probably want this to be about 15% of the size the
MAX_HEAP_SIZE value, but it can be very use-case dependent.
If running the GI garbage collector there is no need to set the HEAP_NEWSIZE value.
If you make a change to the cassandra-env.sh you will need to restart C* for the change to take effect.

Configuring Cassandra - Settings
Cassandra’s main configuration file is called cassandra.yaml
The settings in cassandra.yaml have changed a lot between versions, settings have been added, some have been
expanded upon and a few have been removed, just be aware that things change between versions and pay attention when
upgrading from one C* version to another.
The cassandra.yaml contains setting for, but not limited to: token allocation, hinted handoff, read repair, authentication,
authorization, partitioners, file locations, disk configurations, cache settings, seed settings, bootstrap settings, commitlog
settings, port bindingd, memtable settings….
If you make a change to the cassandra.yaml you will need to restart C* for the change to take effect.

What type of hardware should C* be deployed to?
Answer: Commodity Servers
The current commodity server is spec’d at:
● 8 CPUs
● 64gb RAM
● 1TB of SSD drives (2x 500gb) - directly attached to the backplane!
Do not do any of the following:
● build huge nodes, more nodes are better than larger nodes.
● use load balancers - the driver is data aware.
● use any type of NAS, SAN, CEPH or any other type of shared storage.
● if you need a network cable to get to your storage service, your toast.

How do we provision a C* cluster - Tools
How do we estimate how many nodes we will need for a given workload?
cassandra-stress: the most powerful tool in your arsenal, able to emulate complex read/write/mixed workloads across
many nodes by many emulated clients on many threads.
http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsCStress_t.html
Sebastian’s ‘CQL Data Modeler’: allows to to design a table and see a visual representation, then configure the table. On
the completion of the design and configuration it will generate a stress.yaml as input to the cassandra-stress tool so that
you can immediately stress test your design. http://www.sestevez.com/sestevez/CASTableSizer/
Jonathan Shook’s ‘CQL Test Client’: Many users need a starting point for testing application-specific data models with
some degree of performance. It is not a comprehensive testing tool, nor even a feature rich one like the new Cassandra
stress tool. https://github.com/jshook/testclient

How do we provision a C* cluster - Calculation
● Run your test workload with cassandra-stress emulating clients and read/write
profile.
● Monitor your response times, system.logs and nodetool statistics.
● Tune your JVM heap to your workload.
● Add nodes until all your diagnostics show a healthy system within your
response time SLA over a sustained period.
● Add 20% capacity to your node count for repair.
● Add ?% capacity to your node count for spikes (use-case dependent).

The Apache Cassandra ecosystem

More Related Content

What's hot (18)

Similar to The Apache Cassandra ecosystem (20)

Recently uploaded (20)

The Apache Cassandra ecosystem