Apache Cassandra Lunch #74: ScyllaDB - Peter Corless

Editor's Notes

• #3: Hello to everyone attending the Cassandra.lunch today. I wanted to first take the time to thank our hosts, Anant Corporation. You will be able to find this and other great talks about NoSQL and distributed databases in their Github repository.
• #4: My name is Peter Corless. I’m the Director of Technical Advocacy at ScyllaDB. I listen to our users’ stories and discover wisdom in each to share with our other practitioners in the industry at large, whether through blogs or in technical presentations.
• #5: Today we’re going to compare Apache Cassandra against Scylla using some objective, observable analysis — from software release cycles to github commits. Also, I want to come here today to shatter an illusion — that of the drop-in replacement. Yes, I know that we at ScyllaDB have ourselves often said we’re just that., But I want to point out that while there are many similarities, there are also many points of departure — in design, in implementation, and in capabilities. Lastly, I wanted to share with you some benchmarking results from tests we conducted. Both Cassandra 3.11 against its own successor, Cassandra 4.0 as well as against Scylla Open Source 4.4.
• #6: So let’s get right into it.
• #7: We talked about Cassandra 4.0 in 2019. And 2020. By then it had reached beta. Where it would linger for another calendar year. It even missed the deadline for its own world party. So in many ways I was as excited and relieved as anyone in the industry when I heard it was really, finally here in July 2021. But that was four long years between the last minor release and this major release. A long time in database years. Also, I am glad to note the first Cassandra 4.x maintenance release also came out in September, which means they are establishing a new cadence for software release. And let me show you why that’s important.
• #8: Here you can see the calendar of milestone releases as of the time when we conducted our benchmarks of Cassandra 4.0 this past August. You can just see that glaring multiyear chasm between the release of Cassandra 3.11 and Cassandra 4.0. If you go back earlier to the last major release of Cassandra, 3.0 you can see that it was back in 2015. That was around the time the Scylla Open Source Github repository was being created. I think we were on 0.1.0 around then. So over this same historical span of time, Scylla was birthed into the world and has had four major releases. And within those major releases multiple minor releases. As well as dozens and dozens of maintenance releases. Now, the reason these cadences matter is because it shows the velocity of your software supply chain. It shows how quickly you can innovate. It shows how quickly you can close critical bugs. It was one of the main reasons that some companies, like Expedia, felt they could trust Scylla more than Cassandra. Simply because of a dependable, regular release cadence.
• #9: Also, on top of our Scylla Open Source releases we have an annual cadence of Enterprise releases, with additional testing, with additional features, and with additional customer expectations for stability. In most cases, our users will see new features and capabilities in early deployment in Scylla Open Source first. Then, as they mature and harden, we’ll move them to General Deployment, or GD status. Only then will you see them synched down into an Enterprise release. There are also some special Enterprise-only features, which I won’t get into today, but just know that we’re running these two trains side-by-side.
• #10: Users see the velocity of development in the number of commits per week. Sometimes Scylla exceeds a hundred a week. You might also be surprised to discover that, across 2021, there are about 50% more people committing to Scylla Open Source than to Apache Cassandra. In terms of the major and minor releases made between 2018 and today, Apache Cassandra 4.0 is the only major release they have had across this period of time. Whereas the Scylla team pumped out a baker’s dozen of freshly baked releases. We have more github stars than Cassandra now, and the gap between the two is widening. There are a lot of “Scylla curious” people out there. We’d love to set them up with a cluster.
• #11: Historically, since Scylla released, we have been perceived as “chasing Cassandra,” working to achieve feature parity. So this meant that thorough 2020 we were playing catch up. However, with Scylla Open Source 4.0 we went beyond feature completeness. We suddenly had features Cassandra didn’t have at all. Plus we had features that might have a similar name, but we had implemented differently. Often radically so. Lastly, they keep adding commands, features and formats to Cassandra. For example the SSTables formats changed once between 4.0 beta and release candidate 1 and then again in the final release.
• #12: What this results in is the following kind of buckets of features. Some core features of Cassandra, which Scylla has also implemented in its core — the same-same. Same configuration. Same command line inputs and outputs. Same wire protocol. And so on. Then there are some things that are unique to Cassandra. For instance, the Cassandra 4.0 features. Some of these we plan to add in due time. Some simply may not be appropriate to implement because of the very different infrastructure and design philosophies, even the code bases. For instance, you won’t find Java-specific features like you would have in Cassandra. Inversely, you’ll have some features in Scylla that they just won’t implement in Cassandra. These are points of divergence which could become showstoppers for migration, if you depended on them in your use case. Or they may be specific reasons to migrate if they represent features or capabilities the other database just will never offer you, and you really need them. Lastly, there is a mix of features that may be called by the same name, or may sound quite similar, but are actually implemented uniquely across Cassandra and Scylla. So while Scylla began by chasing Cassandra, now, many of our features are beyond Cassandra. While we remain committed to making our database as feature complete and compliant to Cassandra as possible and pragmatic, it will be quite interesting to watch as current points of departure between the two become narrowed or widened over the coming years.
• #13: Let’s start with the common ground. If you are familiar with Cassandra today, what can you expect to feel as comfortable and natural in Scylla?
• #14: First, the common ancestry. Many of the principles between Cassandra and Scylla are directly correlated. In many ways, you could call Cassandra the “mother” of Scylla in our little database mythological family tree here. We also pull in some of our DNA from Amazon DynamoDB.
• #15: Ah! Home sweet home. The concepts of keyspaces and tables. Pretty much standard CQL here.
• #16: Basic CRUD operations as well. All quite familiar. Comfortable as a favorite sweater.
• #17: The high availability architecture that Cassandra is known for is likewise found in Scylla. Peer-to-peer leaderless topology. Replication factors and consistency levels set per request. Multi datacenter replication which allows you to be able to survive a full datacenter loss. All typical “AP”-mode database behavior.
• #18: Lastly, you have the underlying ring architecture. The key-key-value scheme of a wide column database. Partition keys and clustering keys. Nodes and vNodes. Automatic sharding and the murmur3 partitioner.
• #19: While there are still more features that are alike, let’s not be exhaustive. Let’s move on to what seems similar between the two, but really, are just not the same.
• #20: CQL. That’s right. Cassandra Query Language implementation itself. While most of the basic commands are the same, you will note Scylla may have implemented CQL commands that do not appear in Cassandra. Or vice versa. There’s also version level completeness. For example, Cassandra’s CQL is up to 3.4.5, while Scylla’s implementation is only at 3.4.0. What are the specific differences between them? I’ll let you scour the docs as a homework assignment. While some of what you find may read as pretty trivial differences, if you were migrating between the two, any unexpected discoveries might represent unpleasant show-stoppers until we finally reach CQL parity and completeness again.
• #21: Next: SSTables! Scylla is compatible with Cassandra 3.11’s latest “md” format. But did you spot the difference with Cassandra 4.0?In the first release candidate they snuck out the “na” format which added a bunch of small changes. And then when 4.0 itself shipped, they added a way to store the originating hostID in “nb” format SSTable files. We’ve opened up a Github issue to make sure Scylla will have “na” and “nb” format compatibility in due time — #8593 — but this is the sort of common, everyday feature chasing you have whenever new releases of anything are spun, and everyone else needs to ensure compatibility. There’s always a little lag and gap time before implementation.
• #22: Lightweight Transactions, or LWTa, are pretty much the same sort of thing on both systems to do compare-and-set or conditional updates. But on Scylla they are simply more performant, because instead of four round trips as with Cassandra, we only require three. What this has led to in practice is that some folks have tried LWT on Cassandra, only to back them out when performance tanked, or didn’t meet expectations. So if you experimented with LWTs in Cassandra, you might want to try them again with Scylla.
• #23: Materialized Views, or MVs, are another case where Scylla put more polish into the apple. While Cassandra has had materialized views since 2017, they’ve been problematic since first introduced. At Distributed Data Summit 2018 Cassandra PMC Chair Nate McCall told the audience that “If you have them, take them out.” I remember sitting in the audience absorbing the varied reactions as Nate spoke frankly and honestly about the shortcomings of the implementation. Meanwhile, the following year Scylla introduced its own implementation of materialized views. They served as the foundation for other features, such as secondary indexes. While MVs can still get out of sync from the base table, it is not as likely or easy to do. ScyllaDB engineers have poured a lot of effort over the past few years to get materialized views “right.” And we consider the feature ready for production.
• #24: Speaking of secondary indexes, while you have them in Cassandra, they are only local secondary indexes — limited to the same base partition. They are efficient but they don’t scale. Global secondary indexes, which are only present in Scylla, allow you to index across your entire dataset, but can be more complicated and lead to unpredictable performance. So you want to be more judicious how and when you implement them. The good news is Scylla supports both local and global secondary indexes, and you can apply both on a column, to run your queries as narrow or as broad as you wish.
• #25: Change Data Capture, or CDC, is the most dramatic difference between Cassandra and Scylla. Cassandra implements CDC as a commitlog-like structure. Each node gets a CDC log, and then, when you want to query them, you have to take these structures off-box, combine them and de-dupe them.Now if I misunderstand that, please follow-up with me afterwards. I’d love to hear how Cassandra practitioners are using CDC. But think about the design decisions that went into Scylla’s CDC implementation. First, it uses a CDC table that resides on the same node as the base table data, shadowing any changes to those partitions. Those CDC tables are then queryable using standard CQL. So the results you get are already going to be deduped for you. There’s no log merges necessary. You get a stream of data, whether that includes the diffs, the pre-images, and/or the post-images. However you want to consume it. We also have a TTL set on the CDC tables so they don’t grow unbounded over time.
• #26: This made it very easy for us to make a Kafka CDC Source Connector based on Debezium. It simply consumes the data from the CDC tables using CQL and pumps it out to Kafka topics. No muss, no fuss.
• #27: Here’s another example of a point of departure. Cassandra historically had problems with streaming SSTables. This can be important when you are doing topology changes and you need to bring up or down nodes and rebalance your cluster. Zero copy streaming means you can take a whole SSTable — all of its partitions — and copy it over to another node without breaking an SSTable into objects, which creates unnecessary garbage that then needs to be collected. It also avoids bringing data into userspace on the transmitting and receiving nodes. Ideally this was to get you closer to hardware IO bounds. Scylla, however, had already radically changed how it was going to do internode copying. We used row-level repairs instead of standard streaming methodology. This was more robust, allowing mid-point stops and restarts of transfers, was more granular — meaning you were only sending the needed rows instead of the entire table — and more efficient overall. So these are fundamentally different ways to solve a problem. I’ll show you the results of these different approaches when I get to the benchmark section.
• #28: Netty Async Messaging is a good thing. Any way to avoid blocking and bottlenecking operations is awesome. Also, the way it does thread pools meant you weren’t setting a fixed number of threads per peer, which could mismatch actual real-world requirements. Scylla has always believed in non-blocking IO. It is famous for its “async everywhere” C++ architecture. Plus, the shard-per-core design meant that you were minimizing inter-core communications as much as possible in the first place. Again, these were good things. But for Cassandra it was an evolutionary realization they wove into their existing design, whereas for Scylla it was a Day One design decision, which we’ve improved upon a lot since. In summation, sort of doing the same thing, but in very different ways.
• #29: Kubernetes. Cassandra now has a range of options for Kubernetes, from DataStax’ K8ssandra (which replaces the now-deprecated cass-operator), or KassCop by Orange, to Bitnami Charts. If anyone has feedback on any of them in particular, I’d encourage you to contact me after this talk. I’d love to learn from your experiences. For Scylla, we have our own Scylla Operator. Our users like it just swell. So yes, Kubernetes is available for both. But each operator is purpose-built for each respective database.
• #30: Now let’s look at things that are just simply… different. From fundamental design decisions to implementation philosophies to even the vision of what these database platforms are and can do.
• #31: A critical Day One decision for Scylla was to build a highly distributed database upon a shared-nothing, shard-per-core architecture — the Seastar framework.. Scale it up, or scale it out, or both. it is a greedy system, and will gladly utilize whatever resources you grant it. Scylla is designed to make maximum utilization out of all the hardware you can throw at it. Because of this, Scylla can take advantage of any size server. 100 cores? Sure. A 1,000 cores? Don’t laugh. I know of a company working on a 2,000 core system. Such hyperscale servers will be available before you know it. In comparison, Cassandra shards per node. It also gets relatively low utilization out of the system it’s running in.That’s just the nature of a JVM — it doesn’t permit you knowledge of or control over the underlying hardware. Which is why people seek to run multi-tenant in the box — to utilize all those cycles that Cassandra can’t harness.
• #32: Alternator. It’s our name for the Amazon DynamoDB-compatible API we’ve build into Scylla. Which means you now have freedom. You can still run your workloads on AWS, but you might find that you get a better TCO out of our implementation running on our Scylla Cloud Database-as-a-Service instead. Or you might be able to migrate your workload to Google Cloud, or Azure, or even put in on-premises. An interesting example of the latter is AWS Outposts. These are cages with AWS servers installed in your own premises. These servers act as an on-premises extension of AWS. Because we were capable of being deployed anywhere, Scylla was chosen as AWS’ method to deploy your DynamoDB workloads directly into an AWS Outposts environment. Getting back to our Alternator implementation, using our CDC feature as the underlying infrastructure, we also support DynamoDB Streams. Plus we have a load balancer to round out the same-same expectations of existing DynamoDB users. Lastly, our Scylla Spark Migrator makes it easy to take those DynamoDB workloads and place them wherever your customer desires.
• #33: There are many, many other things I could have picked out, but I just wanted to show this as one more example of a “quality of life” feature for the database administrators. Seedless gossip. There’s been a lot of pain and suffering if you lost a seed node. It requires manual assignment. Seed nodes won’t just bootstrap themselves. It can cause a lot of real-world, real-time frustrations when your cluster is at its most temperamental. So one of our engineers came up with the brilliant idea of just … getting rid of seed nodes entirely. Reworking the way gossip is implemented to be more symmetric and seamless. I hope you have a chance to read this article on how it was done. Because I promise it’s pretty juicy!
• #34: So next, let’s get down to more hard numbers. These benchmark results were published in August 2021, so they are pretty current.
• #35: The first thing to note is that throughputs for Cassandra between 3.11 and 4.0 aren’t that much different. And in all cases they are significantly lower than what Scylla could achieve on the exact same hardware — 2 to 5 times slower. There are some workloads for Cassandra 4.0 that saw improvements of 25% or more, up to 100% improvement in a high cache hit rate scenario. But for many tests, the achievable throughputs for Cassandra 4.0 over Cassandra 3.11 didn’t budge much at all. What did radically change for Cassandra 4.0 were the latencies. In some cases, p99 latencies dropped by 95% to 99% between Cassandra 3.11 and 4.0. But even then, in many cases the latencies were often still far higher than Scylla, up to 100x worse. So sure, if you are running Cassandra 3.11, under many conditions it makes sense to upgrade to Cassandra 4.0. You will see massive latency drops and depending on the workload, some significant performance gains. But now, having waited patiently into 2021, are these performance gains sufficient to warrant recommending Cassandra 4.0? In many cases, it just makes more sense to migrate to Scylla. Or just start there for a greenfield project.
• #36: Let’s look at throughput versus latencies. In this use case, you can see how Cassandra’s long-tail latencies went nonlinear really quickly, in the 30k-40k operations per second range. Scylla, meanwhile, continued to scale far more linearly and kept p99 latencies lower than 10ms out to around 170k operations per second, providing four to six times the throughput — all on the exact same hardware.
• #37: Replacing a node. I hear it happens in actual production networks from time-to-time. Here you can see that while Zero Copy Streaming and other performance improvements in Cassandra 4.0 definitely does provide some speed improvements, it’s still going to be multiple times slower than Scylla can perform the same node replacement. Why are you waiting more than three hours in a day to replace a single node, when you can do the same operation in less than an hour? Because while you are busy streaming, you’re still down down a node, with your performance suffering a low grade fever until the cluster finally rebalances.
• #38: This real-time delay is even more pronounced if you need to scale a cluster by more than one node. Say you want to double capacity. With Scylla’s faster per-node provisioning, you can double the entire cluster — the whole thing — faster than it would take Cassandra 4.0 to even get the first node up and running.
• #39: But this is killer. Look at this closely. Your eyes might have missed that first blue bar in the chart. That’s Scylla. Scylla optimizes compaction speed so much that it can run compactions in all of 36 minutes on a 3-node cluster. Boom! Done. Meanwhile, Cassandra can take nearly a day, even after we tuned it (by default it took a day and a half to run compactions). So, yes, with tuning, Cassandra 4.0 is moderately better than Cassandra 3.11, but why are you needlessly inflicting pain and suffering on your DevOps team?
• #40: Okay. In this test, we decided to switch things up. Rather than run on the exact same hardware, this test ran on clusters more optimized for the respective databases. Here, we chose i3.metal instances for Scylla — because Scylla, written in C++, can run close to the metal. Whereas Cassandra, written in Java, still doesn’t have full utility of the server its running on, even with the newer JVMs. They are better, yes, but you’re still walled off from truly understanding the hardware you’re operating on. So in this case, we have Cassandra on i3.4xlarge instances. We gave Cassandra 40 of those nodes. For 640 vCPUs in total. Meanwhile, Scylla was on 4 i3.metals. That’s only had 288 vCPUs — Less than half of the Cassandra test cluster. But… Scylla was able to maintain the same or better performance, even being outgunned. Because we really take advantage of every CPU cycle we can get our hands on. What resulted was a 60% savings in total cost of ownership, but also, a reduction of 90% of the administrative burden — and attack surface. You simply have a tenth of the nodes to watch over. Which makes a difference as you continue to scale.
• #41: These are links to the published benchmarks you can find on our website. You can also watch the on-demand webinar we did with the engineers who ran the benchmark. The Q&A that followed was also really engaging, and I think you’ll get a lot of insight out of it. It goes into far more depth than I can cover here today.
• #42: Meanwhile, if you’re intrigued and to learn more about Scylla, we have a wonderful upcoming free online Scylla University LIVE training event this coming Tuesday, November 9th for the Americas, and Wednesday, November 10th for Europe, Africa and Asia/Pacific. You can check out the link to register.
• #43: Also, if you can’t make that live event, we have free self-paced coursework available in Scylla University to add to your NoSQL expertise. We even give you course completion certificates you can post on LinkedIn.
• #44: Okay. Let’s open up for questions and answers. I’d love to hear your thoughts. … After Q&A… Thank you all for attending my Cassandra.lunch. As a thank-you, I just wanted to share with you all a link to get yourself some free Scylla goodies. I’ll put the link in the chat.
• #45: With that, thank you. You can always follow up with me at petercorless on Twitter. And if you want to learn more about highly available databases like Scylla, again, check out the free online courses at university.scylladb.com. Have yourself a great day, and, as ever, “onwards to adventure!”