What Kiwi.com Has Learned Running ScyllaDB and Go
Download as PPTX, PDF0 likes1,021 views
Martin Sucha, a senior software engineer, shares insights on migrating to ScyllaDB and Go at Kiwi.com, focusing on performance improvements and best practices for handling data across multiple data centers. Key improvements include enhanced cache behavior and custom unmarshal methods to optimize CPU usage during data processing. He also discusses the development of 'easycql', a code generator for marshal/unmarshal implementations, indicating future enhancements to support all data types.
![UnmarshalUDT
type myCustomType struct {
Field1 int
Field2 string
}
func (m *myCustomType) UnmarshalUDT(name string, info gocql.TypeInfo, data []byte) error {
switch name {
case "field1":
return gocql.Unmarshal(info, data, &m.Field1)
case "field2":
return gocql.Unmarshal(info, data, &m.Field2)
default:
return nil // Skip unknown fields
}
}](https://image.slidesharecdn.com/cm12tues245kiwimartinsuchawhatwehave-191111214532/85/What-Kiwi-com-Has-Learned-Running-ScyllaDB-and-Go-14-320.jpg)
What Kiwi.com Has Learned Running ScyllaDB and Go
- 1. What We Have Learned Running ScyllaDB and Go Martin Sucha, Sr. Software Engineer
- 2. Presenter Martin Sucha, Sr. Software Engineer Always amazed what computers can do, Martin loves learning new stuff about them. Previously employee number four at Exponea, a fast-growing European SaaS company, Martin worked with Python and MongoDB and later learned to love Go. Now working on Go services that utilize ScyllaDB at Kiwi.com.
- 3. Migration to Scylla and Go
- 4. Before migration Service written in Python Service written in Python Service written in Python Cache storing data in Redis™ Persistent data stored in Cassandra®
- 5. After migration Service written in Python Service written in Python Service written in Go Persistent data stored & cached in ScyllaDB
- 6. What we use on Go side ■ gocql ■ GOGC=500 ■ gocqlx
- 7. Our cluster ■ 3 data centers in 3 different locations ■ Data replicated to each ■ One replica per data center {'class': 'NetworkTopologyStrategy', 'dc1': '1', 'dc2': '1', 'dc3': '1'} ■ 400 000 reads/second ■ 600 000 writes/second
- 8. What does it mean for applications? ■ Don’t use LOCAL_ONE or LOCAL_QUORUM consistency level ■ Retries will always be to remote DC
- 9. TokenAwareHostPolicy with DCAwareRoundRobinPolicy DC1 Node 1.2 Node 1.1 Node 1.3 Node 1.4 Node 2.2 Node 2.1 Node 2.3 Node 2.4 Node 3.2 Node 3.1 Node 3.3 Node 3.4 DC2 DC3 1 2 3 4 5 6 7 8 9 10 11 12
- 10. NonLocalReplicasFallback option DC1 Node 1.2 Node 1.1 Node 1.3 Node 1.4 Node 2.2 Node 2.1 Node 2.3 Node 2.4 Node 3.2 Node 3.1 Node 3.3 Node 3.4 DC2 DC3 1 2 7 8 3 4 9 10 5 6 11 12
- 12. Upgrade to Scylla 2019.1 ■ Better cache behavior ■ BYPASS CACHE ■ Switch to MC sstables saved us ~35% of disk space Row hits
- 13. Optimizing unmarshalling ■ Our Go service mostly reads data ■ We have some large user defined types ■ We use named types (type MyCustomID = int) ■ gocql.Unmarshal uses reflection and type switches at runtime ■ You can implement custom UnmarshalUDT method (for structs) or UnmarshalCQL method (for any type)
- 14. UnmarshalUDT type myCustomType struct { Field1 int Field2 string } func (m *myCustomType) UnmarshalUDT(name string, info gocql.TypeInfo, data []byte) error { switch name { case "field1": return gocql.Unmarshal(info, data, &m.Field1) case "field2": return gocql.Unmarshal(info, data, &m.Field2) default: return nil // Skip unknown fields } }
- 15. 30% CPU saving in 30 minutes
- 16. Downsides ■ Writing unmarshallers manually is boring ■ Still calls generic gocql.Unmarshal for non-UDT fields ■ Potential for errors
- 17. Generate code
- 18. github.com/kiwicom/easycql ■ EasyJSON + gocql = easycql ■ Code generator for marshal/unmarshal implementations ■ Conservative mode similar to UnmarshalUDT ■ Optimized mode with more optimizations for unmarshalling
- 19. Next steps ■ Support all types ■ Optimize based on field order ■ Optimized mode for marshaling
- 20. Thank you Stay in touch Any questions? Martin Sucha martin.sucha@kiwi.com github.com/martin-sucha
Editor's Notes
- #3: Thanks for introduction. I joined Go team in kiwi.com almost a year ago (=375 days), about a week (9 days) before guys installed last of our ScyllaDB servers during Scylla Summit 2018. So let’s take a look at what happened at Kiwi.com during the last 12 months.
- #4: Back then we were in the middle of migration from Cassandra to ScyllaDB.
- #5: Before the migration we had Cassandra to store our persistent data about flight, bus and train connections. The data was accessed by multiple services that were written in Python. However the performance of Cassandra alone was not sufficient to handle the traffic so there was a cache layer that was storing the data in a bunch of Redis instances. The old setup was not ideal and we seeked to replace it. If you are interested in the details of early phases of the migration, you can watch our talks from last year.
- #6: No need for redis servers Rewritten service from Python to Go Using Scylla allowed us to remove the separate caching layer as Scylla can manage the cache on its own and provides consistent latency. Removing the caching layer simplified operations and also saved us cost for those servers. However, that is not the only change we did. While we still have most of the services written in Python, we have rewritten one of our core services from Python to Go. Go is well suited for writing networking servers that handle a lot of concurrent requests so we don’t need as many worker processes and we also get the benefits of compiled and statically typed code.
- #7: On the Go side, our setup is pretty standard, we use gocql driver to communicate with the database. We initially started with the version maintained by Scylla developers because it has support for shard-aware protocol extension which helps to reduce latency when the driver is used with Scylla. Later we started making changes to the driver to fix bugs and introduce new features so we forked it. We contribute our changes back to upstream projects, both to original gocql and the scylla fork. This sometimes results in duplicate work so I hope those two projects will be merged in the future. As described in gocql documentation, we had to tune the Go garbage collector for better performance. We settled with GOCC=500. We also use gocqlx, which has extensions on top of gocql driver. We use it mainly for query building, binding query arguments from structs and scanning query results back to structs.
- #8: We are running scylla in 3 datacenters in Europe and our data is replicated in all of them. Those data centers were chosen so that they are far enough apart from each other so that a disaster at one site shouldn’t happen at the same time in other sites. However, the locations are close enough so that the latency between the data centers is acceptable for running queries. This allows us to keep only one replica per data center, which saves cost for servers. Audience interaction: Is there anyone else running with one replica per data center?
- #9: LOCAL consistency levels lose meaning. First of all, we don’t want to use consistency levels that work in single data center as we only have one node in a DC holding a piece of data. Using the levels would We evaluated also possibility of using LOCAL_ONE just for reads (and retrying in another DC higher in the stack), but decided not to do that since this would interfere with Scylla feature called heat-based load balancing that helps to keep latency low after nodes start up with cold cache. With a single replica per datacenter, retries will always be to remote datacenters which is not typical case.
- #10: The clients have a policy for selecting the order of hosts to try when executing a query. With multi DC deployments, what is usually used is token aware host policy combined with DC aware host policy, which tries to select hosts based on token from local datacenter. Notice that after the hosts from local datacenter selected by token, the policy selects other nodes from local datacenter - those nodes don’t hold copy of the data and need to forward the request for read queries. We only have one replica per datacenter so in case the request to the node holding the data fails, other nodes in local datacenter can only forward the request to the node we tried or to other datacenter.
- #11: So we introduced a new option for the host selection policy in gocql driver that changes the order in which nodes are selected. With NonLocalReplicasFallback option, we first select hosts that hold a copy of the data by token from the local datacenter, then we select nodes that hold a copy of the data from remote datacenter and only then fallback to nodes that don’t hold a copy of the row.
- #13: During the initial migration from Cassandra to Scylla we used Scylla version 2018.1. Our search engine precomputes data for searching so we are doing full table scans regularly. We have observed that this was affecting our cache performance. We had to drop the caches once in a while to get back the performance. The upgrade to 2019.1 and adding BYPASS CACHE solved the issue for us. After the upgrade we also switched to MC format of sstables which saved us around 35% of disk space
- #14: As we were optimizing other Go components of our service (for example we switched to easyjson for json marshalling) the time has come to look at gocql as the deserialization of data become one of the major parts of our CPU usage. At one point in time it accounted for about 40% of the CPU time the service was using. We have some large user defined types in our database schema with maybe two dozen fields and we use named types in Go for some of the fields for increased type safety. Similar to other marshalling/unmarshalling implementations, gocql uses reflection and type switches internally because it cannot know how your types look in advance. Gocql provides hooks to implement custom unmarshalling code so you can plug a more efficient implementation there.
- #15: It was not immediately obvious from the documentation how the unmarshaller should be implemented at the time and Google search returned only a few results, one of them was the implementation in jaeger. So we learned that UnmarshalUDT implementation could be as simple as a single switch statement calling Unmarshal for each field. This skips some of the reflection work for matching fields in the cql type to those of the Go structure.
- #16: So we went and implemented custom UnmarshalUDT methods for our struct types in like 30 minutes and unmarshalling cql was suddenly 4 times faster. This amounts to about 30% of CPU time that we saved this way.
- #17: There are a few downsides to implementing custom unmarshaller like this. First of all, it is just boilerplate code, we want to focus on business logic instead. The unmarshaller could be even faster if it was tailored to the actual field type that is used instead of using a generic unmarshalling function. And we can introduce errors to our codebase since we have to add fields to multiple places. With more complex unmarshalling logic the potential error is even greater.
- #18: We already used easyjson to generate marshaling code for JSON and similarly we wanted to generate code for unmarshaling user defined types in gocql.
- #19: But there wasn’t a tool like that for gocql. So we combined EasyJSON and gocql into a code generator that can generate unmarshaling code for cql types. First of all, we wanted to automate generating the UnmarshalUDT code so that we don’t need to maintain the marshalers and unmarshalers manually.