Fine-tuning Group Replication for Performance
- 1. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. Fine-tuning Group Replication for performance FOSDEM, 4th of February 2017 Vítor Oliveira (vitor.s.p.oliveira@oracle.com) Senior Performance Engineer 1Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
- 2. Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 2
- 3. In this session, I will present Group Replication from the perspective of a performance optimizer. It will show the main moving parts, how we can use the available options to tune its behaviour, and show a few significant benchmark results. For a performance evaluation of Group Replication please visit: http://mysqlhighavailability.com/performance-evaluation-mysql-5-7 -group-replication/ Summary 3
- 4. Anatomy of Group Replication Performance enhancement options Flow-control considerations A bit more benchmarking Conclusion Contents 4 1 2 3 4 5
- 5. Anatomy of Group Replication (from a performance perspective) 1
- 6. 6 Transaction Hook Begin Execute transaction (until prepare) Needs to throttle? Rollback Commit Delay until next flow-control period Yes Collect writeset information Send message for ordering by GCS Wait result from certification thread Certification positive? No Certification certification result Group Communication WRITER-SIDE
- 7. 7 Certifier Loop Start Quit? End Yes Get next transaction in queue Certify transaction Is transaction local? Applier Certification positive? User Thread Send certification result to user thread Add transaction events to relay log Yes Yes signal release GCS Receive Thread Ordered Transaction Queue CERITFIER
- 8. Main factors affecting performance: • Network bandwidth and latency To agree in a specific order of transactions Group Replication needs to send them to all members of the group and wait for the majority to respond, which consumes the network bandwidth and adds at least one network RTT to the transaction latency. • Certification throughput Transactions are certified in the agreed message delivery order and sent to the relay log by a single thread. That can become a contention point if the certification rate is high or the storage system that stores the relay log is slow. • Remote transaction applier throughput Remote transactions can be applied by the single or the multi-threaded applier, but parallelism should be properly explored so that the non-writer members can keep up with the writers. 8
- 9. 9 Transaction Hook Begin Execute transaction (until prepare) Needs to throttle? Rollback Commit Delay until next flow-control period Yes Collect writeset information Send message for ordering by GCS Wait result from certification thread Certification positive? No Certification certification result Group Communication WRITER-SIDE
- 10. 10 Certifier Loop Start Quit? End Yes Get next transaction in queue Certify transaction Is transaction local? Applier Certification positive? User Threads Send certification result to user thread Add transaction events to relay log Yes Yes signal release GCS Receive Thread Ordered Transaction Queue CERITFIER
- 12. Network bandwidth and latency: 1. Use high bandwidth and low-latency networks – If needed hide latencies by using many concurrent connections 2. Options to reduce the bandwidth required – group_replication_compression_threshold=<size: 100> – binlog_row_image=MINIMAL 3. Reduce latency with busy waiting before sleeping – group_replication_poll_spin_loops=<num_spins> 12
- 13. Certifier throughput: 13 1. Use high-performance storage for the relay log – Improve the disk bandwidth as each event is sent by the certifier thread to the relay log while also being read by the applier. 2. Use fast temporary directory – Transaction writesets are extracted using the IO_CACHE, and that may need to spill-over to disk on large transactions. – tmpdir=/dev/shm/... 3. Reduce certification complexity in multi-master – group_replication_gtid_assignment_block_size=<size: 10M>
- 14. Applier throughput: 14 1. Apply remote transactions with the LOGICAL_CLOCK scheduler and enough worker thread parallelism to keep up with the writers: – slave-parallel-type=LOGICAL_CLOCK – slave-parallel-workers=8/16/+ 2. Take advantage of writeset-based transaction dependency tracking
- 15. GRAPPLIER 15 Applier throughput: 2. Writeset-based transaction dependency tracking allows Group Replication to reach higher slave throughput with less client threads. 1. When using high performance storage (RAMDISK) and reduced number of clients the throughput of the slave applier based on binary log group commit is reduced.
- 17. Flow-control goals: • Allow new members to join the group when writing intensively Nodes entering the group need to catch up previous work while also storing current work to apply later. This is more demanding then just applying remote transactions, so the cluster may need to be put at lower speed for new members to catch up. • Reduce the number of transactions aborts in multi-master Rows from transactions in the relay log cannot be used by new transactions, otherwise the transaction will be forced to fail at certification time. • Keep members closer for faster failover Failing over to an up-to-date member is faster as the back-log to apply is smaller. • Keep members closer for reduced replication lag Applications using one writer and multiple readers will see more up-to-date information when reading from other members then the writer. 17
- 18. Flow-control approach: • Writer decision The writers will throttle themselves when they see a large queue on the other members, the delayed members don’t have to spend extra time dealing with that. • Coarse grain The flow-control does not micro-manage the synchrony between nodes, it just expects that it is enough to correct course over the long run. • Can be disabled as needed Flow-control can disabled as needed to address situations less fit for it, in which case it works just like asynchronous replication. • Options to use --flow-control-mode=QUOTA* | DISABLED --flow-control-certifier/applier-threshold=25000* 18
- 19. 19 FLOW-CONTROL MAINLOOP Flow-control Loop Start Quit? End Yes Find members with excessive queueing Send stats message to group Are all members ok? Throttling Active? quota = min. capacity / number of writers (minus 10%, up to 5% of thresholds) Release throttling gradually (50% increase per step) Yes Stats Messages Receiver no wait one second & release trans Member Execution Stats
- 20. Flow-control throughput effects 8 16 32 64 128 256 0 2 500 5 000 7 500 10 000 12 500 15 000 Throughput varying Flow-control: Sysbench OLTP RW (9 members) flow-control disabled default threshold (25000) small threshold (1000) number of client threads totaltransactionspersecond(TPS) GROUPSIZE=9 20
- 21. A bit more benchmarking (using Sysbench OLTP RW) 4
- 22. Single-master throughput and latency: 8 16 32 64 128 256 0 2 500 5 000 7 500 10 000 12 500 15 000 0 15 30 45 Sysbench OLTP RW on a 9 member group Asynchronous (sustained throughput) Group Replication (sustained throughput) Latency Latency number of clients/threads maximumsustainedthroughput(transactionspersecond) clientlatency(ms) (throughput: higher is better) (latency: lower is better) GROUPSIZE=9 22
- 23. Multi-master maximum throughput: 3 members 5 members 7 members 9 members 0 2 500 5 000 7 500 10 000 12 500 15 000 Single- and multi-master scalability: Sysbench RW Asynchronous one writer Group Replication: one writer two writers three writers number of group members maximumsustainedthroughput(TPS) SCALABILITY (non-durablesettings) 23
- 24. Effects of network round-trip latency 8 16 32 64 128 256 512 1024 0 2 500 5 000 7 500 10 000 12 500 15 000 Effects of network round-trip latency: Sysbench RW 10Gbps LAN 1 node @ 7ms, 200Mbps 1 node @ 50ms, 200Mbps number of client threads throughput(TPS) HIGH-LATENCYNETWORKS THREEMEMBERS 24
- 25. 0 60 120 180 240 0 2 500 5 000 7 500 10 000 Peak Single-master Throughput over Time: Sysbench OLTP RW (durable settings, 64 clients, 9 members) Asynchronous Group Replication time (sec) averagethroughput(transactionspersecond) (higher is better) STABILITYStability over time 25
- 26. Conclusion5
- 27. Conclusion • For performance one needs to focus on the three areas: group communication, certifier and applier throughputs. • Group Replication has high-performance out of the box: – High throughput and low-latency vs asynchronous replication; – Scalable to a significant number of members and client threads; – Optimized for low-latency network but can already widthstand high- latency networks well. • Group Replication has reach GA status very recently, so the fun is just begining... 27
- 28. Thank you. Any questions? • Documentation – http://dev.mysql.com/doc/refman/5.7/en/group-replication.html • Performance blog posts related to Group Replication: – http://mysqlhighavailability.com/category/performance/ 28