DataStax: The Cassandra Validation Harness: Achieving More Stable Releases
2 likes669 views
The document discusses the Cassandra Validation Harness (CVH) aimed at improving the stability and reliability of Cassandra releases by addressing challenges like functional bugs and testing under load. It outlines the architecture of CVH, including its modules and test writing process, and highlights capabilities such as simulating real production environments. The document also mentions upcoming improvements, how to contribute, and encourages community involvement for testing suggestions.
![©2015 DataStax
Sample Test
bridge: CCMBridge
nodeCount: 3
modules:
- [LargeStressWriteModule]
- [TableModule, ArbitraryStressOperationModule]
cassandrayaml: !!map
concurrent_reads: 1000
moduleArgs: !!map
ArbitraryStressOperationModule: !!map
stress_settings: write n=1M
ignoredErrors:
- "RejectedExecutionException"
requiredErrors:
- "Could not validate"
15](https://image.slidesharecdn.com/summittalk-151002225918-lva1-app6891/85/DataStax-The-Cassandra-Validation-Harness-Achieving-More-Stable-Releases-15-320.jpg)
DataStax: The Cassandra Validation Harness: Achieving More Stable Releases
- 1. ©2015 DataStax. Do not distribute without consent. DataStax Philip Thompson Software Engineer The Cassandra Validation Harness: Achieving More Stable Releases 1
- 2. ©2015 DataStax Release Stability • New Major Releases, X.Y.0 • Historically challenging • Few functional bugs, many “at scale” bugs • Problems with doing operator tasks while under load • 6285 - HSHA corrupt data • 10270 - C* stops compacting • 9687 - Wrong partitioner after upgrade • 9116 - Indexes lost on upgrade 2
- 3. ©2015 DataStax Tick Tock • 3.1, 3.2, etc • Reducing scope not enough to reduce defect rate • Need to catch more bugs in testing 3
- 4. ©2015 DataStax Test Retrospective Process • Analyzing every bug filed to the Apache C* Jira by OSS users • http://www.mail-archive.com/ dev@cassandra.apache.org/msg08106.html 4
- 5. ©2015 DataStax Jepsen • Correctness testing • Made famous by Aphyr’s Call Me Maybe series • https://github.com/riptano/jepsen 5
- 6. ©2015 DataStax Cassandra Dtest • Current non-unit test framework • github.com/riptano/cassandra-dtest • Python based tests, backed by CCM 6
- 7. ©2015 DataStax CCM 7 • CCM - Cassandra Cluster Manager • https://github.com/pcmanus/ccm • Simulates a C* cluster on localhost • Starts multiple JVMs, separated with loopback addresses
- 8. ©2015 DataStax Problems w/ existing tests • Mostly short-lived, functional tests • Difficult to coordinate testing features in parallel 8
- 9. ©2015 DataStax CVH • Simulate a real production environment • Multiple, concurrent administrative actions, while cluster is under load • Test the entire configuration space 9
- 10. ©2015 DataStax Architecture • Harness • Bridges • Modules 10
- 11. ©2015 DataStax Bridges • How do we test a real C* cluster? • Entire lifecycle spin up/destroy • Talk to nodes, why not use existing tools • Inspiration from DataStax’s Java driver’s CCMBridge 11
- 12. ©2015 DataStax Modules • Totally independent thread pools, that perform their own operations and validation, communicate to the harness • Signal failures, request shutdown, etc • Arbitrary modules can run in parallel, serial, or any chain of ordering 12
- 13. ©2015 DataStax Module Validation • Data Correctness • OS and C* metrics 13
- 14. ©2015 DataStax Test writing process • Simply write up a .yaml file that defines the modules you need run, and the options to pass them. • Code is called via reflection from yaml options • If the modules you want exist, no need to write new code. 14
- 15. ©2015 DataStax Sample Test bridge: CCMBridge nodeCount: 3 modules: - [LargeStressWriteModule] - [TableModule, ArbitraryStressOperationModule] cassandrayaml: !!map concurrent_reads: 1000 moduleArgs: !!map ArbitraryStressOperationModule: !!map stress_settings: write n=1M ignoredErrors: - "RejectedExecutionException" requiredErrors: - "Could not validate" 15
- 16. ©2015 DataStax Executor Services • Nice way to do concurrent, async message passing • Asynchronous execution of tasks in the background • Accepts Runnables as tasks, returns Futures that complete once the task is done • The calling thread can continue with its own execution, then block on the future to wait for the executor 16
- 17. ©2015 DataStax Harness • WrappedRunnables • CompletableFuture • Receives/Handles signals from Modules • Passes messages between/to Modules • JUnit’s Parameterized Runner 17
- 18. ©2015 DataStax Upcoming Improvements • Bridge for manipulating arbitrary, physical hardware • Attacking the test retrospective backlog • “Permanent” cluster testing • Dynamically created tests • Performance testing 18
- 19. ©2015 DataStax CVH in action • CASSANDRA-9913 - Select * Fails to return all data • Not testable through unit tests • Caught by the LargeStressWriteModule 19
- 20. ©2015 DataStax How to Contribute • For now, work is being done a fork, in preparation for merging into tree • https://github.com/ptnapoleon/cassandra/tree/ validation-notest • Eventually, will either be merged into C* trunk, or turned into its own codebase • Patches/PRs for code, tests, or documentation are welcome and appreciated 20
- 21. ©2015 DataStax How to Contribute • Suggestions/Ideas for new areas to test? • We love those too 21
- 22. ©2015 DataStax Questions? • Ask for ptnapoleon on freenode IRC #cassandra-dev • @ptnapoleon 22
- 23. Thank you