DataStax | Effective Testing in DSE (Lessons Learned) (Predrag Knezevic) | Cassandra Summit 2016
2 likes810 views
The document discusses the transition to a Gradle-based testing framework in DSE for improved automated testing and quality control. It highlights the benefits of parallel test execution, shorter testing times, and enhanced feedback cycles. Additionally, it covers the use of Docker for consistent test environments and logging strategies for effective test management.
![Log Event Router
<appender name="SIFT" class="ch.qos.logback.classic.sift.SiftingAppender">
<discriminator class="com.datastax.bdp.test.ng.LogFileDiscriminator">
<key>TEST_LOG_FILE</key>
<defaultValue>system.log</defaultValue>
</discriminator>
<sift>
<appender name="FILE-${TEST_LOG_FILE}" class="ch.qos.logback.core.FileAppender">
<filter class="com.datastax.bdp.test.ng.ForbiddenLogEventsDetector">
<logFile>${TEST_LOG_FILE}</logFile>
<detector class="com.datastax.bdp.test.ng.NettyLeakDetector"/>
</filter>
<encoder>
<pattern>%X{nodeid} %5p [%t] %d{ISO8601} %F (line %line) %m%n</pattern>
<immediateFlush>false</immediateFlush>
</encoder>
<file>${LOG_DIR}/${TEST_LOG_FILE}</file>
<append>true</append>
</appender>
</sift>
</appender>
© DataStax, All Rights Reserved. 19](https://image.slidesharecdn.com/effectivetestingindse-160928181032/85/DataStax-Effective-Testing-in-DSE-Lessons-Learned-Predrag-Knezevic-Cassandra-Summit-2016-19-320.jpg)
DataStax | Effective Testing in DSE (Lessons Learned) (Predrag Knezevic) | Cassandra Summit 2016
- 1. Effective Testing in DSE Predrag Knežević predrag.knezevic@datastax.com @pedjakknezevic
- 2. Why Taking Care? • Automated testing for quality control of shipped product/service • Number of tests and total testing times increase over time • Shorter delivery cycles → continuous testing • Run tests on each pre-merge check, but • Keep feedback cycles short • Ensure repeatable test execution anywhere © DataStax, All Rights Reserved. 2
- 3. DSE Build Facts • Junit based test infrastructure • December 2014 (DSE 4.6) • Ant based build system • ~5h for running all tests on Jenkins, with a rather complicated job layout • July 2016 (DSE 4.7+) • Gradle based build system • 40-60mins for running all tests on Jenkins • 16 hours of total testing time • The number of tests doubled! • Repeatable test execution across all machines • Simple setup © DataStax, All Rights Reserved. 3
- 4. Why Moving to Gradle? • Built-in support for parallel test execution • Readable - build scripts based on Groovy (easy learning for Java devs) • Repeatable builds/environment setup across machines • Powerful dependency management • Sane conventions, but configurable when needed • Easy project modularization • Excellent Eclipse/IntelliJ support • Easy extendable through plugins or additional Java/Groovy code living in the script or project • All Ant tasks still available © DataStax, All Rights Reserved. 4
- 5. Running Tests • All: gradlew test • Single: gradlew test -Dtest.single=FooTest • By default sequential execution • Low resources usage on modern multicore hardware • Long test round duration © DataStax, All Rights Reserved. 5 Main Test Worker Test Queue TestClass1 TestClass2 TestClass3 TestClass4
- 6. Parallel Test Execution © DataStax, All Rights Reserved. 6 build.gradle test { maxParallelForks = N } Main Test Worker Test Queue TestClass1 TestClass2 TestClass3 TestClass4 Test Worker Test Queue TestClass1 TestClass2 TestClass3 TestClass4 . . N
- 7. Dockerized Test Execution © DataStax, All Rights Reserved. 7 Main . . Test Worker Test Queue TestClass1 TestClass2 TestClass3 TestClass4 Test Worker Test Queue TestClass1 TestClass2 TestClass3 TestClass4 N github.com/datastax/gradle-dockerized-test-plugin
- 8. Dockerized Test Execution (2) © DataStax, All Rights Reserved. 8 build.gradle test { maxParallelForks = N docker { image = ’test-image’ } } • Install Docker locally (native or boot2docker) • No changes on production or test code required • Test environment • Consistent across all machines (dev + CI) → no more “it works on my machine” • Managed as code (Dockerfiles) within project • Easy machine bootstraping • Fully isolated • Easy testing against several and/or appropriate environment
- 9. Worker Starvation © DataStax, All Rights Reserved. 9 Main . . Test Worker Test Queue TestClass1 TestClass2 TestClass3 TestClass4 Test Worker Test Queue TestClass1 TestClass2 TestClass3 TestClass4 Static queue allocation!
- 10. Improved Queue Management © DataStax, All Rights Reserved. 10 Test Worker Main Test Queue TestClass1 TestClass2 TestClass3 TestClass4 Test Worker
- 11. Test Scheduling © DataStax, All Rights Reserved. 11 Main . . Test Worker Test Queue TestClass1 TestClass2 TestClass3 TestClass4 Test Worker Test Queue TestClass1 TestClass2 TestClass3 TestClass4 • NP-hard • Longest Processing Time (LPT) algorithm • History access required
- 12. More Workers for Faster Feedback • Powerful multi-core machine • Docker Swarm • Virtual Docker engine • Test workers run on a Swam node • Cluster can shrink or grow • Lower bound for the test round duration is equal to the duration of slowest test class © DataStax, All Rights Reserved. 12 Swarm master Main Swarm node Swarm node Test Worker Test Worker Test Worker Test Worker
- 13. Split Slow Test Classes for More Throughput class FooTest { @Test void bar1() { } @Test void bar2() { } } class Foo1Test { @Test void bar1() { } } class Foo2Test { @Test void bar2() { } } ● Manual ○ Group by common fixture ○ Extreme: one test per class ● Bytecode manipulations (auto split annotated class) ● Grouping tests classes into JUnit suites forces their sequential execution!
- 14. No Embedded DSE Nodes • Running cluster within single test worker JVM possible only by using separate classpath loaders • Still requires a good amount of hacking to make it work decently • Node shutdowns might still be problematic • Thread can hang around • Some objects cannot be garbage collected → memory leaks • Standalone nodes enable reusage across test classes © DataStax, All Rights Reserved. 14
- 15. Remote Code Execution • MobilityRPC library (http://github.com/npgall/mobility-rpc) • Remote JUnit Testrunner (http://github.com/datastax/remote-junit-runner) • Useful for integration tests requiring application context, but application cannot be easily embedded into test JVM • Support any existing JUnit runner on the remote side (Parametrized, Spock, Suites) © DataStax, All Rights Reserved. 15 @RunWith(Remote.class) public class RemoteTest { @Test public void foo() { // test } }
- 16. Injecting Faults • JBoss Byteman (http://byteman.jboss.org) • Inject at runtime • Exceptions • Delays • Arbitrary side-effects • Enables/simplifies testing of edge/uncommon cases © DataStax, All Rights Reserved. 16
- 17. Logging • Logback (http://logback.qos.ch/) based infrastructure • Log file per a test case • Send all log messages to the single logback server asynchronously (reactor-logback adapter) keeping DSE nodes responsive • Route log statements to the proper file using SiftingAdapter and appropriate discriminator • Use JUnit rules to mark the beginning and the end of a test and propagate this information to the logback discriminator • Write thread dumps in case of a failure • Scan log files for known issues and fail tests if they occur (e.g. Netty memory leaks) • Turn DEBUG messages on to help later digging in a case of failure © DataStax, All Rights Reserved. 17
- 18. Log Event Sender <appender name="SOCKET" class="com.datastax.bdp.logback.SocketAppender"> <remoteHost>${logbackServer}</remoteHost> <port>12345</port> <reconnectionDelay>1 seconds</reconnectionDelay> <nodeId>${nodeid}</nodeId> <eventDelayLimit>120 seconds</eventDelayLimit> </appender> <appender name="ASYNC" class="reactor.logback.AsyncAppender"> <includeCallerData>true</includeCallerData> <appender-ref ref="SOCKET"/> </appender> <root level="DEBUG"> <appender-ref ref="ASYNC"/> </root> © DataStax, All Rights Reserved. 18
- 19. Log Event Router <appender name="SIFT" class="ch.qos.logback.classic.sift.SiftingAppender"> <discriminator class="com.datastax.bdp.test.ng.LogFileDiscriminator"> <key>TEST_LOG_FILE</key> <defaultValue>system.log</defaultValue> </discriminator> <sift> <appender name="FILE-${TEST_LOG_FILE}" class="ch.qos.logback.core.FileAppender"> <filter class="com.datastax.bdp.test.ng.ForbiddenLogEventsDetector"> <logFile>${TEST_LOG_FILE}</logFile> <detector class="com.datastax.bdp.test.ng.NettyLeakDetector"/> </filter> <encoder> <pattern>%X{nodeid} %5p [%t] %d{ISO8601} %F (line %line) %m%n</pattern> <immediateFlush>false</immediateFlush> </encoder> <file>${LOG_DIR}/${TEST_LOG_FILE}</file> <append>true</append> </appender> </sift> </appender> © DataStax, All Rights Reserved. 19
- 20. Test Start/End Detection @Rule public TestRule watcher = new TestWatcher() { private String logFile; @Override protected void starting(Description description) { logFile = description.getClassName()+"/"+description.getMethodName()+".log"; logToFile(logFile); } protected void failed(Throwable e, Description description) { threadDumpLogger.error("Test {}.{} failed, thread dump:n{}n", description.getClassName(), description.getMethodName(), getThreadDumps()); logToFile(description.getClassName()+"/after.log"); } protected void finished(Description description) { logToFile(description.getClassName()+"/after.log"); ForbiddenLogEventsDetector.checkForIssues(logFile); } }; © DataStax, All Rights Reserved. 20
- 21. Resources • Gradle (gradle.org) • Docker (docker.com) • Gradle Dockerized Test Plugin (github.com/datastax/gradle-dockerized-test-plugin) • MobilityRPC (http://github.com/npgall/mobility-rpc) • Remote JUnit Testrunner (http://github.com/datastax/remote-junit-runner) • JBoss Byteman (http://byteman.jboss.org) • Logback (http://logback.qos.ch/) • Project Reactor Addons (github.com/reactor/reactor-addons), Logback adapter © DataStax, All Rights Reserved. 21
- 22. Thank you! Questions? © DataStax, All Rights Reserved.22