DataStax: Making Cassandra Fail (for effective testing)
2 likes2,167 views
This document discusses testing Cassandra applications by making Cassandra fail deterministically. It introduces Stubbed Cassandra, a tool that allows priming Cassandra to respond to queries and prepared statements with different failures like timeouts, unavailability, and coordinator issues. Tests can verify application behavior and retries under failures. Stubbed Cassandra runs as a separate process and exposes REST APIs to prime failures and verify query activity, allowing integration into testing frameworks. It aims to help test edge cases and fault tolerance more effectively than existing Cassandra testing tools.
DataStax: Making Cassandra Fail (for effective testing)
- 1. Making Cassandra fail for effective testing 1
- 2. @chbatey Who am I? • Based in London • Technical Evangelist for Apache Cassandra •Work on Stubbed Cassandra •Help out Apache Cassandra users • Previous: Cassandra backed apps at BSkyB
- 3. @chbatey Agenda • Realistic production environment • How does Cassandra fail - What should you do • How to test your application
- 4. @chbatey Motivation • Continuous integration • Continuous deployment Commit code CI - tests Deploy to Test/UAT Deploy to realistic preprod Production •Slower tests •Exploratory tests
- 8. @chbatey Read timeout Application C R1 R2 R3 C=QUROUM Replication factor: 3 timeout timeout Read timeout
- 9. @chbatey Read timeout • Received acknowledgements • Required acknowledgements • Consistency level • Data received
- 10. @chbatey Write timeout Application C R1 R2 R3 C=QUROUM Replication factor: 3 timeout timeout Write timeout
- 11. @chbatey Retrying writes • Cassandra does not roll back!
- 12. @chbatey Write timeout • Received acknowledgements • Required acknowledgements • Consistency level • Batches are more complicated: - WriteType: SIMPLE, BATCH, UNLOGGED_BATCH, BATCH_LOG
- 13. @chbatey Batches • BATCH_LOG - Timed out waiting for batch log replicas - Retry if you like • BATCH - Written to batch log but timed out waiting for actual replica - Will eventually be committed
- 14. @chbatey Idempotent writes • All writes are idempotent with the following exceptions: - Counters - lists
- 15. @chbatey Unavailable Application C R1 R2 R3 C=QUROUM Replication factor: 3 Unavailable
- 16. @chbatey Unavailable • Alive Replicas • Required Replicas • Consistency
- 17. @chbatey Coordinator issue Application C R1 R2 R3 C=QUROUM Replication factor: 3 ??? Application level timeout
- 18. @chbatey Unit & Integration testing • Requirements: Quick to run, deterministic, not brittle!! • Integration tests against an embedded Cassandra? • cassandra-unit • Integration tests against an external Cassandra running on developer / CI machines • CCM • Regular install • Docker • Vagrant
- 20. @chbatey How do this if it was a HTTP service? • Wiremock - mocking HTTP services • Saboteur - adding network latency If you want to build a fault tolerant application you better test faults!
- 21. @chbatey The options • Disrupt traffic between driver and the cluster • Disrupt traffic between Cassandra nodes • Make Cassandra fail deterministically? - Modify C* code - Build a protocol level test double
- 24. @chbatey Stubbed Cassandra Application Acceptance test prime on admin port (REST) verification on admin port Admin endpoints Native protocol
- 26. @chbatey Features • Prime to respond to: - Queries - Prepared statements - Batches • Simulate all failures • Disconnect clients
- 27. @chbatey Activity Client • Query ‣ Query text ‣ Consistency • PrepreparedStatementExecution ‣ Prepared statement text ‣ Bound variables public List<Query> retrieveQueries(); public List<PreparedStatementExecution> retrievePreparedStatementExecutions() public List<Connection> retrieveConnections(); public void clearAllRecordedActivity()
- 28. @chbatey Priming Client • PrimingRequest ‣ Either a Query or PreparedStatement ‣ Query text or QueryPattern (regex) ‣ Consistency (default all) ‣ Result (success, read timeout, unavailable etc) ‣ Rows for successful response ‣ Column types for rows ‣ Variable types for prepared statements public void prime(PrimingRequest prime) public List<PrimingRequest> retrievePreparedPrimes() public List<PrimingRequest> retrieveQueryPrimes() public void clearAllPrimes() public void clearQueryPrimes() public void clearPreparedPrimes()
- 29. @chbatey Example time public interface PersonDao { void connect(); void disconnect(); List<Person> retrievePeople(); List<Person> retrievePeopleByName(String firstName); void storePerson(Person person); }
- 30. @chbatey Testing the connect method //given activityClient.clearConnections();
- 31. @chbatey Testing the connect method //given activityClient.clearConnections(); //when underTest.connect();
- 32. @chbatey Testing the connect method //given activityClient.clearConnections(); //when underTest.connect(); //then assertTrue("Expected at least one connection", activityClient.retrieveConnections().size() > 0);
- 33. @chbatey Testing retrieve public List<Person> retrieveNames() { ResultSet result; try { Statement statement = new SimpleStatement("select * from person"); statement.setConsistencyLevel(ConsistencyLevel.QUORUM); result = session.execute(statement); } catch (ReadTimeoutException e) { throw new UnableToRetrievePeopleException(); } List<Person> people = new ArrayList<>(); for (Row row : result) { people.add(new Person(row.getString("first_name"), row.getInt("age"))); } return people; }
- 34. @chbatey Test the query + consistency //given Query expectedQuery = Query.builder() .withQuery("select * from person") .withConsistency("QUORUM").build();
- 35. @chbatey Test the query + consistency //given Query expectedQuery = Query.builder() .withQuery("select * from person") .withConsistency("QUORUM").build(); //when underTest.retrievePeople();
- 36. @chbatey Test the query + consistency //given Query expectedQuery = Query.builder() .withQuery("select * from person") .withConsistency("QUORUM").build(); //when underTest.retrievePeople(); //then assertThat(activityClient.retrieveQueries(), containsQuery(expectedQuery)); Hamcrest matcher
- 37. @chbatey Testing behaviour Map<String, ?> row = ImmutableMap.of( "first_name", "Chris", "last_name", "Batey", "age", 29); primingClient.prime(PrimingRequest.queryBuilder() .withQuery("select * from person") .withColumnTypes(column("age", PrimitiveType.INT)) .withRows(row) .build()); Each Cassandra Row == Java map Tell Scassandra about your schema
- 38. @chbatey Testing behaviour Map<String, ?> row = ImmutableMap.of( "first_name", "Chris", "last_name", "Batey", "age", 29); primingClient.prime(PrimingRequest.queryBuilder() .withQuery("select * from person") .withColumnTypes(column("age", PrimitiveType.INT)) .withRows(row) .build()); //when List<Person> names = underTest.retrievePeople(); Each Cassandra Row == Java map Tell Scassandra about your schema
- 39. @chbatey Testing behaviour Map<String, ?> row = ImmutableMap.of( "first_name", "Chris", "last_name", "Batey", "age", 29); primingClient.prime(PrimingRequest.queryBuilder() .withQuery("select * from person") .withColumnTypes(column("age", PrimitiveType.INT)) .withRows(row) .build()); //when List<Person> names = underTest.retrievePeople(); //then assertEquals(1, names.size()); assertEquals(“Chris Batey", names.get(0).getName()); Each Cassandra Row == Java map Tell Scassandra about your schema
- 40. @chbatey Testing errors // given PrimingRequest primeReadRequestTimeout = PrimingRequest.queryBuilder() .withQuery("select * from person") .withResult(Result.read_request_timeout) .build(); primingClient.prime(primeReadRequestTimeout);
- 41. @chbatey Testing errors @Test(expected = UnableToRetrievePeopleException.class) public void testHandlingOfReadRequestTimeout() throws Exception { // given PrimingRequest primeReadRequestTimeout = PrimingRequest.queryBuilder() .withQuery("select * from person") .withResult(Result.read_request_timeout) .build(); primingClient.prime(primeReadRequestTimeout); //when underTest.retrievePeople(); //then } Expecting custom exception
- 42. @chbatey Testing retries @Override public RetryDecision onReadTimeout(Statement statement, ConsistencyLevel cl, int requiredResponses, int receivedResponses, boolean dataRetrieved, int nbRetry) { if (nbRetry < configuredRetries) { return RetryDecision.retry(cl); } else { return RetryDecision.rethrow(); } }
- 43. @chbatey Testing retries PrimingRequest readTimeoutPrime = PrimingRequest.queryBuilder() .withQuery("select * from person") .withResult(Result.read_request_timeout) .build(); primingClient.prime(readTimeoutPrime);
- 44. @chbatey Testing retries PrimingRequest readTimeoutPrime = PrimingRequest.queryBuilder() .withQuery("select * from person") .withResult(Result.read_request_timeout) .build(); primingClient.prime(readTimeoutPrime); try { underTest.retrievePeople(); } catch (UnableToRetrievePeopleException e) { }
- 45. @chbatey Testing retries PrimingRequest readTimeoutPrime = PrimingRequest.queryBuilder() .withQuery("select * from person") .withResult(Result.read_request_timeout) .build(); primingClient.prime(readTimeoutPrime); try { underTest.retrievePeople(); } catch (UnableToRetrievePeopleException e) { } assertEquals(CONFIGURED_RETRIES + 1, activityClient.retrieveQueries().size());
- 46. @chbatey Coordinator issue Application C R1 R2 R3 C=QUROUM Replication factor: 3 ???
- 47. @chbatey Testing slow connection // given PrimingRequest preparedStatementPrime = PrimingRequest.preparedStatementBuilder() .withQueryPattern("insert into person.*") .withVariableTypes(VARCHAR, INT, list(TIMESTAMP)) .withFixedDelay(1000) .build(); primingClient.prime(preparedStatementPrime); Delays the response by 1000ms
- 48. @chbatey Testing slow connection @Test(timeout = 500, expected = UnableToSavePersonException.class) public void testThatSlowQueriesTimeout() throws Exception { // given PrimingRequest preparedStatementPrime = PrimingRequest.preparedStatementBuilder() .withQueryPattern("insert into person.*") .withVariableTypes(VARCHAR, INT, list(TIMESTAMP)) .withFixedDelay(1000) .build(); primingClient.prime(preparedStatementPrime); underTest.connect(); underTest.storePerson(new Person("Christopher", 29, Collections.emptyList())); } Delays the response by 1000ms Expect a custom exception
- 49. @chbatey Separate process Application Acceptance test prime on admin port (REST) verification on admin port Admin endpoints Native protocol
- 50. @chbatey How do I get it? •It is on maven central •Or go to www.scassandra.org •Executable jar + REST API • https://github.com/scassandra/scassandra-server <dependency> <groupId>org.scassandra</groupId> <artifactId>java-client</artifactId> <version>0.9.1</version> <scope>test</scope> </dependency>
- 51. @chbatey Summary • Testing is good - I want more tools to help me • Existing tools for Cassandra are great at happy path scenarios • Stubbed Cassandra allows testing of edge cases
- 52. @chbatey The end - Questions? www.scassandra.org http://christopher-batey.blogspot.co.uk/ Ping me on twitter @chbatey