Kyle Kingsbury Talks about the Jepsen Test: What VoltDB Learned About Data Accuracy and Consistency
2 likes573 views
Kyle Kingsbury discusses the Jepsen test and its implications for Voltdb, emphasizing the importance of data accuracy and consistency. Voltdb, a clustered SQL relational database, utilizes Jepsen to identify and fix bugs, prioritizing consistency or data loss as critical. The conversation highlights the value of transparency in addressing issues and the benefits of Jepsen's credibility and reproducible tests for improving their product.
Kyle Kingsbury Talks about the Jepsen Test: What VoltDB Learned About Data Accuracy and Consistency
- 1. Kyle Kingsbury Talks about the Jepsen Test: ! ! What VoltDB Learned About Data Accuracy and Consistency! or! presents…!
- 2. Presenters! John Hugg, VoltDB Inc. Founding Engineer at VoltDB Was involved with both adding bugs found by Jepsen and fixing them. Kyle Kingsbury, Jepsen.io (aphyr) Creator of Jepsen, Riemann, Tesser Likes to break databases. Broke VoltDB, in fact.
- 3. Why did VoltDB pick Jepsen? What is VoltDB and what’s it good for? Learn More So What?
- 5. voltdb.com/jepsen jepsen.io pages VoltDB’s Jepsen-specific content Learn more about VoltDB Hacker News comments
- 6. What is VoltDB? • Scale-out, clustered, SQL Relational database • Strong serializable transactions by default, even at high scale • An excellent processing engine with rich import/ export functionality
- 7. Use Cases • Anything where decisions based on logic are made in real time for incoming events: • Policy Enforcement • Fraud Detection • Ad Tech • Real-time personalization • Payments • Anything where math / calculations are done: • Billing and reporting on live data • State tracking
- 8. Example: Telco Mobile phone is dialed. Request sent to VoltDB to decide if it should be let through. Single transaction looks at state and decides if this call: is fraudulent? is permitted under plan? has prepaid balance to cover? State Blacklists Fraud Rules Billing Info Recent Activity for both Numbers Export to OLAP
- 9. Example: Micro Personalization User clicks link on a website. This generates a request to VoltDB. VoltDB transaction scans a table of rules and checks which apply to this event. Eventually the transaction decides what to show the user next. That decision is exported to HDFS Spark ML is used to look at historical data in HDFS and generate new rules. These rules are loaded into VoltDB every few hours. User sees personalized content
- 10. StateProcessing
- 11. Complex, Transactional Business Logic Scale-Out Performance Streaming Events SQL Relational State
- 12. Why Jepsen? • We are always hungry for tests! • Could build VoltDB-Jepsen harness ourselves, but… Wouldn’t be as good and wouldn’t have Kingsbury’s credibility. • Customers have asked about it. • Kingsbury has a built-in audience (marketing)
- 61. VoltDB’s Takeaway Serious question: What’s the worst that could happen?
- 62. VoltDB Takeaway • Our policy: Consistency or data loss bugs are blocking bugs to be prioritized above all else. • So if Jepsen finds bugs, we need to fix them ASAP. • The risk is that Jepsen finds bugs that we have to fix, which might impact our schedule. • But that’s dumb. If our product has bugs, not knowing about them doesn’t make them not there.
- 63. VoltDB Takeaway Marketing & Perception: • Passing Jepsen is good. People talking about VoltDB is good. Showing we care about this stuff is good. • Having bugs is bad, but discussing and fixing issues openly and seriously can be positive.
- 65. Reproducible! • 100% reproducible test: • Set up Jepsen from Github • Clone Jepen VoltDB driver from Github • Run! • Can’t do this with systems you don’t control. github.com/jepsen-io/voltdb!
- 66. I want to learn more! chat.voltdb.com forum.voltdb.com askanengineer @voltdb.com @johnhugg @voltdb @aphyr voltdb.com/jepsen