Overhauling a database engine in 2 months
2 likes1,172 views
The document outlines the development of ArangoDB, a multi-model database, and its transition to include a write-ahead log (WAL) for improved transaction management and performance. It details the challenges faced in reengineering the database engine, the planned approach phases, and the extensive testing and methodologies implemented to ensure successful integration without degrading user experience. ArangoDB is open source, widely supported, and designed for high efficiency with capabilities for complex queries and schema-less data processing.
Overhauling a database engine in 2 months
- 1. Overhauling a database engine in 2 months Max Neunhöffer Move Fast and Break Things, 12 March 2015 www.arangodb.com
- 2. Max Neunhöffer I am a mathematician “Earlier life”: Research in Computer Algebra (Computational Group Theory) Always juggled with big data Now: working in database development, NoSQL, ArangoDB I like: research, hacking, teaching, tickling the highest performance out of computer systems. 1
- 3. ArangoDB GmbH triAGENS GmbH offers consulting services since 2004: software architecture project management software development business analysis a lot of experience with specialised database systems. have done NoSQL, before the term was coined at all 2011/2012, an idea emerged: to build the database one had wished to have all those years! development of ArangoDB as open source software since 2012 ArangoDB GmbH: spin-off to take care of ArangoDB (2014) 2
- 4. is a multi-model database (document store & graph database), is open source and free (Apache 2 license), offers convenient queries (via HTTP/REST and AQL), including joins between different collections, configurable consistency guarantees using transactions is memory efficient by shape detection, uses JavaScript throughout (Google’s V8 built into server), API extensible by JS code in the Foxx Microservice Framework, offers many drivers for a wide range of languages, is easy to use with web front end and good documentation, and enjoys good community as well as professional support. 3
- 5. Architecture DB Engine Transactions Sharding Cluster Infrastructure V8 JavaScript libev zliblibICU Unicode Lib: TCP server, HTTP, JSON, OS−dep., V8 helpers API CRUD high level API in JavaScript Foxx etcd (Go) Client Scheduler RequestsHTTP Dispatcher HTTP Responses 4
- 6. ArangoDB in numbers DB engine written in C++ embeds Google’s V8 (∼ 130 000 lines of code) mostly in memory, using memory mapped files processes JSON data, schema-less but “shapes” library: ∼ 128 000 lines (C++) DB engine: ∼ 210 000 lines (C++, including 12 000 for utilities) JavaScript layer: ∼ 1 232 000 lines of code ∼ 85 000 standard API implementation ∼ 592 000 Foxx apps (API extensions, web front end) ∼ 298 000 unit tests ∼ 327 000 node.js modules further unit tests: ∼ 10 000 C++ and ∼ 24 000 Ruby for HTTP plus documentation and drivers (in other repositories) 5
- 7. The Task It is March 2014, we have just released V2.0. V2.1 is scheduled for end of May, V2.2 is scheduled for July V2.1 is incremental, V2.2 is “Write-Ahead-Log” work for V2.2 started in March unfortunately, introducing a WAL is akin to open heart surgery, → essentially need to reengineer the database engine do not have the capacity to assign 10 developers to the job 6
- 8. The old setup New data: { name: "watch", price: 99 } Data files: Collection: products Collection: sales (append only) (append only) For transactions: Locks and commit markers on all collections necessary. 7
- 9. The new setup "Collector" (later) Collection: sales Collection: products (append only) (append only) Data files: New data: Write Ahead Log (WAL) (append only) { name: "watch", price: 99 } For transactions: Less locks and commit markers only in WAL. 8
- 10. Advantages of a WAL have a single history of events have a single place to note the commit of a transaction easy asynchronous replication efficient sync to disk uncommitted stuff does not hit the data files at all better support for transactions → much higher performance better crash recovery deterministic, well defined behaviour 9
- 11. Challenges need fundamental change in the storage engine the collector changes stuff that is potentially being read need to be careful not to create a bottleneck need to get locking right crashes hard to test if possible, users must not notice the change (except better performance) 10
- 12. Our testing setup We do continuous tests after every push to github and nightly. We have separate test suites for single server and cluster. Different types of test for good coverage: low-level C++ library unit tests (10000 LOC C++) JS tests (separately on server and JS shell, 290000 LOC) AQL query engine (230000 LOC of the above) HTTP interface (TCP and SSL, 24000 LOC Ruby) dump/restore and bulk import benchmarks (3000 LOC C++) user interface (phantomjs, comparing screen shots) run tests with valgrind check coverage 11
- 13. Test methodologies Tests can have different aims: Unit tests: ensure that individual components work according to specifications Integration tests: ensure that multiple components work together correctly Benchmark tests: ensure performance End to End tests: ensure that complex systems as a whole do their job User interface tests: ensure that the user interface works and behaves as documented/specified 12
- 14. Test characteristics needed for our task Characteristics of our task well-defined, deterministic behaviour if possible, no observable change in functionality changes relatively far down in the software stack but reach wide temporary breakage expected and accepted =⇒ For our task, we needed: unit tests, integration tests and benchmarks. Fortunately, we had all these in place! 13
- 15. Approach — preparation phase 1. design work: WAL, markers, collection, compaction, where lock what 2. implement infrastructure for WAL: mmapped files, append op, marker format 3. add “write to WAL” to write operations 4. test filling of WAL 14
- 16. Approach — breaking phase 4. remove old write operations 5. implement collector thread and adjust compactor thread 6. repair by adjusting read operations for WAL/data file 15
- 17. Approach — repairing phase 7. fix transaction management 8. implement startup with non-empty WAL: crash recovery 9. fix/simplify replication 10. fix dump/restore 11. fix cluster 12. tune performance (legends) 13. Hurray, tests work again! 14. release beta version 15. fix, fix, fix and tune 16. publish V2.2 16