MongoTx: Transactions with Sharding and Queries
- 1. MongoTx: Transactions with Sharding and Queries IBM Research – Tokyo Hiroshi Horii
- 2. NOTICES AND DISCLAIMERS • No part of this document may be reproduced or transmitted in any form without written permission from IBM Corporation. Any data has been reviewed only for publication in MongoDB World 2016. Any data is subject to change without notice. This document could include technical inaccuracies or typographical errors. IBM may make improvements and/or changes in the code described herein at any time without notice. Any statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Any reference to an IBM Program Product in this document is not intended to state or imply that only that program product may be used. Any functionally equivalent program, that does not infringe IBM's intellectually property rights, may be used instead. • THE INFORMATION PROVIDED IN THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER OR IMPLIED. IBM LY DISCLAIMS ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NONINFRINGEMENT. IBM shall have no responsibility to update this information. IBM products are warranted, if at all, according to the terms and conditions of the agreements (e.g., IBM Customer Agreement, Statement of Limited Warranty, International Program License Agreement, etc.) under which they are provided. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. IBM makes no representations or warranties, ed or implied, regarding non-IBM products and services.
- 3. § DB-Engines ranks MongoDB is the 4th based on § Search engine results, Google trends, Q&A sites’ discussion, job offerings, LinkedIn, and Tweets § However, Gartner’s report ranks lower than DB-Engines for use cases caused by lack of ACID support § Traditional Transactions: 20th/20 § Lightweight Events and Observations: 17th/20 § Stack Overflow has 714 questions about transactions of MongoDB § Typical answers: MongoDB is not a good for transaction processing systems Introduction: Many people say, MongoDB is not for transactions Source: Critical Capabilities for Operational Database Management Systems, Gartner (December 2015) Gartner’s Score Gartner is a registered trademark of Gartner, Inc. or its affiliates The logo of Stack Overflow is a trademark of StackExchange, Inc.
- 4. Example: Transfer 1000 from A to B Pseudo Application Code //2. Update A’s balance //3. Select B’s balance //4. Update B’s balance //1. Select A’s balance accounts.replaceOne( a, { _id:’A’, bal: a.bal - 1000}); b = accounts.find({_id: ’B’}).next(); accounts.replaceOne(b, { _id:’B’, bal: b.bal + 1000}); a = accounts.find({_id: ’A’}).next();{ _id: ’A’ bal: 6000 } { _id: ’B’ bal: 4000 } { _id: ’A’ bal: 5000 } { _id: ’B’ bal: 4000 } { _id: ’A’ bal: 5000 } { _id: ’B’ bal: 4000 } { _id: ’A’ bal: 5000 } { _id: ’B’ bal: 5000 } Stored Documents
- 5. Example: Transfer 1000 from A to B § Without ACID support, application is difficult to keep consistency of database in failure cases Pseudo Application Code //2. Update A’s balance //3. Select B’s balance //4. Update B’s balance //1. Select A’s balance accounts.replaceOne( a, { _id:’A’, bal: a.bal - 1000}); b = accounts.find({_id: ’B’}).next(); accounts.replaceOne(b, { _id:’B’, bal: b.bal + 1000}); a = accounts.find({_id: ’A’}).next();{ _id: ’A’ bal: 6000 } { _id: ’B’ bal: 4000 } { _id: ’A’ bal: 5000 } { _id: ’B’ bal: 4000 } Stored Documents Not Processed caused by a Failure InconsistencyInconsistency
- 6. Existing solutions require change of storage engines and disable sharding or queries § TokuMxTM enables transactions on the basis of the sacrifice of sharding – Transactions with multiple statements are available only for single servers or replica sets, not for sharded cluster (according to the online manual) – To add transactions to an existing system, its storage engines must be migrated to TokuMx storage engine. On the other hand, applications can use the same semantics of queries of MongoDB. § A method of two phase commit, which written in the official site, enables transactions on the basis of the sacrifice of query flexibility – A new data model is used to store dirty data. Only single-document read and write are available though it is possible to enable sharding – Existing application must be modified for the data model and single-document access https://docs.mongodb.com/v3.0/tutorial/ perform-two-phase-commits/ https://www.percona.com/doc/percona-tokumx/transactions.html TokuMx is a trademark of Percona, LLC
- 7. MongoTx Architecture Manage transaction states TxMap User’s DB Collections Manage application documents with SD2V MongoDB Driver MongoTx Driver Application MongoDB Driver MongoTx Driver Application MongoDB Driver MongoTx Driver Application MongoDB Driver MongoTx Driver Application MongoDB Driver MongoTx Driver Application mongos mongos mongos mongosmongos mongos mongos mongosUse the official driver Use the same sharding configuration No interaction between clients Transactional interface
- 8. Example: Transfer 1000 from A to B Pseudo Application Code MongoClient client; MongoDatabase db; Database db = new LatestReadCommittedTxDB (client, db); MongoCollection accounts = db.getCollection("accounts"); Tx tx = ((TxDatabase)db).beginTransaction(); try { a = accounts.find({_id: ’A’}).next(); accounts.replaceOne(a, { _id:’A’, bal: a.bal - 1000}); b = accounts.find({_id: ’B’}).next(); accounts.replaceOne(b, { _id:’B’, bal: b.bal + 1000}); tx.commit(); } catch (Exception ex) { tx.rollback(); }
- 9. SD2V: Single Document Two Versions § Store two versions in one document § Safe version: version that must be committed § Unsafe version: version that may not be committed § MongoTx Drivers maintain these versions with metadata Safe Version Unsafe Version Metadata { _id: ’A’ bal: 6000, _unsafe: { _id: ’A’ bal: 5000, }, _xTxId: ’tx1’, _insert: false }
- 10. (2) (1) (5) Add an unsafe (5) (3) Add an unsafe (4) (3) MongoDB in Transfer Transaction Tx tx = txDb.beginTransaction(); a = accounts.find({_id: ’A’}).next(); accounts.replaceOne(a, { _id:’A’, bal: a.bal - 1000}); b = accounts.find({_id: ’B’}).next(); accounts.replaceOne(b, { _id:’B’, bal: b.bal + 1000}); tx.commit(); { _id: ’A’ bal: 6000, _unsafe: { _id: ’A’, bal: 5000} _xTxId: ’tx1’}} { _id: ’B’, bal: 4000, _unsafe: { _id: ’B’, bal: 5000} _xTxId: ’tx1’}} (1) Insert a tx state { _id: ’tx1’, state:’active’ } TxMap accounts
- 11. (7) (6) (7) Commit the unsafe versions MongoDB in Transfer Transaction Tx tx = txDb.beginTransaction(); a = accounts.find({_id: ’A’}).next(); accounts.replaceOne(a, { _id:’A’, bal: a.bal - 1000}); b = accounts.find({_id: ’B’}).next(); accounts.replaceOne(b, { _id:’B’, bal: b.bal + 1000}); tx.commit(); { _id: ’A’ bal: 5000, } { _id: ’B’, bal: 5000, } (6) Change the state { _id: ’tx1’, state:’committed’ } TxMap accounts
- 12. Three Phase Query 1. Query only unsafe versions – Repair safe versions with unsafe versions if they have been committed 2. Query only safe versions – Repair safe versions with unsafe versions if they have been committed 3. Query safe versions in 2nd result if one ore more safe versions were repaired {bal:{ ’$gt’: 4000 }} {_unsafe.bal:{ ’$gt’: 4000 }} {bal:{ ’$gt’: 4000 }} Application query: {bal:{ ’$gt’: 4000 }} Easy to generate a query to select only unsafe versions Compatible with existing MongoDB data model
- 13. GitHub: https://github.com/hhorii/mongo-tx All source codes are available in GitHub with Apache 2 license DayTrader a J2EE application has been ported to MongoTx GitHub is a trademark of Github, LLC
- 14. Discussions • Performance overheads – Begin and commit a transaction – Move unsafe to safe versions (for each write) – Query unsafe versions (for each application query) • Other supported functionalities – Aggregation functions – Non-transactional operations • Optimization (in plan) – Reduce overheads to begin and commit transactions – Reduce overheads to move unsafe versions • Snapshot isolation (in plan) – Port our research prototype to the GitHub