EUNIS 2018 - Migration of a web service back-end from a relational to a document-oriented database
- 1. Migration of a web service back-end from a relational to a document-oriented database Sebastian Drenckberg, Marius Politze IT Center RWTH Aachen University
- 2. Migration of a web service back-end from a relational to a document-oriented database Sebastian Drenckberg, Marius Politze EUNIS 2018 | 05. - 08. June 2018 2 Outline • Motivation • From Relational to Document oriented • Validation of Migration • Generalization • Conclusion
- 3. Migration of a web service back-end from a relational to a document-oriented database Sebastian Drenckberg, Marius Politze EUNIS 2018 | 05. - 08. June 2018 3 Motivation: Infrastructure • ~20 Software developers for Process Supporting Software 4-5 Agile Teams Student Lifecycle, RWTHApp, eLearning, Quality Management, eScience • Database server used for several “small” software projects Agile development process regular changes Some are interactively used by 1000+ users • Previously single instance of MS SQL Server 2008 R2 No redundancy No scalability Nightly Backups Single Point of Failure • Goal: more flexibility, scalability and redundancy Consider new database systems / technologies Limit migration effort / costs
- 4. Migration of a web service back-end from a relational to a document-oriented database Sebastian Drenckberg, Marius Politze EUNIS 2018 | 05. - 08. June 2018 4 Motivation: Case Study for Migration • Audience Response System that is part of RWTHApp Targeting large audiences 500-1200 students Anonymous usage • “Chat-like” 1:n communication during lectures Teacher – Student Teacher – all Students • Multiple Message Types Images Polls • Available via RWTHApp HTML5 Web Application
- 5. Migration of a web service back-end from a relational to a document-oriented database Sebastian Drenckberg, Marius Politze EUNIS 2018 | 05. - 08. June 2018 5 SQL Server: Relations
- 6. Migration of a web service back-end from a relational to a document-oriented database Sebastian Drenckberg, Marius Politze EUNIS 2018 | 05. - 08. June 2018 6 Language Integrated Query (LINQ) • LINQ is a language extension to formulate queries on collection Classes like Lists • LINQ to SQL Code generator to access relational databases Relations can be accessed like Lists For example: is (roughly) translated to:
- 7. Migration of a web service back-end from a relational to a document-oriented database Sebastian Drenckberg, Marius Politze EUNIS 2018 | 05. - 08. June 2018 7 Popularity of Document Oriented Databases Source: https://db-engines.com/en/ranking
- 8. Migration of a web service back-end from a relational to a document-oriented database Sebastian Drenckberg, Marius Politze EUNIS 2018 | 05. - 08. June 2018 8 Document Oriented Databases / NoSQL • Popularity is recently rising from niche to common use • No predefined structure Holds Documents consisting of key-value-pairs Documents are organized in collections • Common Formats: JavaScript Object Notation (JSON) Extended Markup Language (XML) (or dialects) • Implementation specific query languages { "address": { "street": "Seffenter Weg", "no": 23, "zip": 52074, "city": "Aachen", "country": "Germany" } }
- 9. Migration of a web service back-end from a relational to a document-oriented database Sebastian Drenckberg, Marius Politze EUNIS 2018 | 05. - 08. June 2018 9 MongoDB • Published in 2009 • Connector library needed, available for many languages C# library: MongoDB.Driver Supports LINQ • Multiple Collections per database process DB process relatively lightweight (~300MB Disk, ~100MB Memory) Allows DB process on Application servers • Replication Master-Multi Slave Automatic Failover “Every server-VM is equal”
- 10. Migration of a web service back-end from a relational to a document-oriented database Sebastian Drenckberg, Marius Politze EUNIS 2018 | 05. - 08. June 2018 10 Estimation of Migration Effort • Before • After
- 11. Migration of a web service back-end from a relational to a document-oriented database Sebastian Drenckberg, Marius Politze EUNIS 2018 | 05. - 08. June 2018 11 Migration I: Replacing Code Generated By LINQ2SQL • Replace generated classes by own code DB connection Serializable Types for stored Information • Add explicitly typed methods for current application • Generic connection class is reused in future migrations
- 12. Migration of a web service back-end from a relational to a document-oriented database Sebastian Drenckberg, Marius Politze EUNIS 2018 | 05. - 08. June 2018 12 Generalization: 1:1 Relation A B 1 1 A B … b: int / gguid id: int / gguid … Association by reference A B … b: B … Association by embedding
- 13. Migration of a web service back-end from a relational to a document-oriented database Sebastian Drenckberg, Marius Politze EUNIS 2018 | 05. - 08. June 2018 13 Generalization: 1:n Relation A B 1 n A B 1 1 I 1 n A B … b: List<int> id: int … Association by reference List A B … b: List<B> … Association by embedding multiple documents
- 14. Migration of a web service back-end from a relational to a document-oriented database Sebastian Drenckberg, Marius Politze EUNIS 2018 | 05. - 08. June 2018 14 Generalization: n:m Relation A B n m A B 1 1 I n m Association by embedding? A B id: int b: List<int> id: int a: List<int> Association by reference List
- 15. Migration of a web service back-end from a relational to a document-oriented database Sebastian Drenckberg, Marius Politze EUNIS 2018 | 05. - 08. June 2018 15 Validation Using Coded Tests • Integration Level Tests API Level / Blackbox Compare actual and expected results • Independent for different use cases Init and Cleanup always create the initial setup Coded tests are executed on check-in • Tests remain untouched Results before and after can be compared Creates a check-list during migration • Additional Unit Tests should be considered
- 16. Migration of a web service back-end from a relational to a document-oriented database Sebastian Drenckberg, Marius Politze EUNIS 2018 | 05. - 08. June 2018 16 Conclusion • Migration successful Validation using automated integration tests Generalization guides future migrations • Production system running since Aug 2017 Clear documentation, steep learning curve Running without significant issues • Major Version Update was successful Updating one server after another 0 downtime • More optimizations for MongoDB “native” applications Server side aggregation pipelines Map-Reduce Sharding
- 17. Thank you for your attention Vielen Dank für Ihre Aufmerksamkeit
Editor's Notes
- #4: Die Arbeit ist im Kontext…entstanden Bestehende AW: Audience Response System: Direktfeedback in der RWTHApp
- #6: DirectFeedbackMessageTag: Tag für eine Nachricht DirectFeedbackTag: Tags, die im Channel möglich sind
- #7: SQL: SQLConnection, query erstellen, ausführen, reader erstellen, reader auslesen
- #9: NoSQL: Not only SQL
- #10: Master-Multi Slave erklären Failover: Wechseln des ansprechbaren Servers aus unvorhersehbaren Gründen
- #11: und
- #12: MongoDBConnector: stellt Verbindung zur DB her DASDataConnector: enthält Zugriffe auf Collections; Hilfsobjekte; stellt Hilfsmethoden zur Verfügung, die DB-Op. Kapselt DASData: beinhaltet ein Objekt von DASDataConnector;
- #13: Links ER, rechts „MongoDB“-Darstellung Einbetten des Dokumentes spart zusätzliches Dokument für NodeB
- #14: Verknüpfung ebenfalls durch neues Attribut
- #16: Blackbox: Testen auf Ergebnis und nicht intern korrekten Ablauf