Model driven engineering for big data management systems
Download as PPTX, PDF2 likes382 views
The document discusses the challenges and solutions for managing big data using model-driven engineering (MDE), particularly through the Juniper project, which aims to address data heterogeneity across different technologies. It highlights MDE's role in automating processes such as code generation and model transformations to facilitate the development of real-time big data systems. Additionally, the document outlines the advantages and limitations of MDE, along with examples of its implementation with various database systems like MongoDB and PostgreSQL.
![[VIDEO]
www.modeliosoft.com 38](https://image.slidesharecdn.com/juniper-160530134714/85/Model-driven-engineering-for-big-data-management-systems-38-320.jpg)
Model driven engineering for big data management systems
- 1. www.modeliosoft.com Model driven engineering for big data management systems Marcos ALMEIDA marcos.almeida@softeam.fr Sarah DAHAB sarah.dahab@telecom-sudparis.eu Andrey SADOVYKH andrey.sadovykh@softeam.fr 1
- 3. 20 ME 2006 17,5 ME 2005 70 ME 2013 Paris Rennes Nantes Sophia SOFTEAM – a French IT services / Software vendor •SOFTEAM, a growing company 25 years’ experience 900 experts Regular growth • Specialist in OO technologies, new architectures, methodologies • Banking, Defense, Telecom, … www.modeliosoft.com 3 23 ME 2008
- 4. Modelio for Software and System Engineering • UML editor with 20 years’ history o CloudML o SysML o MARTE o Code generation o Documentation o Teamwork www.modeliosoft.com 4 • Available under open source at Modelio.org
- 6. It is all about models … Starting with UML www.modeliosoft.com 6 Requirements UML Use Cases Architecture UML Components and Classes Design Refined Classes or Domain Specific Language Implementation Code generation Java, C++, Frameworks
- 8. Typical example: Control system for a frigate • 800+ components • Developed by 100+ engineers • 1M+ LOC • MDD fosters Productivity and Quality with o Code generation o Components reuse o Tracing o Automation www.modeliosoft.com 8
- 9. Curious DSL example: Ruby on Rails Haml HTML %br{:clear => left’} <br clear=”left”/> %p.foo Hello <p class=”foo”>Hello</p> %p#foo Hello <p id=”foo”>Hello</p> .foo <div class=”foo”>...</div> #foo.bar <div id=”foo” class=”bar”>...</div> www.modeliosoft.com 9 Feature: User can manually add movie Scenario: Add a movie Given I am on the RottenPotatoes home page When I follow "Add new movie" Then I should be on the Create New Movie page When I fill in "Title" with "Men In Black" And I should see "Men In Black" Cucumber and Capybara HAML
- 10. What do we get from MDD? Pros • Design once, deploy everywhere! • Write your transformation once, transform anything! Cons • Transformations are hard to write… • How to make sure they are CORRECT? i.e. – Is there any data/semantic loss? www.modeliosoft.com 10
- 12. Volume, variety, velocity 1. @-mails sent every second : 2,9 million 2. Video uploaded to YouTube every minute: 25 hours 3. Data processed by Google every day: 24 petabytes 4. Tweets per day: 50 million 5. Products ordered on Amazon per second: 73 items www.modeliosoft.com 12
- 13. Only 0,5 % of data is analyzed • In 2012, 2 837EB generated - just 0,5% actually analyzed. That still amounts to 14EB (or 14.185 million terabytes) Source: IDC & EMC www.modeliosoft.com 13
- 14. The main problem is Heterogeneity! • Many different database management systems o Ex: • MySQL (www.mysql.com/), • Big Table (http://research.google.com/archive/bigtable.html) • SimpleDB (http://aws.amazon.com/simpledb/) • Memcached (http://memcached.org/) • … • Many underlying data representation paradigms o Ex: • Relational Databases • Key-value Stores • Object-oriented Databases • Big Tables • … www.modeliosoft.com 14
- 15. The basis of our solution is MDE… Why? • Separating the problem from the solution o In JUNIPER we model the solution • Fostering automation o Analysis o Code generation www.modeliosoft.com 15 Business Objects Transformation HDFS MySQL MongoDB Abstract Models Specific Models / code Transformation Transformation
- 16. Understanding the problem… Why is it so HARD? (1/2) • Target Technologies based on different paradigms • Example: www.modeliosoft.com 16 A B JPA @Entity public class A { @Basic public B getB(){ … } … } SQL create table A (…) create table B (…) create table A_B (…)
- 17. Understanding the problem… Why is it so HARD? (2/2) • Target structure is variable • Example: www.modeliosoft.com 17 A B ER NoSQL A BAB Here A and B are independent entities Here, for performance reasons, B is embedded in A A B
- 18. Illustration: comparative features of MongoDB and PostgreSQL www.modeliosoft.com 18
- 19. Our solution: a component based approach to NoSQL heterogeneity • Generic model transformation chain oIntegrated to other Juniper tools •Audit rules •Model to model transformations •Code generators • Database specific instantiations oApplication architecture modelling oData modelling oHardware architecture (deployment) modelling www.modeliosoft.com 19
- 21. The Juniper FP7 EU project Website: http://www.juniper-project.org/ Start Date: 2012-12-01 Duration: 36 months Total cost: 4 M€ www.modeliosoft.com 21
- 22. JUNIPER integrates Big Data technologies over MPI www.modeliosoft.com 22 DOCs StreamsDBs Data Processing Stage 1 Stage N Business Intelligence Analytical DBs Visualization dbdb DOCsDOCs Data Processing in JUNIPER S1 S3 S2 Analytical DBs mpi mpi mpi mpi FPGA-enabled nodes Hadoop HPC
- 23. Modelling in Juniper www.modeliosoft.com 23 Models High level Architecture (Nodes,Programs, Streams…) Real-time constraints Java Code Code Generation (+MPI initialization, communication, etc) Reverse Engineering Schedulability Analysis Tool Scheduling Advisor Measurements & Advice Deployment Scripts ConfigurationModel Export Code Generation
- 24. Mapping Programming Model, UML and MARTE www.modeliosoft.com 24 JUNIPER Program Channel Cloud Node Programming Model UML MARTE
- 25. Modelling the application and real-time constraints www.modeliosoft.com 25 Real-time constrains - response time - bandwidth Big Data flow JUNIPER Programs
- 26. Modelling the hardware infrastructure at a high level www.modeliosoft.com 26 Cloud Node CPU with 4 cores Hard drive
- 27. MPI code generation www.modeliosoft.com 27 Code Generation JUNIPER Application Model
- 28. Overview of the Juniper programming model concepts Next step: integrating data modelling to the programming model www.modeliosoft.com 28
- 29. Business data modelling in Juniper • Example www.modeliosoft.com 29 Uniquely identifying pieces of data Partitioning data In different nodes
- 30. Business data modelling in Juniper • Concepts www.modeliosoft.com 30
- 31. Approach taken for dealing with heterogeneity in JUNIPER 1. Define a generic template for Modelio modules to provide support for big data management systems 2. Instantiate the template for MongoDB and PostgreSQL www.modeliosoft.com 31
- 34. MongoDB Example (2/2) + DEMO Video Database schema configuration scripts Deployment scripts Configuration scripts www.modeliosoft.com 34
- 36. PostgreSQL Example master installation script standby installation script configuration files www.modeliosoft.com 36
- 40. In short… • Challenge: o Big data applications How should we handle heterogeneous data ?? • Juniper response: o Model driven solution for designing real-time big data systems o Component based solution to heterogeneity • General business objects + big data concepts modelling • Database specific concepts – Modelling – Model transformations – Code generation www.modeliosoft.com 40
- 41. … and Perspectives / Exploitation • Source code and documentation available on our website o http://forge.modelio.org/projects/juniper o http://forge.modelio.org/projects/mongodb-modeler o http://forge.modelio.org/projects/postgresql-modeler • Tutorial + Dissemination on our forum www.modeliosoft.com 41
- 42. Questions? Marcos Almeida SOFTEAM | ModelioSoft {name.surname}@softeam.fr SOFTEAM R&D Web Site: http://rd.softeam.com Modelio Web Site : http://www.modelio.org http://forge.modelio.org/projects/juniper JUNIPER Web Site : http://www.juniper-project.org www.modeliosoft.com 42 * *for your questions