In-Memory Data Management for Systems Medicine
1 like1,366 views
The document presents an overview of in-memory data management for systems medicine, focusing on real-time analysis of healthcare data to enhance patient treatment and research efforts. It discusses the challenges of managing large, heterogeneous data sources and proposes a federated in-memory database system as a solution. The methodology emphasizes design thinking, real-time data analysis capabilities, and compliance with data privacy requirements.
In-Memory Data Management for Systems Medicine
- 1. In-Memory Data Management for Systems Medicine Dr. Matthieu-P. Schapranow e:Med Focus Workshop Data Management in Systems Medicine, Berlin June 10, 2016
- 2. Heart Failure Sleeping disorder Fibrosis Blood pressure Blood volume Gene ex- pression Hyper- trophyCalcium meta- bolism Energy meta- bolism Iron deficiency Vitamin-D deficiency Gender Epi- genetics ■ Integrated systems medicine based on real-time analysis of healthcare data ■ Initial funding period: Mar ‘15 – Feb ‘18 ■ Funded consortium partners: App Example: Systems Medicine Model of Heart Failure (SMART) Schapranow, e:Med Workshop, Jun 10, 2016 In-Memory Data Management for Systems Medicine 2
- 3. ■ Patients □ Individual anamnesis, family history, and background □ Require fast access to individualized therapy ■ Clinicians □ Identify root and extent of disease using laboratory tests □ Evaluate therapy alternatives, adapt existing therapy ■ Researchers □ Conduct laboratory work, e.g. analyze patient samples □ Create new research findings and come-up with treatment alternatives Actors in Systems Medicine Schapranow, e:Med Workshop, Jun 10, 2016 3 In-Memory Data Management for Systems Medicine
- 4. Schapranow, e:Med Workshop, Jun 10, 2016 In-Memory Data Management for Systems Medicine 4
- 5. Schapranow, e:Med Workshop, Jun 10, 2016 In-Memory Data Management for Systems Medicine 5
- 6. IT Challenges Distributed Heterogeneous Data Sources 6 Human genome/biological data 600GB per full genome 15PB+ in databases of leading institutes Prescription data 1.5B records from 10,000 doctors and 10M Patients (100 GB) Clinical trials Currently more than 30k recruiting on ClinicalTrials.gov Human proteome 160M data points (2.4GB) per sample >3TB raw proteome data in ProteomicsDB PubMed database >23M articles Hospital information systems Often more than 50GB Medical sensor data Scan of a single organ in 1s creates 10GB of raw dataCancer patient records >160k records at NCT In-Memory Data Management for Systems Medicine Schapranow, e:Med Workshop, Jun 10, 2016
- 7. Our Methodology Design Thinking Schapranow, e:Med Workshop, Jun 10, 2016 In-Memory Data Management for Systems Medicine 7
- 8. ■ Joint process definition ■ Identification of long running steps ■ Aims □ Improved communication □ Sharing of data □ Reproducible data processing Requirements Engineering for System Medicine Computer-aided Systems Medicine Process Schapranow, e:Med Workshop, Jun 10, 2016 In-Memory Data Management for Systems Medicine 8 20160407_eCardiohealth_Whole_Process HeartCenter Study Assessor Study Assessor Study Assessment Eligible Patient Available Radiologist Radiologist MRI MR Images Patient Meta Data, Hemo- dynamic Parameters, and Clinical Data Cardiologist Cardiologist Surgery Performed? Hemodyna- mic Evaluation Surgeon Surgeon Surgery ITplatform IT platform Update Notification SMART Data Storage Data processing WetLab WetLab Wet Lab Wet Lab Experiments Validation Wet Lab Results, e.g. Expression Data Message: Biopsy Sample Condition: 20 Biopsy Samples for batch processing Bioinformatici- an Bioinformatician RNA Sequencing FASTQ Files ProteomicsLab Proteome Analyzer Proteome Analyzer Protein Expressions Proteome Experiments Cardiomyocyte Modeler Cardiomyocyte Modeler Cardiomyocyte Modeling Cardiomyo- cyte Electro- mechanical Model Modeling Multi-scale modeller Multi-scale modeller Message: Post-surgery visit completed with data entry Multi-Scale Modeling Model output Hemodynamic Parameters Protein Expression Levels
- 9. Data Processing Pipelines From Model to Execution 1. Design time (researcher, process expert) □ Definition of parameterized process model □ Uses graphical editor and jobs from repository 2. Configuration time (researcher, lab assistant) □ Select model and specify parameters, e.g. aln opts □ Results in model instance stored in repository 3. Execution time (researcher) □ Select model instance □ Specify execution parameters, e.g. input files In-Memory Data Management for Systems Medicine Schapranow, e:Med Workshop, Jun 10, 2016 9
- 10. ■ Requirements □ Real-time data analysis □ Maintained software ■ Restrictions □ Data privacy □ Data locality □ Volume of “big medical data” ■ Solution? □ Federated In-Memory Database System vs. Cloud Computing Software Requirements in Systems Medicine Schapranow, e:Med Workshop, Jun 10, 2016 In-Memory Data Management for Systems Medicine 10
- 11. Where are all those Clouds go to? Schapranow, e:Med Workshop, Jun 10, 2016 In-Memory Data Management for Systems Medicine 11 Gartner's 2014 Hype Cycle for Emerging Technologies
- 12. Multiple Cloud Service Providers Schapranow, e:Med Workshop, Jun 10, 2016 In-Memory Data Management for Systems Medicine 12 Local System C loud Synchronization Service R Local Storage Local Synchronization Service R Shared C loud Storage Site A Local System R Local Storage Local Synchronization Service Site B C loud Synchronization Service Shared C loud Storage R Cloud Provider Site A C loud Provider Site B
- 13. A Single Service Provider Schapranow, e:Med Workshop, Jun 10, 2016 In-Memory Data Management for Systems Medicine 13 Cloud Synchronization Service Shared Cloud Storage Site A Site BCloud Provider Cloud System R R
- 14. Multiple Sites Forming the Federated In-Memory Database System Schapranow, e:Med Workshop, Jun 10, 2016 In-Memory Data Management for Systems Medicine 14 Federated In-M em ory D atabase System M aster Data and Shared Algorithm s Site A Site BCloud Provider Cloud IM D B Instance Local IM DB Instance Sensitive D ata, e.g. Patient Data R Local IM DB Instance Sensitive Data, e.g. Patient D ata R
- 15. Schapranow, e:Med Workshop, Jun 10, 2016 we.analyzegenomes.com Real-time Analysis of Big Medical Data 15 In-Memory Database Extensions for Life Sciences Data Exchange, App Store Access Control, Data Protection Fair Use Statistical Tools Real-time Analysis App-spanning User Profiles Combined and Linked Data Genome Data Cellular Pathways Genome Metadata Research Publications Pipeline and Analysis Models Drugs and Interactions In-Memory Data Management for Systems Medicine Drug Response Analysis Pathway Topology Analysis Medical Knowledge CockpitOncolyzer Clinical Trial Recruitment Cohort Analysis ... Indexed Sources
- 16. Combined column and row store Map/Reduce Single and multi-tenancy Lightweight compression Insert only for time travel Real-time replication Working on integers SQL interface on columns and rows Active/passive data store Minimal projections Group key Reduction of software layers Dynamic multi- threading Bulk load of data Object- relational mapping Text retrieval and extraction engine No aggregate tables Data partitioning Any attribute as index No disk On-the-fly extensibility Analytics on historical data Multi-core/ parallelization Our Technology In-Memory Database Technology + ++ + + P v +++ t SQL x x T disk 16 Schapranow, e:Med Workshop, Jun 10, 2016 In-Memory Data Management for Systems Medicine
- 17. ■ Traditional databases allow four data operations: □ INSERT, SELECT and □ DELETE, UPDATE ■ Insert-only requires only INSERT, SELECT to maintain a complete history (bookkeeping systems) ■ Insert-only enables time travelling, e.g. to □ Trace changes and reconstruct decisions □ Document complete history of changes, therapies, etc. □ Enable statistical observations Insert-Only / Append-Only Schapranow, e:Med Workshop, Jun 10, 2016 In-Memory Data Management for Systems Medicine 17 ++ + +
- 18. ■ Main memory access is the new bottleneck ■ Lightweight compression can reduce this bottleneck, i.e. □ Lossless □ Improved usage of data bus capacity □ Work directly on compressed data Lightweight Compression Schapranow, e:Med Workshop, Jun 10, 2016 In-Memory Data Management for Systems Medicine 18 Attribute Vector RecId ValueId 1 C18.0 2 C32.0 3 C00.9 4 C18.0 5 C20.0 6 C20.0 7 C50.9 8 C18.0 Inverted Index ValueId RecIdList 1 2 2 3 3 5,6 4 1,4,8 5 7 Data Dictionary ValueId Value 1 Larynx 2 Lip 3 Rectum 4 Colon 5 MamaTable ……… C18.0Colon646470 C50.9Mama167898 C20.0Rectum647912 C20.0Rectum215678 C18.0Colon998711 C00.9Lip123489 C32.0Larynx357982 C18.0Colon091487RecId 1 RecId 2 RecId 3 RecId 4 RecId 5 RecId 6 RecId 7 RecId 8 … • Typical compression factor of 10:1 for enterprise software • In financial applications up to 50:1
- 19. ■ Horizontal Partitioning □ Cut long tables into shorter segments □ E.g. to group samples with same relevance ■ Vertical Partitioning □ Split off columns to individual resources □ E.g. to separate personalized data from experiment data ■ Partitioning is the basis for □ Parallel execution of database queries □ Implementation of data aging and data retention management Data Partitioning Schapranow, e:Med Workshop, Jun 10, 2016 In-Memory Data Management for Systems Medicine 19
- 20. ■ Modern server systems consist of x CPUs, e.g. ■ Each CPU consists of y CPU cores, e.g. 12 ■ Consider each of the x*y CPU core as individual workers, e.g. 6x12 ■ Each worker can perform one task at the same time in parallel ■ Full table scan of database table w/ 1M entries results in 1/x*1/y search time when traversing in parallel □ Reduced response time □ No need for pre-aggregated totals and redundant data □ Improved usage of hardware □ Instant analysis of data Multi-core and Parallelization Schapranow, e:Med Workshop, Jun 10, 2016 In-Memory Data Management for Systems Medicine 20
- 21. ■ Online: Visit we.analyzegenomes.com for latest research results, slides, videos, tools, and publications ■ Offline: Read more about it, e.g. High-Performance In-Memory Genome Data Analysis: How In-Memory Database Technology Accelerates Personalized Medicine, In-Memory Data Management Research, Springer, ISBN: 978-3-319-03034-0, 2014 ■ In Person: Join us for the Symposium “Diagnostics in the Era of Big Data and Systems Medicine” Oct 5-6, 2016 in Potsdam Where to find additional information? Schapranow, e:Med Workshop, Jun 10, 2016 In-Memory Data Management for Systems Medicine 21
- 22. Keep in contact with us! Dr. Matthieu-P. Schapranow Program Manager E-Health & Life Sciences Hasso Plattner Institute August-Bebel-Str. 88 14482 Potsdam, Germany schapranow@hpi.de http://we.analyzegenomes.com/ Schapranow, e:Med Workshop, Jun 10, 2016 In-Memory Data Management for Systems Medicine 22