2018 10 igneous
Download as PPTX, PDF1 like282 views
- Data challenges are growing in terms of volume, variety, velocity and quality. There is no single solution and real-world solutions will be hybrid. - Metadata management is a huge challenge, even basic metadata is beyond most small organizations. Federated systems are needed to transform medicine. - The document discusses challenges with data management across various domains including life sciences, healthcare, genomics, machine learning, artificial intelligence, and personal data. It emphasizes the importance of data visibility, quality, and integration across siloed systems.
2018 10 igneous
- 1. Data Patterns Life Sciences / Healthcare Chris Dwan (chris@dwan.org) https://dwan.org
- 2. Take-home messages Data challenges are large and growing – Not just volume – Also variety, velocity, quality There is no one single perfect solution – Requirements are diverse – Real world solutions will be hybrid Metadata management is a huge challenge – Even the basics are beyond most small organizations – We need federated systems to transform medicine
- 4. Geek Cred: My First Petabyte, 2008 My first Petabyte: 2008
- 5. Geek Cred: My First Petabyte, 2008 My first Petabyte: 2008
- 7. The evolution of data transfer …
- 8. Genomic Data Production in ContextGenomic data production @ Broad
- 9. Genomic Data Production in ContextGenomic data production @ Broad I did research computing at Broad from 2014 - 2017
- 10. Geek Cred: My First Petabyte, 2008 My first Exabyte: 2014
- 11. Data: The new oil* Data Base: Structure, queries Data Warehouse: All the data in one place. Limited integration. Data Mart: Serve up warehoused data to users (Shiny counts) Big Data: Volume, Variety, Velocity Data Lake: Data warehouse, but designed for in-situ analytics Data Ocean: A data lake, for the cromulently embiggened! Data Commons: When the benefits of sharing data outweigh the competitive instinct to horde it Data Biosphere: A data commons, but for the cool kids An immature ‘tyrant flycatcher. Needs a data mart, because it doesn’t know R or Linux yet. Hype-o-meter Impact-o-meter
- 12. Primary Data Production Data are produced on instruments … Sequencer / Mass Spec / … Analysis Systems High Performance Storage … Transformed and distilled … … Delivered to downstream processes … Customer facing storage
- 13. Primary Data Production Data are produced on instruments … Sequencer / Mass Spec / … Analysis Systems High Performance Storage … Transformed and distilled … … Delivered to downstream processes … … And archived for various purposes (FDA, HIPAA, Intellectual property, …). Customer facing storage Durable, cost effective storage
- 14. Primary Data Production Data are produced on instruments … Sequencer / Mass Spec / … Analysis Systems High Performance Storage … Transformed and distilled … … Delivered to downstream processes … … And archived for various purposes (FDA, HIPAA, Intellectual property, …). Customer facing storage Durable, cost effective storage I recommend an ‘archive first’ approach,
- 15. EMR ELN Primary Data Production Data are produced on instruments … Sequencer / Mass Spec / … Analysis Systems High Performance Storage … Transformed and distilled … … Delivered to downstream processes … … And archived for various purposes (FDA, HIPAA, Intellectual property, …). Customer facing storage Durable, cost effective storage I recommend an ‘archive first’ approach, LIMS LIS Metadata management is still a massive challenge Lab_Sample_tracker.xls Filename_as_ metadata_for _eric_v2
- 16. Quality Matters
- 17. Quality Matters Ask a computational biologist / data scientist what fraction of their time is spent fighting data quality, formatting, and similar issues. Multiply that by an entire industry They deserve better.
- 18. Machine Learning (ML) Algorithms that optimize and tune based on large amounts of data These have been around for a very long time (KNN and Linear Regression are totally ML). Algorithm innovations (deep neural nets), plus ubiquitous big data, plus improvements in computing, storage, network, and software. Killer apps everywhere in image recognition, natural language processing, clustering, categorization Hype-o-meter Impact-o-meter A ‘swan pink yellow’ columbine flower. Identifying objects in images is machine work now.
- 19. Data for Analytics / ML / AI Analysis Systems High Performance Storage A large and growing set of data is curated… Commercial / outsource labs Public or licensed datasets In-house labs Curation … and mined for insights. Analyst
- 20. Data for analytics Analysis Systems High Performance Storage A large and growing set of data is curated… Commercial / outsource labs Public or licensed datasets In-house labs Curation … and mined for insights. insights take both short and long paths back into the system Analyst
- 21. Data for analytics Analysis Systems High Performance Storage A large and growing set of data is curated… Commercial / outsource labs Public or licensed datasets In-house labs Curation … and mined for insights. insights take both short and long paths back into the system Analyst Durable, cost effective storage • What does “backup” mean, exactly? • How do we capture provenance without massive duplication?
- 22. Artificial Intelligence (AI) Distinguished (for me) by autonomous behavior and clever-looking behavior in the face of unanticipated situations. No requirement that “intelligent” mean “like a human.” Machine learning algorithms are a great (but not the only) way to create AI systems. Beware “bread machine AI.” Hype-o-meter Impact-o-meter Getting there! My cat shows surprising intelligence despite having a brain the size of a walnut
- 23. Artificial Intelligence (AI) Distinguished (for me) by autonomous behavior and clever-looking behavior in the face of unanticipated situations. No requirement that intelligence be human style. Machine learning algorithms are a great (but not the only) way to build AI systems. Beware “bread machine AI.” Hype-o-meter Impact-o-meter Getting there! My cat shows surprising intelligence despite having a brain the size of a walnut
- 24. Incredible opportunities here, and rapidly developing data silos The Clinical Data Ecosystem There is an incredible wealth of data available to support both clinical care and research Patient Journals Consumer products Unfortunately, it is carved up and isolated Longitudinal Data from other providers … Electronic Medical Records Possibility of a self-normal (N of 1) over time Diagnostic Imaging Natural language processing has strong potentialClinical Notes Innovations in the basics of clinical observation Hospital Telemetry Pressure to avoid incidental findings prevent bias Primary Lab Data There are both good and bad reasons for this
- 25. Personal Data Impacts Behavior I use a commercial service that combines labwork with wearable data They provide insights and coaching I have, personally, found this transformational in how I approach my health.
- 26. Personal Data Impacts Behavior I use a commercial service that combines labwork with wearable data They provide insights and coaching I have, personally, found this transformational in how I approach my health.
- 27. Personal Data Impacts Behavior I use a commercial service that combines labwork with wearable data They provide insights and coaching I have, personally, found this transformational in how I approach my health.
- 28. Personal Data Impacts Behavior I use a commercial service that combines labwork with wearable data They provide insights and coaching I have, personally, found this transformational in how I approach my health.
- 29. Personal Data Impacts Behavior I use a commercial service that combines labwork with wearable data They provide insights and coaching I have, personally, found this transformational in how I approach my health.
- 30. Why are we here? • Improved health outcomes • Quality-adjusted life-years • Increased therapeutic effectiveness • Reduced barriers to access • Publications / Patents / Druggable leads • Accelerated innovation cycle • Reduced time to market • Speeds & Feeds • Improved performance on benchmarks • Lower cost per unit • Infrastructure agility Social Mission Scientific / Business Goals Technology / Infrastructure
- 31. Maslow’s Hierarchy of Needs Friendship, connectedness, belonging Confidence, achievement Creativity, Purpose Safety, physical and economic stability Air, food, shelter, sleep If you lack this You don’t get to engage here
- 32. Maslow’s Hierarchy of Needs Friendship, connectedness, belonging Confidence, achievement Creativity, Purpose Safety, physical and economic stability Air, food, shelter, sleep Wireless Internet, Fully charged battery If you lack this You don’t get to engage here
- 33. IT Hierarchy of Needs Productivity and Security, Applications, disaster preparedness Automation and compliance “Thought Partner” Files, formats, naming conventions, access controls Phones, Projectors, Internet, Email, Chat Power, Building Access, Laptops, Wifi, Identity If you lack this You don’t get to engage here
- 34. Data Visibility Saves Money Private Data Holdings Public Data Backups … Private copy of public data $$ !! Lack of data visibility leads to increased costs and engineering challenges. It is depressingly common to see multiple representations of the same data, all being archived together. BAM BCL FASTQ This is also a metadata challenge
- 35. Challenge Architecture: The data DMZ • An architecture to support data creation, delivery, and use • … for seamless collaboration between organizations … • … without sacrificing security, appropriate usage, or privacy … • … and that delivers on the potential of modern analytic capabilities.
- 36. Blockchain ”The clown car of our industry in 2018” • Distributed ledger: trustworthy data / records without a central authority. • Self executing contracts: Shared, trustworthy code to operate on that data. • Initial Coin Offerings: massively accelerated (and deregulated) way to set monetary value on a data ecosystem. Amazing possibilities in permission / consent management. When I make snarky comments on LinkedIn, people ask if they can invest. Hype-o-meter Impact-o-meter The angel weeps because there are some really compelling use cases for blockchain, but the hype is deafening.
- 37. Take-home messages Data challenges are large and growing – Not just volume – Also variety, velocity, quality There is no one single perfect solution – Requirements are diverse – Real world solutions will be hybrid Metadata management is a huge challenge – Even the basics are beyond most small organizations – We need federated systems in order to transform medicine.