Strata 2014: Design Challenges for Real Predictive Platforms
2 likes1,350 views
The document discusses the design challenges of predictive platforms, emphasizing the need for flexibility, robustness, and user-friendliness in developing analytical models. Key insights draw from database lessons, advocating for decoupling implementation to enhance usability and adaptability to various data types. It addresses various hard problems related to data handling and the importance of exposing uncertainty in predictive systems for effective decision-making.
Strata 2014: Design Challenges for Real Predictive Platforms
- 1. Design Challenges for Real Predictive Platforms Max Gasner @gasnerpants
- 2. What I’ve Been Doing ! Navia Systems (2010-2011) - probabilistic programming ! Prior Knowledge (2011-2012) - the Veritable API (went live in July 2012) ! acquired by salesforce.com (2012–2014) - predictive intelligence team ! APIs and cloud services to expose nonparametric Bayes (!?) …. to a general audience (?!)
- 3. Predictive Platforms? ! Methods have advanced to support flexible use ! ! Market is getting there - lots of data, many frustrated business users ! ! Let’s not mistake specialist problems for the wider need ! ! For most business problems, it’s a cold start and the competition is no predictive solution
- 4. The Database Analogy “Just” deterministic storage, collation, query, sorting, aggregation Early database systems were purpose-built by consultants and tied tightly to implementation details RETRIEVE FORCE STATUS WITH RUNWAY LENGTH > 8000, GCD(DENVA)>GCD(DENVA,BEVENS) THEN LIST AFLD NAME, GCD, RUNWAY LENGTH (System 473L, 1966) Relational database revolution (largely) decoupled schema from storage, interface from implementation
- 5. The Database Analogy Sensible defaults, but highly configurable and extensible by experts Ingest from many sources; data is typed Flexible query; some queries will fail; sensible defaults, but highly configurable and extensible by experts Indexing Query Modeling “Prediction” Many clients; databases outlive initial applications
- 6. Database Lessons ! “Decouple” implementation so users can be productive at different levels of abstraction ! ! Extensive gains (more applications are possible) and intensive gains (applications are easier to develop and maintain) ! ! Quantity >> quality: more is much more than better
- 7. Database Lessons “Everyone” writes SQL SELECT * FROM Patients WHERE Icd9 LIKE “250” AND DischargeDate = 2/11/2014 ! It needs to be this easy to deploy and query models INFER WillReadmit FROM (SELECT * FROM Patients WHERE Icd9 LIKE “250” AND DischargeDate = 2/11/2014)
- 8. Desiderata for Real Platforms ! Be robust to the real world: data is messy, users are inexperienced, and problems are underspecified ! Be honest about limitations: fail gracefully, but always fail when to do otherwise would be misleading ! Be flexible to changes in the structure of data and the questions that matter ! Be simple to use and provide the basic building blocks for complex applications (but don’t try to solve language, vision, and dating)
- 9. Robust ! Far more data is usually available than is understood ! Every dataset has missing values ! Every value is noisy ! ! Systems shouldn’t fail in the presence of irrelevant or partially observed data ! Systems should be conservative in the face of uncertainty
- 10. Honest ! No system is adequate to every problem or dataset ! Some mistakes are expensive and some are cheap ! Black boxes are easy to use and hard to trust ! ! Systems should provide measures of uncertainty ! Systems should explain their reasoning (in the sense of EXPLAIN)
- 11. Flexible ! The world isn’t a real-valued matrix ! Modeling choices shouldn’t mean we fake our datatypes ! The world is nonstationary and every predictive problem is streaming ! ! Systems should handle heterogeneous data natively ! Systems should retrain (and validate) continuously
- 12. Simple ! Predictive systems need to be easy to engineer with ! And they need to be easy to engineer ! The business user and the modeler both have valid interests in a predictive system and both need to be able to use it ! ! Systems should be decoupled into pieces ! Systems should expose a small set of operations that can compose to form complex predictive systems
- 13. Case Study: BayesDB ! Built on flexible general model for denormalized flat data tables* ! Separates index backend(s) from query frontend ! Exposes query interface through SQLlike language, “BQL” ! Open-source project (looking for @vmansinghka hackers) *Exercise for the reader: what about relational, graph, free text, ! http://probcomp.csail.mit.edu/bayesdb and time series data?
- 14. Building Blocks ! ANALYZE Construct models (like table views) from the dataset (table) ! SIMULATE Generate new (unobserved) rows like those in the table ! INFER Fill in “missing” (or target) values for partially-observed rows ! ESTIMATE PAIRWISE DEPENDENCE PROBABILITIES (!!) Exposes the structure of the learned model
- 15. Separate Concerns ! ANALYZE abstracts “what is doing the analysis”, decouples model choice, inference strategy, validation from query ! Enables heterogeneous backends, on-the-fly model selection, incremental model updates, cost-based modeling ! Analyses of the same data might treat it differently for different purposes ! Challenge: training the predictive DBA?
- 16. Flexible Query ! SIMULATE exposes the joint distribution but no actual values (anonymization, synthetic data generation) ! INFER supports traditional single-valued prediction, but also joint prediction, conditioned on any combination of values ! Flexibility goes hand in hand with consistency: expect that the results will be consistent in distribution ! Challenge: exposing query to the interactive end user?
- 17. Structure Discovery ! ESTIMATE PAIRWISE DEPENDENCE PROBABILITIES (eppdepp?) exposes the structure of the model ! Moving to broader measures of dependence than correlation ! Structure is key for iterative, exploratory workflows ! Structure feeds into optimization of query and inference strategies ! Challenge: representing structural uncertainty?
- 18. Expose Uncertainty ! Values should come with error bars, and explanations of how they were derived ! Automated systems can use uncertainty to make costbenefit tradeoffs (do show this ad, but don’t let this patient be discharged without this test being reviewed) ! Uncertainty lets us to move beyond anecdotes ! Challenge: uncertainty is hard to understand and explain
- 19. Hard Problems: Getting Data In ! : Source vs. Dataset (vs. Transformation, vs. Multi-Dataset, ….) ! Can we add more semantics to data definitions and schemas, to lever our prior knowledge? ! Can we use cloud services/crowdsource to better transform and interpret inputs? ! We need to design the entire data collection and storage pipeline to better support analytic consumers of data
- 20. Hard Problems: Getting Results Out ! Not clear what the right default presentation is ! Much work to be done in exposing and explaining predicted values and uncertainty ! As predictive systems start to support UIs (beyond News Feed), we need to design new paradigms for interaction with imputed and uncertain values ! It’s hard to form mental models of reactive/adaptive systems
- 21. Hard Problems: PL/BQL? ! The holy grail of “custom data types” — columns with custom models written in probabilistic programming languages ! Think ICD9: we have a really strong prior (medicine + biology) but no way to express it, let alone do inference ! Put domain-specific modeling in the hands of “anyone” ! How many people have written some PL/SQL? How many people have written production database internals?
- 22. Predictive in the Ecosystem ! Today: many specialized views of data (extending the basic OLAP/OLTP distinction for a new era of bigger data and new demands) ! Tomorrow: predictive services as true services inter pares, with many clients of their own, deriving data from the same sources of truth as other services ! Lots of work to do flowing provenance, prior knowledge, and schemata through the entire pipeline
- 23. Ecosystem? ! Let a hundred flowers blossom, let a hundred general purpose predictive platforms contend ! Lots of uncertainty about the right (combination of) models to support the interface ! Lots of room to innovate on API and presentation ! Many problems in business very eager for credible solutions ! Database analogy: we are waiting for Codd