Utilizing Human Data Validation For KPI Analysis And Machine Learning
- 1. Utilizing Human Data Validation for KPI Analysis and Machine Learning Dan Morris Radius Intelligence
- 2. Overview and Key Takeaways • Data science problems @ Radius • Human validation: costs and benefits • Sampling and experimentation for multiple consumers • Positive feedback cycles in production
- 3. Radius - B2B Predictive Marketing
- 4. Radius - Data Engineering MLlib
- 5. Why Human Validation? • Business firmographic data is a difficult data problem • Our sources face the same challenges that we do • Each source must be considered a “proposal” • Independent Human Validation is (the closest thing to) ground truth
- 6. Degrees of Human Validation • Prompted Validation • Research Validation • Research Curation Value Cost PV RV RC
- 7. Prompted Validation Example Assignment: Verify the Phone Number and Address of a Business
- 8. Research Validation Example Assignment: Determine if a Business Belongs to a Chain / Franchise Business Name: Bob’s Donuts Address: 123 Main St. Website: www.bobsdonuts.biz Industry: Limited Service Restaurants Is Chain: (Y / N / U) Chain Type: (Local / Regional / National)
- 9. Research Curation Example Assignment: Where is the Headquarters of this Company Located? Company Name: Bob’s Donuts Inc. Website: www.bobsdonuts.biz Has many locations: (Y / N / U) HQ Location: ??? Source of information: ???
- 10. Human Validation: Benefits • Ground Truth • Supervised ML • Internal Metrics • Competitive Analysis • Our customers are humans, too!
- 11. Human Validation: Costs • Money • Time • Us and Them
- 12. Cost: Money • Validated data costs more than aggregated data > Validation + Data Science Pure Validation
- 13. Cost: Time • Automated experimental framework • Shift bottleneck to validation teams • Parallelized validation improves turnaround time • Be mindful of differences in teams / validators • Decay / Obsolescence of validations
- 14. Cost: Us and Them Clearly communicate expectations and interpretations
- 15. Uses for Validated Data • KPI Analysis • ML Training Sets • Spot Hypothesis Validation Challenge: minimize number of validations while meeting all downstream needs
- 16. Multiple-Consumer Sampling Standalone vs. Chain Experiment 1 value per 1 location == Easy Sampling!
- 17. Multiple-Consumer Sampling Phone Accuracy Experiment (0, 1, 2, 3, …) values to 1 location == Difficult Sampling.
- 19. Positive Feedback Production Cycle
- 20. THANKS! email me: dan.morris@radius.com stalk me: @djsensei connect me: linkedin.com/in/danielepmorris work with me: radius.com/jobs