Being a Data Science Product Manager
2 likes177 views
This document provides an overview of being a data science product manager. It discusses the speaker's journey becoming a PM, introduces data science applications in e-commerce, outlines the typical journey of building an AI/ML product, and discusses PM responsibilities. It also covers when to use AI/ML, includes a mini case study on recommendation engines, and discusses challenges including the non-deterministic nature of data science and lack of explainability in models.
Being a Data Science Product Manager
- 1. Being a Data Science PM Ram Narayan Subudhi
- 2. Agenda ● My Journey as a PM ● Introduction to Data Science ● Applications of AI/ML in E-commerce ● Journey of an AI/ML product ● PM Responsibilities ● When to use AI/ML ● Mini Case Study ● Challenges & learnings ● Q&A
- 3. Who am I? Joined GlobalScholar (EdTech) Built e-learning applications 2011 2017 Started working on engagement constructs UGC, Wishlists & Collections Graduated as a CS Engineer Joined Amdocs, built Billing & CRM software 2009 Started my MBA from IIM Bangalore Majors in Finance 2013 Moved to the Marketplace Team (Seller side) Leading Selection Design 2019 Started my PM journey at Flipkart Led Catalog Platformization 2015 ● Engineer turned Product Manager ● Experience across domains - E-commerce, EdTech, Telecom ● Experience of building both consumer facing products as well as platform products ● Have worked extensively with Data Scientists to solve for key customer & business problems at scale
- 4. My DS Journey Identifying fraudulent activity (Building intelligence) Auto answering of customer queries (Creating customer delight) Content Quality & Ranking (Building intelligence) Demand - supply gap analysis (Solving for business needs) Feedback summarisation (Creating customer delight) Selection Benchmarking & Assessment (Solving for business needs) Auto moderation of content (Automation) Highlights -
- 5. Data Science is Everywhere today
- 6. ● Vast & complex ● Fast evolving ● Lot of misconceptions Common Queries ● How much of Data Science should a PM know? ● What are the artefacts that a PM produces while building a data science product? ● How much time does it take to build a Data Science product? ● What does your typical day look it? ● How does data science form a part of product strategy? Introduction to Data Science
- 7. Applications of AI/ML in E-commerce Stakeholders Needs04 ● Monetisation - Ads Platform, Brand insights ● Marketing spends optimisation ● Private Labels Design Fulfilment Journey03 ● Procurement - vendor selection, lot management ● Warehouse - inventory placement, w/h space optimisation, w/h automation (bots) ● Logistics - route planning, delivery SLA prediction & optimisation ● Reverse logistics - returned product grading ● Fraud prevention - theft management Seller Journey02 ● Onboarding - lead generation, document verification ● Listing - catalog enrichment, listing quality measurement ● Inventory mgmt - demand forecasting & planning ● Growth insights - listing ads, pricing recommendations, selection insights ● Seller support - chat assistant, ticket allocation & prioritisation Buyer Journey01 ● Discovery - home page optimisation, search, recommendations ● Decision making - catalog (size reco), UGC (moderation, feedback summarisation) ● Checkout - address verification, reseller identification, payments (COD eligibility) ● Post order - customer support (chat bot), fraud detection ● Core services - user profiling
- 8. Journey of an AI/ML Product Problem Definition Define the business problem Formulate hypotheses Translate it into one or more DS problem statements Identify the right metrics & establish clear success criteria (PM) Data Exploration Identify/create the underlying datasets Identify feature sets (domain knowledge comes in handy) Identify different user segments, corner cases, etc. (PM/DS) Modelling & Optimisation Modularise Explore various models/techniques Train the models & iterate Measure the model metrics, make tradeoffs Validate through business/ops (P - DS, S - PM) Scale up & Maintenance Enable logging & debuggability Setup alerts & dashboards for health metrics Perform periodic checks to identify the need to retrain the models (P - Engg, S - PM/DS) Deployment & Experimentation Integrate with the core Tech stack Implement a fallback flow & have feature flags Instrument the necessary data signals Perform an A/B experiment Design the UX (P - DS/Engg, S - PM)
- 9. Problem Definition - Mini Case Study Aspect Ratings ● Define the business problem & the product to be built ○ User Need - to research about the product & its features ○ Business Problem - enable faster & convenient decision making for the user ○ Target product - summarise customer feedback at an aspect level ● Translate it into DS problems ○ Identify relevant aspects automatically ○ Tag feedback to aspects ○ Grade feedback from positive to negative ○ Summarise feedback at product level ● Business metrics vs. DS metrics ○ Business metrics - engagement, conversion ○ DS metrics - coverage, accuracy ● Challenges ○ Linguistic complexities - incorrect grammar, use of hinglish, etc. ○ Optimising for category nuances
- 10. Applying your domain knowledge - | ● Identify relevant feature sets ○ Proxy data ○ Signals from the broader ecosystem ● Handling data anomalies ○ Filtering out noise ○ Handling data quality issues ● Ensuring data quality ○ Check the checker flows ● Critique the Models ○ Question assumptions ○ Play the devil’s advocate ○ Test out both happy & unhappy flows
- 11. Applying your domain knowledge - || ● Make the right trade-offs ○ Accuracy vs. interpretability ○ Precision vs. recall ● Identify commonalities across products ○ Modularise & re-use
- 12. 1. Filtering out profanity from user generated content - what do you prioritise? a. Precision b. Recall 2. Aiding law & court proceedings - what do you prioritise? a. Accuracy b. Explainability Examples - Tradeoffs
- 13. Capture user preferences ● Design your onboarding experience ● Ask explicitly ● Allow users to update their preferences ● Allow users to blacklist Create feedback loops ● Validate your output regularly ● Respond to new data User Experience Design - I
- 14. Collect data intelligently ● Make it playful ● Solve a customer need Communicate effectively ● Build trust with the user ● Tell the why part Interact naturally ● Make it look & sound humane ● Simulate interactions & review User Experience Design - II Google Duplex phone calls -
- 15. Examples - Whether to use AI/ML? 1. Shortlisting resumes for a job profile? a. Yes b. No 2. Taking decisions during a medical surgery? a. Yes b. No
- 16. When to use AI/ML? Guiding principles: ● Recurring needs which are too costly or time consuming to do manually (eg. content moderation) ● When rules are not enough - either they are too many or they are too complex to define objectively (eg. address intelligence) ● Scale of data is huge to analyse & predict (eg. recommendations) ● Underlying data keeps changing over time (eg. user preferences) How to get started: ● Manual -> Rules -> DS When not to use DS? ● Rules work reasonably well ● Mission critical systems with no scope for errors, where decisions are irreversible ● Explainability is crucial
- 17. Mini Case Study Recommendation Engines & Personalisation ● Why do we need AI/ML? ○ Too many choices ○ Too many users with varying preferences ● Dimensions of Personalisation ○ Level: No -> Cohort level -> User level -> User * context level ○ Aspects: language, genre, content format, etc. ● Data Inputs ○ Implicit signals - browse/watch history, completion rates ○ Explicit signals - selected interests during onboarding ● Context ○ User context vs. session context ● Techniques ○ Collaborative Filtering ○ Facet Similarity
- 18. ● DS is non-deterministic ○ Stakeholders want predictability ○ Predicting the likelihood & extent of success is almost impossible in the beginning ○ Start small, iterate and scale up ● Business decisions ○ Build vs. buy vs. license ○ TTM, strategic importance, skill availability ○ Time bounding the research & development process ● DS models lack explainability ○ Prioritise between accuracy & explainability ○ Communicate & create transparency ○ Leave scope for manual overrides ● User data is no more secure ○ Put the control in the hands of the user ○ Anonymise the data ○ Build organisational firewalls to restrict access ○ Communicate the benefits of using the data Challenges & Learnings
- 19. Examples - PM Decisions 1. You built a feature using AI/ML. The model has a pretty high accuracy of 95%. The feature when launched led to a drop in conversion. What you do? a. Launch the feature b. Kill the feature c. Reimagine the feature
- 20. Q&A Thank You