AI/ML Infra Meetup | Scaling Experimentation Platform in Digital Marketplaces: Architecture, Implementation & Lessons Learned
0 likes733 views
Poshmark is a leading fashion resale marketplace that relies on experimentation and data to enhance its product and user experience. The company aims to build a reliable experimentation platform to accelerate innovation and streamline processes, addressing challenges like inconsistent frameworks and resource allocation. Key components of the platform include centralized operations, feature flagging for improved testing capacity, and a standardized analysis framework to support data-driven decision making.
AI/ML Infra Meetup | Scaling Experimentation Platform in Digital Marketplaces: Architecture, Implementation & Lessons Learned
- 1. Scaling Experimentation @ Poshmark Koundinya Pidaparthi, VP Analytics Aug 2024
- 2. Poshmark is a leading fashion resale marketplace powered by a vibrant, highly engaged community of buyers and sellers and real-time social experiences. Designed to make online selling fun, more social and easier than ever, Poshmark empowers its sellers to turn their closet into a thriving business and share their style with the world.
- 3. POSHMARK NETWORK & SCALE Since its founding in 2011, Poshmark has grown its community to over 130 million users and generated over $10 billion in GMV shop from over 10k brands and 90 categories
- 4. EXPERIMENTATION IS CORE TO DELIVER QUALITY PRODUCT Dynamic marketplace, rely on experimentation and data to build tools, measure impact and regression on metrics. ● Continuous Weekly Releases ● Varied Use cases ○ Code Migrations ○ New Feature Launches ○ App Redesign ○ Campaign Management ● 100s of metrics monitored for regression ● 200+ Internal Users of Experimentation Platform
- 5. EXPERIMENTATION PLATFORM CHALLENGES ● Lack of consistent and reliable frameworks can slow down decision making. ● High exposure can lead to too much risk. ● Low experiment concurrency can slow down product velocity. ● Analytics resources are often tied up with tedious workflows and repetitive analysis. ● Experiment operations and coordination can be time consuming. ● Lack of shared learnings can lead to more cold starts.
- 6. Our goal is to build a reliable experimentation platform to accelerate product innovation, ship high quality product, streamline experiment processes and save time.
- 7. 3 MAIN COMPONENTS OF OUR EXPERIMENTATION PLATFORM Analysis and Decision Framework Standardization Robustness Generalized Use Cases 1 Experiment Operations Centralized Code Repo Metrics Management Analysis Sharing Experiment Management Tools 2 Feature Flagging Lowered exposure Expanded Capacity Assignment Algorithm 3
- 8. Analysis and Decision Framework ● Experiment Design Guidelines ○ Clarify Goals ○ Standardized Metrics Nomenclature ■ North Star ■ Feature Metrics ■ Guardrail Metrics ○ Sizing ○ Standardized Stats Engine ● Standardized Operating Rhythm and Decision Frameworks ○ Readout Reviews ○ Pre defined success criteria ~ Cut Experimentation Cycle time, achieved 3X Lift in Experimentation Volume!
- 9. Streamlined Experiment Operations - Analytics Architecture ● Lightspeed Metrics ○ Standardized Business Logic ○ Python Stats Package ○ Pre Processed ETLs ● Knowledge Repo for collaborative reporting ○ Templated Notebooks ○ Converts Notebooks into user friendly readouts ○ Centralized Learnings ● AB Console ○ On Demand Experiment Scheduling ○ Improved visibility to Tests on Platform ~ 20% to 30% Efficiency Gains for Analysts
- 10. Feature Flagging Enhancements Segments ~ 10x increase in capacity to run concurrent experiments ● Created more layers and segments ○ Increased Capacity to run more concurrent experiments ○ Allow for smaller exposure tests, mitigate risk ○ Support expanded Use cases, Holdouts, Multivariate Tests ● Built assignment algorithm ○ On demand assignment ○ More confidence in randomization
- 11. Lessons Learned ● Actively listened to user feedback and built MVP to address key friction points which led to immediate value and adoption of the platform. ○ Standardization and consistency were key to build confidence. ○ Reducing friction increases adoption ● Open data culture, knowledge sharing and healthy debate was critical in building a culture of experimentation. ○ Focus on learnings, every AB test is a success regardless of outcome. ○ Continuous training and accountability are key to sustained success. ● Be open to new use cases for experimentation. ○ Campaign management, infrastructure migrations, holdouts. ● Be strategic about investments. Continuous investments not always a good ROI due to competing priorities and limited resources.
- 12. THANK YOU!