Test driven relevancy
0 likes275 views
- Eric Pugh is the co-founder of OpenSource Connections, an Elasticsearch and Solr consultancy. - OpenSource Connections helps clients improve their search relevance through consultancy, training, and community initiatives like meetups and conferences. - Many websites have "broken" search relevance due to issues like poor collaboration, difficult testing processes, and slow iterations. OpenSource Connections aims to help clients address these issues through tools like their search dashboard Quepid. - Improving search relevance is important for better conversion rates, understanding customer intent, and enabling personalization. OpenSource Connections provides strategies and services to help tune clients' search using frameworks, tools like Quepid, and a focus on measurement
Test driven relevancy
- 1. www.opensourceconnections.com Test Driven Relevance opensourceconnections.com.com Eric Pugh - @dep4b - epugh@opensourceconnections.com Dublin TestHeads Meetup - Lockdown Series 1st December 2020
- 2. www.opensourceconnections.com Who am I? ● Co-founder of OpenSource Connections ● Apache Software Foundation member ● Committer on the Apache Solr project ● Steward of Quepid: A Query Relevance Workbench ● Creator of Chorus: An Open Source Search Stack for Ecommerce ● Member of The Search Network
- 3. www.opensourceconnections.com OpenSource Connections ● “Empowering The World’s Search Teams” ● Elasticsearch & Solr experts, committers ● Consultancy & training - “Think Like a Relevance Engineer”, LTR, NLP ● US & UK offices ● Clients include USPTO, NHS Wales, LexisNexis, Wikimedia Foundation, Snagajob, Under Armour ● Community initiatives include Meetups & conferences, Relevance Slack - join at www.opensourceconnections.com/slack www.haystackconf.com The Search Relevance Conference (Haystack LIVE! Meetups every few weeks)
- 4. www.opensourceconnections.com Your search relevance is probably broken!!
- 5. www.opensourceconnections.com What is Bad Relevance? Easier to show versus tell... Nike.com - shirts without stripes Tesco Ireland - Guinness Tesco Ireland - Organic Cat Food Aldi UK - Almond Milk
- 6. www.opensourceconnections.com How I break (test?) Relevance! ● Pick a site (Sainsbury, Gap…) ● Types of Test queries ○ “White sugar” - Descriptive Language ○ “Cafiteer” - Spelling Errors ○ “Coffee maker” vs. “cafetiere” - Jargon/Terminology ○ “Cheap” vs “Expensive” - Vague yet specific ○ “Tshirt” and “T Shirt” - Abbreviations ○ “Shirt without stripes” - Natural Language Style
- 7. www.opensourceconnections.com This is why your search is broken!
- 8. www.opensourceconnections.com A broken workflow! ● Our collaboration stinks ● Search testing is hard ● Iterations are slow Where you come in!
- 9. www.opensourceconnections.com Why is this important? ● Better search means better conversion rates ○ Your customers aren’t using the same language as you are ○ Zero results pages burn money ○ Great search helps retention ● Search knowledge informs strategy ○ What’s in your query logs? ○ Understanding intent ● Search is a gateway to personalisation ○ A ‘virtual aisle’ for each customer ○ ML, LTR, recommendations, chat bots... “UK retail ecommerce sales will account for 27.5% of total retail sales this year, and that proportion will approach one-third by 2024. The post-pandemic hit to physical retail will be felt far into the future.” Emarketer “Online sales are now expected to grow from initial predictions of 11% to 19% in 2020, reaching £78.9bn, this is up from £66.3bn in 2019….Amazon is expected to benefit the most from this surge in demand for online retail” Edge by Ascential
- 10. www.opensourceconnections.com How OSC tunes search
- 11. www.opensourceconnections.com 5 Steps to Better Relevance ● Measurement ● Experimentation ● Rapidity ● Collaboration ● Help
- 12. www.opensourceconnections.com LAB EVALUATION OFFLINE RELEVANCE AUTOMATED TESTING Search Quality: A Business-Friendly Perspective Peter Fries - Haystack 2018 Relevancy Framework
- 13. www.opensourceconnections.com LAB OPERATIONS INTEGRATION CLICKSTREAM A/B TESTING EVALUATION OFFLINE RELEVANCE ONLINE AUTOMATED TESTING Search Quality: A Business-Friendly Perspective Peter Fries - Haystack 2018 Relevancy Framework
- 15. www.opensourceconnections.com Demo time of Quepid!
- 16. www.opensourceconnections.com Quepid ● Search Relevance dashboard & testing ● Used by clients such as Advance Auto, O'Reilly Media, Historic England, Salesforce - and NICE! ● 1500+ registered users ● Free hosted version at www.quepid.com ● Open source at github.com/o19s/quepid ● iSpy - Prototype connectors to any website frontend (in development )
- 18. www.opensourceconnections.com State of play LAB OPERATIONS INTEGRATION CLICKSTREAM A/B TESTING EVALUATION OFFLINE RELEVANCE ONLINE AUTOMATED TESTING RRE AgentQ Quepid Supahands? Ugh.. Google Analytics? Nothing good
- 19. www.opensourceconnections.com We need more QA folks in the Relevancy World! ● Test your site with ww.opensourceconnections.com/guide/ecommerce ● Meet Pete the Product Owner - blog & video series on the OSC website ● https://github.com/o19s/quepid for code ● www.quepid.com for documentation ● Free online workshops running at https://plainschwarz.com/ps-salon/ ● Join Relevance Slack at www.opensourceconnections.com/slack ● Let OSC help you build better ecommerce search!
- 20. www.opensourceconnections.com Thank you. Contact me at epugh@opensourceconnections.com and follow @dep4b