Better search engine testing for solr
0 likes1,791 views
The document discusses effective search engine testing for Solr, emphasizing the importance of both human and automated testing approaches. It outlines the benefits of automated tests for repeatability, performance, and regression testing while acknowledging the necessity of human judgment in handling ambiguities in relevancy. The presentation concludes by advocating for a hybrid testing strategy that combines both methodologies for optimal results.
Better search engine testing for solr
- 1. BETTER SEARCH ENGINE TESTING FOR SOLR CVILLE MEETUP | EPUGH@O19S.COM | @DEP4B 1
- 2. WHY AM I QUALIFIED TO BE UP HERE? • Principal of OpenSource Connections - Solr Search Consultancy • Member of Apache Software Foundation • Presenter at conferences (OSCON, ApacheCON, UberConf, ExpoQA, STPcon 2009/2011!) • Fascinated by the art of software development 2
- 3. 3. X Co So m on ing AUTHOR 3
- 4. FATHER 4
- 5. AGENDA Why Test Search? Two General Approaches Test Tools for Solr Wrap Up 5
- 6. REDUCE HUMAN ERROR 6
- 7. photo credit to Jay Grieves @ www.bigswingingdeveloper.com 7
- 8. TWO APPROACHES TO TEST Humans Automated Tests 8
- 9. HUMAN TESTING • Better deals with ambiguity 9
- 11. I KNOW IT WHEN I SEE IT! http://en.wikipedia.org/wiki/Les_Amants 11
- 12. AUTOMATED TESTING 12
- 13. SWEET SPOT • Repeat Testing for Relevancy Tuning • Performance Testing: hitting SLA? • Regression Testing (Typical use case for automated testing) 13
- 14. “ONE TRUE ANSWER” • Absolute Truth / Matrix / Grid / TREC / Relevancy Assertions • The correct answers for each search are known ahead of time • Humans judges often decide these correct answers, stored as Relevancy Assertions • Can be labor intensive to setup • A “Numerical Grade” is produced for comparision 14
- 15. SO WHAT IS A JUDGEMENT LIST? 15
- 16. 16
- 17. COMPARABLE RESULTS Original AjaxSolr WebSolr gata 4 40.0 80.0 80.0 HIF 1 alpha 90.0 100.0 30 HIF-1 alpha 100.0 100.0 30 HIF1 alpha 70.0 100.0 100.0 HIF1A 100.0 100.0 100.0 iNOS 60.0 60.0 60.0 IRE-1 alpha 30 90.0 90.0 17
- 18. WRAP UP 18
- 19. Search Stack User Interface Search Engine Informa3on 19
- 20. 20
- 21. 21
- 22. A HYBRID APPROACH? Humans Automated Tests 22
- 23. A HYBRID APPROACH? Tool Supported Testing Humans Automated Tests Judgements 23
- 24. TESTING IS A HUMAN ACTIVITY James Bach, satisfice.com 24
- 25. THANK YOU! • twitter: dep4b • speakerrate: http://www.speakerrate.com/epugh/ • email: epugh@opensourceconnections.com 25 25