CommerceSearch: Moving from FAST to Solr on ATG
1 like1,569 views
Backcountry.com, an online retailer of high-end outdoor gear, migrated from a legacy search engine to OpenCommerceSearch integrated with Solr due to issues with index latency, hardware limitations, and high support efforts. The new system aims to enhance user experience, scale more efficiently, and support continuous delivery by utilizing a rule-based system to automate product ranking and improve merchandising tools. The document outlines the challenges faced during the migration process and how the new architecture addresses those issues.
![Search Module
§ Rule’s
context
• Target
Page:
Search,
Category
or
All
Pages
• Query:
shirts,
[the
north
face]
• MulU-‐site
aware
• Catalog
aware
• Category
aware
§ Rule’s
life
Ume
• Start
&
End
Date](https://image.slidesharecdn.com/movingfromfasttosolronatg-130610001456-phpapp02/85/CommerceSearch-Moving-from-FAST-to-Solr-on-ATG-12-320.jpg)
CommerceSearch: Moving from FAST to Solr on ATG
- 1. Moving from FAST to Solr on ATG Ricardo Merizalde Software Development Manager Backcountry.com
- 2. Who We Are § Online retailer of high-‐end outdoor gear § Founded in 1996 by former Olympic skier § 7 Ecommerce sites § Internet Retailer Top 100 “We use the gear we sell”
- 3. More Background § Legacy Perl/Postgres ecommerce plaEorm § Using FAST Impulse/ESP since 2008 § Implemented Oracle/ATG Commerce in 2012 § 1.5MM daily search engine hits during Q4 2012
- 4. Why change search engines? § Product End of Life § Index latency § Hardware footprint § License limitaUons § Support effort is high § Not a good fit for conUnuous delivery
- 5. Goals § Low index latency § High availability § Easy to scale § ConUnuous delivery § Improve user experience § Improve merchandising tools
- 7. What’s OpenCommerceSearch? § Oracle ATG/Solr IntegraUon § Search Module: Business Tools § Framework to ease development
- 8. Integration Overview Each environment has its own collections
- 9. Search Module § Manage search assets • Facets • Redirects • Synonym Lists & Synonyms • Boosts & Blocks • Ranking Rules § Rule engine uses Solr • Has its own index (very small) • Extra search call to get rules (fast & cacheable)
- 10. Search Module
- 11. Search Module
- 12. Search Module § Rule’s context • Target Page: Search, Category or All Pages • Query: shirts, [the north face] • MulU-‐site aware • Catalog aware • Category aware § Rule’s life Ume • Start & End Date
- 13. Search Module § Rule’s custom properUes
- 14. Search Module § Ranking rules • Boost/demote chunks of products instead of individual products • Less manual boosUng • Ideal state is lidle/no manual boosUng. Use mulUple data points to automate ranking (clicks, revenue, etc.) • Uses boost funcUons
- 15. Search Module
- 16. Search Module
- 17. Search Module
- 18. Search Module § Rule based categories • PromoUon landing pages • Keyword landing pages • Price range, discount off, gender, hand pick products, etc.
- 19. Search Module
- 20. Search Module § Asset MigraUon • Redirects: imported as exact matches • Synonyms: manual cleanup • Boosts & Blocks: not imported • Facets: not imported § Query Synonyms • FAST is a black box • Analysis Tool • Synonym-‐Expanding EDisMax Parser wriden by Nolan Lawson
- 21. Who We Are Index Latency
- 22. Index Latency § Feed limited to XML feed/exlt tool § Slow process with mulUple points of failure 1. Generate XML files 2. scp XML files to FAST servers 3. exlt cronjob loads feed into SQL schema 4. JDBC Connector checking for changes 5. Document Processing Pipeline 6. Index documents
- 23. Index Latency § Index documents using API calls § Different SLAs for change types (price, inventory, content, taxonomy) § Auto soj & hard commits § Products & Skus are not versioned
- 24. Index Latency
- 25. Who We Are ConUnuous Delivery
- 26. Continuous Delivery § Hard to do integraUon tesUng § Automated tests were flaky • Data dependencies • No applicaUons states § Leverage EmbeddedSolrServer • RAM index • Custom JUnit Test Runner for IntegraUon tests • Solr XML for data • TestNG & Selenium for Automated tests
- 27. Continuous Delivery § IntegraUon Test • Shared core with bootstrapped data • Clone core & index test data • Test code & server run within same process
- 28. Continuous Delivery § Automated Test • Cores live within applicaUon server • Test data sent over HTTP
- 30. Continuous Delivery § Rundeck jobs
- 31. User Experience
- 32. User Experience § Search results not consistent with product detail page § De-‐normalize data & index skus § Group by product id
- 33. User Experience
- 34. User Experience
- 35. User Experience § Orange jacket not available in XXL
- 36. User Experience
- 37. User Experience § Orange jacket available in XXL
- 38. User Experience § Grouping is awesome but is CPU intensive § Monitor query response Umes § Pay adenUon to parUal match search (use mm parameter) § Use the facets you really need