Solr Migration at Scale: A LexisNexis Journey

Editor's Notes

• #2: I hope everyone is having a good time. A special thanks to Lucidworks for bringing us all together for Activate 2019.
• #3: Just incase you haven’t found all the ways to stay connected here is just a quick reminder.
• #4: If you’re interested I have a little about my background. But in general I have been working in technology, architecture, and product for over 25 years and still love every minute of it.
• #5: Today I will be explaining a little bit about LexisNexis and then jumping into the different aspects of search that we have to deal with.
• #6: The Rule of Law Our people and core business operations are helping to advance the rule of law around the world. We accomplish this by operating in over 130 countries around the globe.
• #7: Search is at the core of almost every product offerings. The content that underpins those searches come from 60,000 trusted legal, news, business, and public records sources and equates to approximately 81 billion documents. As we progress our migration journey we started with our News sources which consists of approximately 2 billion articles and growing very rapidly every year.
• #8: As we migrate the underlying engine we have an opportunity to unwind years of development to determine what features are needed to aide our customers in the future. Observability and logging are fundamentals are at the core of every development activity.
• #9: Count on deep legal expertise Shepard’s® editors and respected, practicing attorneys enrich Lexis Advance content with analysis and in-depth commentary to further enhance your understanding. Providing rapid access to this data is part of the key to providing relevant and on-point search results. Migrating to an Solr we expect to be able to leverage and expand upon our machine learning and artificial intelligence capabilities that are continue to emerge.
• #10: Such a simple statement “Conduct research your way” as many in the room will attest means extreme complexity under the covers. The team has successfully migrating the Boolean syntax and is now tackling the complex queries that our customers are using. To provide early problem detection these queries are tested with every build. Enter keywords, use a Boolean approach or choose advanced, form-based searching. Flexible, cutting-edge technology powers your online legal research.
• #11: At the end of the day our customers are expecting, actually demanding, more from search. Their search experiences continue to expand in area outside of the legal space from areas such as online sales, simple browser searches(Google), and even social media. But their expectation are that we continue to provide results are relevant and on point for the work they are doing. Even though the law may move slowly the legal industry is expecting faster performance, improved relevance, and demands to integrate their usage data to help them navigate the complex mazes that make up the legal systems around the world.
• #12: Scaling Search is more than just the ability to have a fast search engine. It requires the right placement of functionality in the architectural stack. To meet our overall objectives we focused on using data to determine what our customers behaviors were. Looking at the types of search, the complexity of those searches and performance metrics associated with there activities. One focal area is around the features and functionality that the LexisNexis customer base uses. In many areas we found that we could get close but as As part of our effort to experiment and move forward from our learnings we established focal areas. Such as Infrastructure and Content Ingestion and Core Search Functionality.
• #13: To meet the needs of our customers and the competitive nature of content we set forth to provide a better search experience for our customs. We established a vision and high level goals in which we were trying to achieve. To address these goals we established a set of overarching approaches to help guide us along the way.
• #14: Everyone who has participated in a migration effort knows of the Parity Challenge. LexisNexis is no different. What we have done is use data to drive decisions. By analyzing production data we were able to determine the query used by our customers and focus development appropriately.
• #15: The shear volume of data that is need to support our customers is challenging. Being able to index that data in a timely fashion has always been a challenge. As part of our efforts we experimented with a variety of technical options to determine their limitations and our implementation challenges. In the end we found that for bulk load we found that Spark would scaled to levels that we targeted which is approximately 2 billion documents a day.
• #17: The team continues to evolve the platform and develop new features for the consumers of the platform. Throughout the journey the team is focused not only on functional testing but performance, relevance, cost, and consumer experience. Because the re-platforming team cannot be experts in all of these areas we have integrated with partner teams to leverage their tools and expertise.
• #18: We are partnering with the platform consumers to ensure that they are able to meet the external customers needs. The team has partnered with the Search Test Framework team to integrate STF as part of the CI/CD pathway.
• #20: One of the major reason for migrating to Solr is to provide us with the ability to create new and innovative ways to help our customers but along the way we are looking at our own tools and practices to ensure that we have the best chance of meeting those innovation challenges of the future.
• #21:  One such area is the use of the Search Test Framework that LexisNexis developed to help evaluate algorithms for improving relevancy. The replatforming team has pushed beyond its original intent and is now working with them to integrate this as part of the development process. This provides real time feedback and detects problems with functionality changes that inadvertently effected relevance. Another innovative way that the team is approaching development and the use of data is through the managed stack. This allows a developer to standup a production like instance load the data and test sets required for their algorithm development and immediately see how their changes compare to the production system.
• #24: Thanks to all of our service and tool providers without you we would not be able to accomplish everything we set out to.