Interleaving, Evaluation to Self-learning Search @904Labs

Editor's Notes

• #4: In search one of the core problems is to determine the relevance of items for a given search query. Research in this field started in the 1950s and led to the introduction of the popular TF.IDF formula in the 70s. TF.IDF is simple word counting, yet it has been the most popular ranking function for many decades. A parameterized version of it, BM25, was invented in the 70s as well. It is important to note that state-of-the-art open source search technology like Lucene used TF.IDF until 2015, and then replaced it with BM25, almost 40 years after its introduction! No wonder that out-of-the-box ranking performance of systems like Apache Solr and Elasticsearch can be improved as next the paradigm..
• #5: But things are changing!
• #6: As early as 1992, researchers started thinking about learning the parameters of ranking functions from data. From the 2000s, research in this direction really took off, and between 2000 and 2010 most of the currently used learning to rank approaches were introduced. The idea behind learning to rank is rather simple: estimating relevance is too complex a task to solve with a naive model like BM25. Many more features play a role other than word counts. To capture all these relevance feature and their relative weights, we apply machine learning to learn this from usage data and/or relevance assessments. The paradigm shift from trying to model relevance to learning it from actual data led to the introduction of learning to rank plugins for Apache Solr and Elasticsearch, making these methods available to a large audience. The popularity of these plugins is apparent from the number of talks here at Haystack about learning to rank.
• #7: A quick reminder on what learning to rank is all about, although everyone should know by now… We give a machine training data (queries and clicked documents for example), features that describe the query and the documents, and an objective function. The machine learn a ranking model, which is put into production to generate rankings. Learning to rank is a batch process. The pipeline of collecting training data, extracting features, and training a model is repeated every couple of hours, days, or weeks, depending on the organization. This batch processing often requires quite an impressive data processing pipeline. The question is whether we really need this batch process...
• #8: Reinforcement learning from an ML perspective? We would not ask the question if the answer would not be “no”. So, what’s the next step? Rather than periodically retrain and produce a completely new model, we can apply reinforcement learning to update the existing model in real time based on feedback. There are numerous advantages, but the most important ones are that we move away from the need to retrain our model periodically. We do not have to have large data processing pipelines to feed training data to our learner every couple of hours to train a new model, but rather take real time feedback on the quality of the current model, interpret this feedback, and update the model slightly accordingly. Secondly, since the system is learning continuously, it is easy to introduce new features on the fly. Assigning it some initial weight, the system will quickly learn to adjust this weight according to the new feature’s importance. Thirdly, the system can quickly adapt to changes in user behavior instead of having to wait for the next iteration of batch training.
• #9: What does online learning look like on a high level? It starts out much like regular learning to rank. A model is learned from training data, features, and an objective function. The initial model is used to generate a first ranking, which is shown to the user issuing the query. This user then interacts with the ranking, providing (implicit) feedback on the quality of the ranking. This feedback is immediately taken into account by the system, updating the model slightly to match the feedback. The updated model becomes the active model, which is used to generate the next ranking, etc. This probably all sounds very nice, but how can we make such an online learning to rank, or self-learning search system? One of the methods that can be used to power such a system is interleaving. Different between theory vs. practice.
• #10: Interleaving started out as an evaluation method for comparing ranking algorithms. Currently, however, it is also used to power self-learning search engines. I’ll explain how interleaving is used for evaluation and how this can be translated into an online learning to rank setting. A practical offline example using mutiple google search sessions for same search terms: Screen 1: lillistrate Search Engine A in Red & B in Blue.
• #11: Let’s say we’re running a web search engine and we want to find out whether a new ranking algorithm works better than our current algorithm. We can run an interleaving experiment to find out. Our current algorithm is search session A, the new version is search session B. Every query that is issued on our site is fired to both versions of the search engine. In this example, the query “online learning to rank” is issued to both A and B, and both engines return a list of results.
• #12: The next step is to actually interleave the results from A and B into one final result list. For the interleaving of results we can use a variety of methods, but to simplify things, we simply assume that we pick the first result from A, followed by the first result from B, the second result from A, etc. The final interleaved result list is the ranking that is shown to the user who issued the query. To this user there is no difference between results from A or B, they all look identical.
• #13: The user interacts with the final ranking, and clicks a result. In the backend we know that this result came from A, and thus A is the winner for this mini-competition between the two engines.
• #14: So, to summarize, if we want to compare two search algorithms, A and B, we turn it into a competition. Both engines generate results for the same query, these results are interleaved into one final result list. This list is shown to the user, who clicks the results she wants. These clicks are mapped to the engine that produced the particular result, and in the end the search engine with most clicks is the winner.
• #15: Why would we want to use interleaving to compare two algorithms, and not use the more common A/B test? There’s nothing wrong with doing an A/B test, but an interleaving experiment is faster to run and is low-risk. Interleaving is faster than an A/B test because each user evaluates both engines at the same time, unlike with an A/B test, in which a user is assigned to only one version. At the same time, because each user is shown at least some results from the current search engine (A), the experiment has a low risk, especially compared to an A/B test in which a certain percentage of users get to see only results from the new engine (B), which might lead to a bad user experience. Now that we know how we can use interleaving to evaluate search algorithms, we can turn it into a method for self-learning search engines.
• #16: The trick to get an online learning to rank system using interleaving is to continuously run a competition between the current best model and a slight adaptation of that model, and to immediately update the current model when it is beaten by the adapted version. In other words, we have our current model A, and take a slight adaptation of that model to be version B. When a query comes in, we run an interleaving experiment with these two versions, just like before. When clicks come in for that query, we determine the winner right away (instead of waiting for more queries to come in). In case version B wins, we update the current model A in such a way that we move into the direction of B. This updated version of A then becomes the new current model (A), and we generate a new adaptation from that model to be the new B. When a new query comes in, the process repeats itself. As always, the devil is in the details, but these are too complex to discuss here. Examples of such details are how exactly to do the updating of the model A. There’s a lot of scientific literature on this available.
• #17: So, what does online learning to rank look like in practice? Unfortunately, we do not yet have a demo of what is happening. We do have a couple of screenshots from one of our customers and some general numbers to report. When our demo becomes available, we’ll send it around on social media etc.
• #18: On the left are the results from Apache Solr with some manual tuning for the query “case”. Results show mainly tool cases at the top. After adding (online) learning to rank on top of Apache Solr, results become like the right hand side: suitcases have moved to the top, with a slight preference for those on sale.
• #19: Another example comparing Apache Solr with manual tuning to (online) learning to rank, now for the query “kitchen”. Results from Solr show toy kitchens and some kitchen equipment at the top. After learning from what users really want, we see actual kitchens “naturally” moving to the top. Again, we observe a preference for kitchens that are on sale.
• #20: We have compared our online learning to rank system to manually tuned Apache Solr instances for three of our customers using A/B tests. In general, we observe increases in revenue of about 30% when using online learning to rank on top of an Apache Solr index. We observe similar improvements for conversion rates. Note that these numbers could be achieved using “batch” learning to rank as well, but without the before mentioned benefits of online learning.
• #21: Orange denotes customer’s network, purple 904Labs’. (Explain the procedure.) This type of architecture is great for shortcutting the integration process, however, it is prone to network latency.
• #22: 904Labs’ core ranking algorithms are based on scientific publications, which are available to everyone. The software, however, is not open source. 904Labs does encourage the use of open source by its clients. In fact, the implementation assumes that the client is running her own Apache Solr or Elasticsearch index for search, but would like to improve the ranking quality. 904Labs acts as a middleware between the web app and the existing search index, and uses this index for its online learning to rank algorithms. There are plenty of advantages for clients, including easy implementation, data remaining at the client, and the lack of a vendor lock-in. If a client removes 904Labs services, she’s back at the original setup including her original search index. In this way 904Labs is a UNIQUE Search-as-a-Service providers who provide open source search engines as core part of their service. When a client wants to move away from such a solution, she also loses all data stored in the search index, as that is part of the solution. It is an open-source supported vendor lock-in :-)
• #23: There are still a lot of open issues when it comes to online learning to rank. Feature engineering is one, although it also applies to batchL2R. Which features are available and which ones could actually add something? The benefit of an online learning system is that you can easily add new features to analyze their impact. Delayed feedback is a technical issue related to the order in which feedback comes back into the system. What to do when feedback for a particular query comes in after feedback for a more recent query has already led to an update to the model? Should we ignore the feedback or still take it into account? Feel free to volenteer and work with 904Labs on these open issues Efficiency is a potential problem when trying to achieve maximum effectiveness. Learning to rank assumes that there is some initial seed set of documents to rerank. Ideally, this set contains all relevant documents, but we can only select a limited seed set. How to optimize this efficiency-effectiveness trade-off? Finally, when we have learned a model, we would like to make use of it, or exploit it. But if we would only exploit, we can not learn anything anymore. For that, we need to explore as well. How do we balance between exploiting the current best model and exploring to allow for learning?
• #24: To summarize this talk, we observe that a paradigm shift in search is happening, moving from old-fashioned ranking functions to learning from user behavior data. Current state-of-the-art uses learning to rank, but online learning to rank is on its way and offers again certain advantages on top of ltr. As is the case with learning to rank, there are still quite a few open issues for online learning to rank, so keep an eye on the research community to come up with cool new methods!
• #25: If you want to know (much) more about these details about this technology, feel free to contact Manos or Wouter directly.
• #26: If you want to know (much) more about these details, you could for example check out this tutorial on interleaving.
• #27: Two pointers for online learning to rank.
• #28: There are three approaches to learning to rank. Pointwise tries to predict the relevance of each single document. Pairwise looks at two documents and tries to determine which one is more relevant. Listwise, finally, tries to optimize the full ranking in one go, using existing IR metrics like nDCG. Going from pointwise to listwise should lead to better effectiveness, but also to a decrease in efficiency; the usual efficiency-effectiveness trade-off.