Predicting the relevance of search results for e-commerce systems
1 like994 views
The document discusses building an open-source model to measure the relevance of e-commerce search results and the accuracy of their algorithms. It details a data preprocessing methodology, including cleaning, integrating, transforming, and reducing data, and features extraction techniques using word match counting and TF-IDF for enhanced feature expressiveness. The study found that the Random Forest algorithm with TF-IDF features outperformed Support Vector Machine (SVM) methods in predicting search query relevance, highlighting the importance of effective preprocessing in data analysis.
Predicting the relevance of search results for e-commerce systems
- 1. Predicting the Relevance of Search Results for E-Commerce Systems Mohammed Zuhair Al-Taie Joel Pinho Lucas Siti Mariyam Shamsuddin International Workshop Big Data Analytics 2015 “Multi Strategy Learning Analytics for Big Data” Universiti Teknologi Malaysia, Kuala Lumpur, 17-18 Aug 2015 Study Published in International Journal of Advances in Soft Computing and its Applications (IJASCA)
- 2. Introduction ► Search engines (e.g. Google.com, Yahoo.com, and Bing.com) have become the dominant model of online search ► Large business are able to hire the necessary skills to build advanced search engines, while ♦ small online business still lack the ability to evaluate the results of their search engines losing the opportunity to compete ► The purpose of this paper is to build an open-source model that can: ♦ Measure the relevance of search results for online businesses ♦ Measure the accuracy of their underlined search algorithms
- 3. Data Pre-Processing Data preprocessing usually consists of four steps 1. Data cleaning: a.k.a. data cleansing or scrubbing which aims at removing noise, filling missing values and correcting inconsistencies in data ♦ Involves the identification and removal of outliers 2. Data integration: seeks to combine data from varied and different sources into a coherent data storage 3. Data transformation: aims at converting data to more appropriate form. ♦ The goal is to have more efficient data mining operations by making the data more understandable 4. Data reduction: aims at reducing the size of data while minimizing any possible loss of information
- 4. Text Retrieval Systems ► Text retrieval involves retrieving the documents that contain particular keywords for a given query ♦ Given a string T=t1 t2…tn, and a particular keyword pattern P=p1 p2…pm, the goal is to verify whether P is a substring of T ► Pattern matching can be applied in two ways: ♦ forward pattern matching, where the text and pattern are matched in the forward direction; and ♦ Backward pattern matching, where the matching process is done from right to left.
- 5. CrowdFlower Dataset ► CrowdFlower created its dataset with the help of its crowd ♦ The crowd rated a list of products from a handle of ecommerce websites on a scale from 1 to 4 ♦ “4” indicates the product completely satisfied the search query and “1” to indicate that the result did not match the query search ► 261 search terms were generated by CrowdFlower ► The dataset incorporated six attributes: id, query text, product title, product description, median relevance and relevance variance of the query. ♦ The train file contains 10,158 rows in total ♦ The test file contains another portion of data with the same attributes, except the attributes related to the query relevance (median relevance and relevance variance), which is the label attribute to be predicted
- 7. Data Cleaning and Transformation ► The CrownFlower dataset described in last section is just an amount of raw tuples related to text queries and product descriptions and has some noise ♦ Apply data preprocessing which included the removal of undesired content to improve data quality and make it ready for mining and analysis ► We also need to pre-process the data in order to extract some value and then transform in order to: ♦ Provide an input for some machine learning algorithms ► The first task to be accomplished is feature extraction ♦ We need to transform raw text search attributes in valuable features ♦ To provide an input for running in machine learning algorithms
- 8. Word Match Counting ► The dataset encompasses text attributes related to textual searches. To extract features: ♦ The first method to extract feature is counting how many words, in each text search, match the product title and product description ► In order to implement word match counting, we took in advantage facilities and expressiveness of the Python language, ♦ Use the widely employed built-in packages for data manipulation, including NumPy and SciPy
- 9. NumPy and SciPy ► NumPy is an extension to the Python programming language, adding support for large, multi- dimensional arrays and matrices, along with a large library of high- level mathematical functions to operate on these arrays (Wikipedia) ► SciPy (pronounced “Sigh Pie”) is an open source Python library used by scientists, analysts, and engineers doing scientific computing and technical computing (Wikipedia)
- 10. Word Match Counting (cont.) ► The steps of word match counting were as follows: 1. First: we extracted word tokens from text (i.e.: splitting text by spaces) 2. Second: we removed non-alphanumeric characters from text. ♦ Words with just one or two characters (mostly articles and prepositions) were also removed 3. Third: we transformed each list of words (from search text attributes) in an array ► NumPy methods were applied for intersecting the search word vector with product title and product description vectors, respectively.
- 11. Word Match counting with tf-idf ► At this point we simply counted how many words remained in each intersection, ♦ S={t1,t2,…,tn} is the search words (terms) vector, D={t1,t2,…,tn} the product description words vector and T={t1,t2,…,tn} the product title words vector, respectively: Feature1 = count (T S) Feature2 = count (D S) ► Since t 𝑓 − 𝑖𝑑𝑓 weight-based algorithms are broadly employed by e-commerce and publishing companies for text retrieval and searching ♦ We decided to make use of it for enhancing the expressiveness of the two features we acquired so far
- 12. TF-IDF ► Term Frequency-Inverse Document Frequency (TF-IDF) produces a weight for each term fi in each document dj. ► It is basically a combination of two earlier methods: Term Frequency (TF) and Inverse Document Frequency (IDF). 𝑡𝑓 − 𝑖𝑑𝑓 is defined as in Eq. 1: 𝑡𝑓 − 𝑖𝑑𝑓 𝑓𝑖, 𝑑𝑗 = 𝑡𝑓𝑖𝑗 ∗ 𝑖𝑑𝑓 (𝑓𝑖) ► In order to acquire word weights we used the TfidfVectorizer class available in scikit-learn Python package ♦ scikit-learn is a machine learning library for Python to serve different purposes such as classification, regression and clustering
- 13. Word Match counting with tf-idf (cont.) ► Before calculating weights: stop words are ignored, which means that irrelevant words for searching, such as articles, prepositions and pronouns, are ignored ► The features we extracted now are much more expressive, since they take into account weights of word terms, ♦ Higher weights are related to rare terms and lower weights are related to high frequent terms across all documents ► Feature values are now the sum of all intersection word weights ♦ Instead of merely using a string to represent words in vectors, now we use Python dictionary to represent words along with their weights, where words are the keys and weights are the values in dictionaries
- 14. Word Match counting with tf-idf (cont.) ► In this way, we have ♦ a search dictionary Sdict={t1w1, t2w2, …, tnwn}, ♦ a product description words dictionary Ddict={t1w1, t2w2, …, tnwn}, ♦ a product title words dictionary Tdict={t1w1,t2w2,…,tnwn} ♦ We may also represent, as vectors, the keys from dictonary: Skeys={tk1, tk2, …, tkn} ► Thus, features are calculated as follows: Feature1 = sum(values(Tkeys Skeys)) Feature2 = sum(values(Dkeys Skeys))
- 15. Data Analysis and Prediction ► The main goal with CrowdFlower data is to predict the relevance of search queries. ♦ The label attribute for this study will be the median relevance of the search ► We first tried to predict the values of median relevance from CrowdFlower test set rows employing the (SVM) Support Vector Machine ♦ Implementation from the scikit-learn ► We also tried CrowdFlower data with a more sophisticated algorithm: the Random Forest ♦ Random Forest is an ensemble learning method that is largely employed in machine learning ♦ Largely implemented in scikit-learn user community
- 16. scikit-learn • scikit-learn is an open source machine learning library for Python • It features various classification, regression and clustering algorithms including support vector machines, random forests, gradient boosting, k-means and DBSCAN, • It Is designed to interoperate with NumPy and SciPy.
- 17. Data Analysis and Prediction (cont.) ► after acquiring the training data, and storing it properly in data frame, learning in scikit-learn should be taken in three simple steps: 1. initializing the model, 2. fitting it to the training data, and 3. predicting new values ► We stored the train set provided by CrowdFlower, but using our extracted features, in a data frame and provide it as input for both algorithms. ► After initializing and fitting train data, we can now predict the label attribute (search terms median relevance)
- 18. Learning Models Benchmark ► We describe the resulted scores we obtained running the four combination methods we applied: ♦ SVM with word match counting based features, ♦ SVM with tf-idf based features, ♦ Random Forest with word match counting based features and, ♦ Random Forest with tf-idf based features
- 19. Learning Models Benchmark (cont.) ►The best score was obtained with Random Forest with tf-idf based features. ►We can also notice that Random Forest obtained better score than SVM and, ►conversely, that tf-idf obtained better results than word match counting based features. ►However, the impact of applying tf-idf on preprocessing was substantially higher than Random Forest over SVM Match Counting tf-idf SVM 0.51241 0.57654 Random Forest 0.53834 0.59211
- 20. Conclusion ► With our benchmark on CrowdFlower test set, we could attest that Random Forest is an efficient machine-learning algorithm. ► Random Forest shown better accuracy compared to a simple SVM implementation. This, in part, is due to: ♦ The ensemble nature of Random Forest, which allows multiple learning algorithms to be run ♦ Beside this fact, the nature of the dataset is also another reason for SVM disadvantage, since such algorithm is likely to provider poorer performances when the number of features is much greater than the number of samples
- 21. Conclusion (cont.) ► Employing tf-idf for preprocessing features and, Random Forest is a powerful and effective approach for predicting, and measuring, the relevance of text search in e-commerce scenarios. ► Such approach suits small e-commerce businesses emerged in big data necessities. ► The use of scikit-learn package, as well as other built-in Python packages (NumPy and SciPy), saves substantial development time
- 22. Conclusion (cont.) ► The preprocessing step is crucial for the whole data analysis because: 1. It consumes most of the time and implementation efforts needed on the whole analysis 2. Preprocessing may be more critical for precision than the machine-learning algorithm itself ♦ SVM combined with tf-idf have shown higher score than Radom Forest combined with word matches counting
- 23. Thank you