Jairaj Sathyanarayana

The ability to accurately understand and serve customer search queries is critical to the success of e-commerce platforms. This need is amplified in food delivery platforms operating in India due to the wide variety of languages, cuisines and tastes. Our platform alone offers millions of items from hundreds of thousands of restaurants across India. Not only do Indian dish names have a tremendous amount of regional variety, international cuisine served in India is typically customized to Indian… Show more

The ability to accurately understand and serve customer search queries is critical to the success of e-commerce platforms. This need is amplified in food delivery platforms operating in India due to the wide variety of languages, cuisines and tastes. Our platform alone offers millions of items from hundreds of thousands of restaurants across India. Not only do Indian dish names have a tremendous amount of regional variety, international cuisine served in India is typically customized to Indian palate. This variety is reflected in search queries which can be combinations of dish names, dish categories, cuisine, preparation style, occasions, dietary preferences, just to name a few. Additionally, queries can also be generic and conversational in nature when the customer is trying to explore. Given these complexities, language models can play a significant role in intelligently embedding the query context to retrieve relevant items. In this paper, we start with an open source language model almpnet-base-v2 seeking a balance of quality and latency requirements for our use-case. We propose a two-stage fine-tuning approach. In the first stage we use unsupervised training on historical search and ordering data to adapt the model to the food domain. The second stage uses supervised training on a strategically curated dataset containing a diverse set of search queries and items relevant to those queries. We conducted offline evaluation on a curated test dataset of complex, conversation-style natural language queries in two different geographical areas, one with a sparse supply of restaurants and another with dense supply. Our model showed an improvement of 14.94% in MAP and 19.22% in Precision@1 in the dense area and an improvement of 7.04% in mean average precision and 10.27% in Precision@1 in the sparse area, versus our baseline Search. Show less

Paying for deliveries using cash after the delivery is made is a popular mode of payment for customers transacting online for the first time or those that prefer more control, especially in emerging economies like India. While the cash (or pay)-on-delivery (COD or POD) option is a win-win for customers and e-commerce platforms, it also opens up surface area for fraud and abuse. A common risk mitigation strategy is to impose a limit on the order value that can be paid using COD. In our… Show more

Paying for deliveries using cash after the delivery is made is a popular mode of payment for customers transacting online for the first time or those that prefer more control, especially in emerging economies like India. While the cash (or pay)-on-delivery (COD or POD) option is a win-win for customers and e-commerce platforms, it also opens up surface area for fraud and abuse. A common risk mitigation strategy is to impose a limit on the order value that can be paid using COD. In our experience and literature survey, these limits are typically blunt (a single limit for a city or zip code) and set by business teams using heuristics and primarily from a risk-management-backwards view. This one-size-fits-all approach means we leave money on the table on customer groups where the limits are too strict and lose money on groups where they are lax. We need to concurrently balance the risk-management and the customer-preference angles. Note that this is different from a typical credit-scoring approach due to at least two major reasons - 1) the information available in e-commerce, especially online food delivery, is much sparser, 2) the limit needs to be calculated dynamically in real-time depending on the transaction value, restaurant and marketplace constraints and network effects. To this end, we present a framework called DyCOD that maps this to a non-linear constrained optimisation problem. To the best of our knowledge there are no published results in this area and our work is the first. We solve this using both heuristic and model driven approaches and run large-scale A/B experiments. Our approaches delivered a 2.1% lift in margin per order vs. the baseline while not breaching any risk check-metrics, which is highly significant at our scale. Show less

Abstract: In this work we outline an approach to mine insights from calls between delivery partners (DP) and customers involved in hyperlocal food delivery in India. Incorrect addresses/ locations or other impediments prompt the DPs to call customers leading to suboptimal experiences like breaches in the promised arrival-time, cancellation, fraud, etc. We demonstrate an end-to-end system that utilizes a multi- modal approach where we combine data across speech, text and geospatial domains to… Show more

Abstract: In this work we outline an approach to mine insights from calls between delivery partners (DP) and customers involved in hyperlocal food delivery in India. Incorrect addresses/ locations or other impediments prompt the DPs to call customers leading to suboptimal experiences like breaches in the promised arrival-time, cancellation, fraud, etc. We demonstrate an end-to-end system that utilizes a multi- modal approach where we combine data across speech, text and geospatial domains to extract the intent behind these calls. To transcribe calls to text, we develop an Automatic Speech Recognition (ASR) engine that works in the Indian context where the calls are typically highly code-mixed (in our case Hindi and English) along with variations in dialects and pronunciations. Additionally in the hyperlocal delivery space, the calls are also corrupted by high levels of background noise due to the nature of the business. Starting with Wav2Vec2.0 as the base we carried out a series of data and model based experiments to progressively reduce the WER from 85.30% to 31.17%. The transcripts from the ASR engine are encoded into embeddings by adapting an IndicBERT based model. Features extracted from the geospatial markers of calls are concatenated with the embeddings and passed through an XGBoost classification head to classify calls into one of three intents. Through ablation studies we show incremental improvements attributable to signals from different modalities. The winning multi-modal model has a macro average precision of 68.33% which is a 29.3pp lift over the baseline not utilizing all the modalities. Show less

With the increase in online platforms, the surface area of malicious activities has increased manifold. Bad actors abuse policies and services like claims, coupons, payouts, etc. to gain material benefits. These fraudsters often work collusively (rings) and it is difficult to identify underlying relationships between them when analyzing individual actors. Fraud rings’ identification can be modeled as a community detection problem on graphs where nodes are the actors and the edges represent… Show more

With the increase in online platforms, the surface area of malicious activities has increased manifold. Bad actors abuse policies and services like claims, coupons, payouts, etc. to gain material benefits. These fraudsters often work collusively (rings) and it is difficult to identify underlying relationships between them when analyzing individual actors. Fraud rings’ identification can be modeled as a community detection problem on graphs where nodes are the actors and the edges represent common attributes between them. However the challenge lies in incorporating the attributes’ domain-informed importance and hierarchy in coming up with edge weights. Treating all edge types as equal (and binary) can be fairly naive; we show that using domain knowledge considerably outperforms other methods. For community detection itself, while the weight information is expected to be learned automatically in deep learning based methods like GNNs, it is explicitly provided in traditional methods. In this paper, we propose a scalable and extensible end-to-end framework based on domain-aware weighted community detection, to detect fraud rings. We first convert a multi- edge weighted graph into a homogenous weighted graph and perform domain aware edge-weight optimization to maximize modularity, using the Leiden community detection algorithm. We then use features of communities and nodes to classify both community and a node as fraud or not.
We show that our methods achieve up to 9.92% lift in F1-score on internal data, which is significant at our scale, and up to 4.81% F1-score lift on two open datasets (Amazon, Yelp) vs. an XGBoost based baseline. Show less

Bundling complementary dishes into easy-to-order food combos is vital to providing a seamless food ordering experience. Manually curating combos across several thousands of restaurants and millions of dishes is neither scalable nor can be personalized. We propose FoodNet, an attention-based deep learning architecture with a monotonically decreasing constraint of diversity, to recommend personalized two-item combos from across different restaurants. In a large-scale evaluation involving 200… Show more

Bundling complementary dishes into easy-to-order food combos is vital to providing a seamless food ordering experience. Manually curating combos across several thousands of restaurants and millions of dishes is neither scalable nor can be personalized. We propose FoodNet, an attention-based deep learning architecture with a monotonically decreasing constraint of diversity, to recommend personalized two-item combos from across different restaurants. In a large-scale evaluation involving 200 million candidate combos, we show that FoodNet outperforms the Transformer based model by 1.3%, the Siamese network based model by 13.6%, and the traditional Apriori baseline by 18.8% in terms of NDCG, which are significant improvements at our scale. We also present qualitative results to show the importance of attention and lattice layers in the proposed architecture.

We won the 'best paper award' in the Applied DS track for this paper. Show less

Detecting abusive and fraudulent claims is one of the key challenges in online food delivery. This is further aggravated by the fact that it is not practical to do reverse-logistics on food unlike in e-commerce. This makes the already-hard problem of harvesting labels for fraud even harder because we cannot confirm if the claim was legitimate by inspecting the item(s). Using manual effort to analyze transactions to generate labels is often expensive and time-consuming. On the other hand… Show more

Detecting abusive and fraudulent claims is one of the key challenges in online food delivery. This is further aggravated by the fact that it is not practical to do reverse-logistics on food unlike in e-commerce. This makes the already-hard problem of harvesting labels for fraud even harder because we cannot confirm if the claim was legitimate by inspecting the item(s). Using manual effort to analyze transactions to generate labels is often expensive and time-consuming. On the other hand, typically, there is a wealth of ‘noisy’ information about what constitutes fraud, in the form of customer service interactions, weak and hard rules derived from data analytics, business intuition and domain understanding.

In this paper, we present a novel end-to-end framework for detecting fraudulent transactions based on large-scale label generation using weak supervision. We directly use Stanford AI Lab’s (SAIL) Snorkel and tree based methods to do manual and automated discovery of labeling functions, to generate weak labels. We follow this up with an auto-encoder reconstruction-error based method to reduce label noise. The final step is a discriminator model which is an ensemble of an MLP and an LSTM. In addition to cross-sectional and longitudinal features around customer history, transactions, we also harvest customer embeddings from a Graph Convolution Network (GCN) on a customer-customer relationship graph, to capture collusive behavior. The final score is thresholded and used in decision making.

This solution is currently deployed for real-time serving and has yielded a 16% points’ improvement in recall at a given precision level. These results are against a baseline MLP model based on manually labeled data and are highly significant at our scale. Our approach can easily scale to additional fraud scenarios or to use-cases where ‘strong’ labels are hard to get but weak labels are prevalent. Show less

Efficient and effective search is a key driver of business in e-commerce. Functionally, most search systems consist of retrieval and ranking phases. While the use of methods like Learning to Rank (LTR) for (re)ranking has been studied widely, most retrieval systems in the industry are still predominantly based on variants of text matching. Because text matching cannot capture the semantic intent of the query, most out-of-vocabulary (OOV) queries are either not handled at all or poorly handled… Show more

Efficient and effective search is a key driver of business in e-commerce. Functionally, most search systems consist of retrieval and ranking phases. While the use of methods like Learning to Rank (LTR) for (re)ranking has been studied widely, most retrieval systems in the industry are still predominantly based on variants of text matching. Because text matching cannot capture the semantic intent of the query, most out-of-vocabulary (OOV) queries are either not handled at all or poorly handled by matching to similarly-spelled entities. For niche e-commerce like food delivery apps operating on phonetically spelled, non-Western dish names, this problem is even more acute. Pre-trained word embedding models are of limited help because the majority of dish names are words that occur rarely or not at all in most openly available vocabularies. In this work, we present experiments and efficient Siamese network based models to learn dish embeddings from scratch. Compared to current baselines, we demonstrate that these models lead to a 3-5% improvement in Mean Reciprocal Rank (MRR) and Recall@k. We also quantify, using a combination of in-house Food Taxonomy and the Davies-Bouldin (DB) index, that the new embeddings capture semantic information with an improvement of up to 20% over baseline. Show less

Online food ordering marketplaces are multi-stakeholder systems where recommendations impact the experience and growth of each participant in the system. A recommender system in this setting has to encapsulate the objectives and constraints of different stakeholders in order to find utility of an item for recommendation. Constrained-optimization based approaches to this problem typically involve complex formulations and have high computational complexity in production settings involving… Show more

Online food ordering marketplaces are multi-stakeholder systems where recommendations impact the experience and growth of each participant in the system. A recommender system in this setting has to encapsulate the objectives and constraints of different stakeholders in order to find utility of an item for recommendation. Constrained-optimization based approaches to this problem typically involve complex formulations and have high computational complexity in production settings involving millions of entities. Simplifications and relaxation techniques (for example, scalarization) help but introduce sub-optimality and can be time-consuming due to the amount of tuning needed. In this paper, we introduce a method involving multi-goal sampling followed by ranking for user-relevance (Sample-Rank), to nudge recommendations towards multi-objective (MO) goals of the marketplace. The proposed method's novelty is that it reduces the MO recommendation problem to sampling from a desired multi-goal distribution then using it to build a production-friendly learning-to-rank (LTR) model. In offline experiments we show that we are able to bias recommendations towards MO criteria with acceptable trade-offs in metrics like AUC and NDCG. We also show results from a large-scale online A/B experiment where this approach gave a statistically significant lift of 2.64% in average revenue per order (RPO) (objective #1) with no drop in conversion rate (CR) (objective #2) while holding the average last-mile traversed flat (objective #3), vs. the baseline ranking method. This method also significantly reduces time to model development and deployment in MO settings and allows for trivial extensions to more objectives and other types of LTR models. Show less