MetaConfig driven FeatureStore : MakeMyTrip | Presented at Data Con LA 2019 by Piyush Kumar
1 like1,028 views
The document outlines the development of a unified feature store at MakeMyTrip aimed at improving machine learning workflows and enhancing customer personalization for Indian travelers. It discusses the current challenges of siloed datasets, the need for standardization, and the benefits of a centralized feature repository for managing curated features. The presentation also covers the architecture, data flow, and future steps for implementing this feature store to optimize machine learning lifecycle and reduce technical debt.
![FSConfig : Feature Definitions & Metadata
Feature Name :
<Entity>::<Feature_shortname>::<
Data Time Interval>::<Refresh
Frequency>::<Version>
Entity : <UserID>_<profileType> Short Name :
listing_conversion_rank
Versioning : v2 + Process :
RT/BT
FeatureGroup : (System
Generated ID)
8fda73d1_2eee_4cfc_a20f_e9afb1
78fbc3
Entity:
["uuid", "profile_type"]
Features [Array] Time Window(Refresh/
Data - Time duration): (ISO
Time Interval) P1D
Data Source [Array]:
[user_master, txn_search]
Data Store: GLUE/S3 Database Name: blueshift Table Name: [user_master,
txn_search]
Data Sink: Serving [Array] Data Store: GLUE
Catalog/S3/Redis/BoulderDb
Database Name :
rocksDB_<WAL Dir Path>
Table Name :
rocksDB_<columnFamily>
Compute Logic DSL + Spark SQL: metric_expr,
group_by_expr, filter_expr,
window_function,
window_function_alias
Code (Python/Scala/Java)
: GIT/Gerrit URI
Model(sagemaker) /
Embedding
Environment: Production Workspace: Dev/Staging/Production Namespace: <Project
Name>
Apache LIVY + Databricks
JOBs API Config](https://image.slidesharecdn.com/featurestoremakemytrip-190817221948/85/MetaConfig-driven-FeatureStore-MakeMyTrip-Presented-at-Data-Con-LA-2019-by-Piyush-Kumar-11-320.jpg)
MetaConfig driven FeatureStore : MakeMyTrip | Presented at Data Con LA 2019 by Piyush Kumar
- 1. MetaConfig driven FeatureStore@MakeMyTrip ~/Piyush Head Data Platform Engineering Namasté
- 3. Deliverables of this presentation: - Why common feature store? - Productionizating ML via standardization - Machine Learning Life Cycle - Prediction Serving + Challenges - FeatureStore Components - Architecture - Tools - Next Steps - References
- 4. Motivation Developing Unified Personalization platform for improving customer experience of millions of Indian travellers Business Goal: Through Hyper Personalization ● Raise Engagement ● Drive Conversions + Boost Revenue ● Migrating Business Rule Engines to ML Models ( across different LOBs @MakeMyTrip) Tech Goal: ● Machine Learning Models are as good as the data they are trained on. Needs good Data Management. ● ML Systems are trained on set of features, a feature is a input to model which can be a column in a dataset or complex computed metric or some other model output too ● Feature Store is a central common repository for highly curated features which are described through well structured configuration. Enables us to scale machine learning workflows @MakeMyTrip.
- 5. Before Feature Store : state of data platform ● Siloed Data Sets + Serving APIs created per use-case / projects leading to complex data pipelines | Machine Learning if not implemented in right manner creates high tech debt ○ Personalization : Cosmos ○ Customer Segmentation : HYDRA ○ Hotel Ranking / Sequencing + Intendo ○ DP : Dynamic / Differential Pricing : Hotel & Flights ○ Anomaly Detection, Destination trends, Demand Anomalies ● RealTime Features require Data Engineering support from Data Scientists ● Lack of standardization & discovery : Feature definitions are duplicated into the different data pipelines even if it is same / computed multiple times and change to definitions means fixing across different pipelines. ● Features used in training and serving were inconsistent
- 6. Productionizing ML via Standardization ● MetaConfigs & Feature Catalog : Documentation ● Reusability of features across projects / teams ● Standardized access of features between Training & Serving | Data Governance + Data Quality ● More Self-serve : Reduces Data Scientist Time on DE Tasks ● Reduce Time to get to Production for ML Projects ● Reduce Data Tech-Debt & Improved Feature Quality Feature Store : Online + Historical Data Store 1 Data Store 2 Data Store N Raw Data Data Sets 1 Data Sets N Structured Data Feature Engineering MODEL : TRAINING + DEPLOY
- 7. Machine Learning Life Cycle ML LifeCycle Image source : UCB RISE LABs Addition : FEATURE PIPELINES
- 8. Prediction serving - ASK : 10 -30 ms / < 30 ms - Challenges : DNN : Complex models - Hardware : GPUs / TPUs - SageMaker provides abstraction / middle layer between applications and complex models thru docker containers - Online : SageMaker Endpoints - Batch : Scoring : Pre-materialize predictions into a low latency store ( like redis cluster / BoulderDB) - Problems : - Requires substantial computation and space - Example doing the scoring for all customers - Costly update -> rescore everything
- 9. FeatureStore Glossary Feature : a measurable property of a phenomenon under observation defined in FSConfig FSConfig: used for storing config/ DSL + code to compute features, feature version information, feature analysis data and feature documentation FSCompute: Computation Engine developed over SPARK, supports mosts of the spark APIs for historical and Online(Streaming) FeatureStore : serves as a repository of features that can be used for training and evaluation of machine learning models. FeatureGroup: internal to the system, to group common compute jobs of related features having the same entity, input data sources and filter conditions, thereby optimizing the compute process. FSScheduler: Internal service to create a feature DAG(with Dependency Resolution) and trigger their execution while handling retries and back pressure. FS-DSA : Data Science Automation for Model Training + Deployment integrated with Feature Store | Enables versioned and reproducible experiments. FSBrokerAPI : Online Serving RESTful API endpoint for consumer applications
- 10. FeatureStore Components & Data Flow User Funnel Activity Streams Client-Side Server-Side DATA CAPTURE COMPUTE + FSConfig SERVING + STORAGE Transactional Data Booking Master FSConfig : Feature Catalog Master Datastore Product Master, User Master, Device Master New Data Stream s ML Automation BT-Compute BATCH Feature Compute Jobs RT-Compute Feature Compute SERVING API Offline Models Online Models Batch BULK API (DataFrame) Feature Definitions BoulderDB REDIS Feature Storage Job Scheduler Sagemaker TRAIN Training + HPO Deploy Docker / Batch Transform
- 11. FSConfig : Feature Definitions & Metadata Feature Name : <Entity>::<Feature_shortname>::< Data Time Interval>::<Refresh Frequency>::<Version> Entity : <UserID>_<profileType> Short Name : listing_conversion_rank Versioning : v2 + Process : RT/BT FeatureGroup : (System Generated ID) 8fda73d1_2eee_4cfc_a20f_e9afb1 78fbc3 Entity: ["uuid", "profile_type"] Features [Array] Time Window(Refresh/ Data - Time duration): (ISO Time Interval) P1D Data Source [Array]: [user_master, txn_search] Data Store: GLUE/S3 Database Name: blueshift Table Name: [user_master, txn_search] Data Sink: Serving [Array] Data Store: GLUE Catalog/S3/Redis/BoulderDb Database Name : rocksDB_<WAL Dir Path> Table Name : rocksDB_<columnFamily> Compute Logic DSL + Spark SQL: metric_expr, group_by_expr, filter_expr, window_function, window_function_alias Code (Python/Scala/Java) : GIT/Gerrit URI Model(sagemaker) / Embedding Environment: Production Workspace: Dev/Staging/Production Namespace: <Project Name> Apache LIVY + Databricks JOBs API Config
- 12. FS Store | online + historical Output Schema (internal to the system) ● Historical Feature Data schema on S3 Parquet |-- entity: string (nullable = false) |-- uuid_profileType::listing_conv_rank::P30D::P15M::v1: long (nullable = false) |-- uuid_profileType::listing_view_rank::P30D::P15M::v1: long (nullable = false) |-- uuid_profileType::cnt_distinct_bk_bankid::P30D::P15M::v1: map (nullable = false) | |-- key: string | |-- value: integer (valueContainsNull = true) .. .. All features in that feature group ● Online Serving Data Schema on REDIS + BoulderDB ○ Serving at Feature Group level Key -> <Entity_id>#<Feature_group_id>/<Feature_split> Value -> Hashes key -> Feature_name Value -> Feature_value TimeStamp -> Compute_Processed_Time ○ Serving at Feature Level Key -> <Entity_id>#<Feature_name> Value -> Hashes key -> Feature_name Value -> Feature_value SERVING Config - lambda (batch_feature_name linkage for RT features) - Support for linear QUERY DAGs - MVEL based post-processing on any feature per service/model if needed Feature backfill (back_fill_required, back_fill_duration)
- 13. FS-BrokerAPI : Online Feature Serving Framework Data Access LayerREQUEST HANDLER Orchestration Layer Orchestration + Broker Extractors Transport Business Logics + MVEL Extractors Transport <uri>/v1/getFeature s (POST Request) AKKA(Actors) Request Validations Feature Definition Request Handler REDIS Boulder DB FeaturesbyName FeaturesbyModel FeaturesbyService
- 14. BoulderDB : Online Serving Store - Build on top of RocksDB (embedded data store: developed by Facebook) : reducing the distance to data on serving layer. - Steps added to compute layer: post-processing: - BT-Compute Layer after processing data through SPARK(distributed) - writes into SST Files across various executors into shared object storage : S3 - Split spark dataframe into non-overlapping ranges : individual split is sorted by KEY, then it is ingested into sst file per partition / executor - Cluster coordinator : Consul - Atomic switching of DB snapshots - Data is sharded (helps with proximity by Namespace) and replicated(RF=2)
- 15. Tools
- 16. Next Steps - Feature Stats Visualization / Analytics & Monitoring // Feature Catalog - Seamless integration with Experimentation Framework - Per User Databases on top of feature-store for Personalization - Notebook integration : More better Data Science Tools for Data Scientists with Python libraries - Perf Tools : Query Optimization & Analysis
- 17. References - https://www.logicalclocks.com/feature-store/ - https://eng.uber.com/scaling-michelangelo/ - Airbnb : Zipline - HopsML + Hopsworks - Go-JEK : FEAST - The Design of Systems for Real-time Prediction Serving | DataEngConf SF '18 - https://medium.com/makemytrip-engineering
- 18. Piyush Kumar E : piyush.kumar@makemytrip.com W : www.makemytrip.com T : https://twitter.com/piykumar Thank you !!