MLSD18. Real World Use Case II
0 likes147 views
The document describes the development of a house recommender system using machine learning techniques. It discusses using anomaly detection to identify and remove unusual homes from recommendations, using machine learning to impute missing data values, clustering homes into groups to provide more varied recommendations, association discovery to understand relationships between home features and clusters, and topic modeling of home descriptions to provide deeper insights into grouping homes. The overall goal is to build a preference model based on user input to filter and recommend relevant homes for sale.
MLSD18. Real World Use Case II
- 1. 1st edition November 4-5, 2018 Machine Learning School in Doha
- 2. BigML, Inc X Real-world Use Case House Recommender Full Name Role, Company
- 3. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Let’s build a recommender Typical way to shop for a home…
- 4. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Recommender Idea ? ? ? ? Preference Model Preference Data Sample … then use the Preference Model to filter all the homes on the market All Homes Forsale
- 5. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Recommender Problem #1 What if there are really unusual homes in the data? • A mansion with 20 bathrooms • A home with no bedrooms • A lot size that is smaller than the home? We don’t want to show these as suggestions because they are unusual…. How do we detect anomalies?
- 6. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Anomaly Detection
- 7. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · What just happened? • We wanted to find and remove unusual houses. • We created an Anomaly Detector and examined the top anomalies. • We found some unusual houses to remove and discovered bad data (missing values) that we want to fix.
- 8. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · ML to fix missing data… • Let’s use Machine Learning… BEDS BATHSWhat happened to being easy? SQFT PRICE BEDS BATHS 3.125 US$530.000 5 3 2.100 US$460.000 2 1.200 US$250.000 3 3.950 US$610.000 6 4 4 1.5
- 9. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · WhizzML
- 10. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · What just happened? • We had a Dataset with missing values. • We wanted to apply an algorithm to fix the missing values with Machine Learning • Rather than write the algorithm, we found what we needed in the WhizzML public gallery. • Now that we have cloned the Script we can use it again and again. • We can write new ones too!
- 11. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Recommender Problem #2 • How can we avoid showing essentially the same house over and over? All Homes ? ? ? Sample Modern
- 12. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Recommender Problem #2 • How can we avoid showing essentially the same house over and over? All Homes Modern Lots of Land • Great! What if we don’t know how to group them? Or how many groups? ? sample ? sample
- 13. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Clustering
- 14. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · What just happened? • Since we don’t know how many groups of homes there should be, we used G-means Clustering to find the optimum number of groups of homes • Our recommender will use these groups to create a better sampling for user preference • We also tried to understand the home Clusters using “model clusters” but the models were difficult to interpret
- 15. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Understanding Clusters If SQFT >= 3,125 THEN “Cluster 1” What if we could get rules like… SQFT PRICE BEDS BATHS CLUSTER 3.125 US$530.000 5 3 Cluster 1 2.100 US$460.000 4 2 Cluster 3 1.200 US$250.000 3 1,5 Cluster 5 3.950 US$610.000 6 4 Cluster 1
- 16. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Association Discovery
- 17. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · What just happened? • We used a Batch Centroid to add the Cluster assignment of each home as a feature to the Dataset • We use Association Discovery to find “interesting” relationships between the features including the Cluster assignment
- 18. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Recommender Problem #3 There is much more interesting information than just the number of BEDS, BATHS, etc. • Unfortunately, these "remarks" are not available in the Redfin download • Adding them to our dataset requires crawling the website • Like most ML projects, preparing the data is 80% of the difficulty (fortunately I already did it!)
- 19. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Topic Modeling
- 20. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · What just happened? • We extending the home dataset with the syndicated remarks text field • We built a Model to predict sale price and explored how key words discovered in the remarks impacted price • We used Topic Modeling to create a deeper thematic understanding of the remarks • Homes that are "in-town" or "out-of-town" • We extended the Dataset with fields that represent for each home how related they are to each of these topics • This will allow our Clustering to group homes by a deeper meaning than just BEDS, BATHS, etc
- 21. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · Recommender Idea ? ? Modern Lots of Land Small ? ? ? ? Preference Model Preference Data
- 22. BigML, Inc X· @bigmlcom · @QatarComputing · #MLSD18 · House Recommender