Leveraging Social Media with Computer Vision
3 likes1,217 views
This document discusses using computer vision and social media images to improve fashion recommendations. It motivates using recommendations due to large inventory sizes and limits on human memory. It then outlines using deep learning on social media images to extract visual features for recommendations, addressing cold start problems. Metric learning is discussed to learn image similarities across different domains. The document concludes there is potential to improve recommendations using attributes from social media images and predictive models.
Leveraging Social Media with Computer Vision
- 1. Leveraging Social Media with Computer Vision TJ Torres Data Scientist, Stitch Fix Big Data Applications in Fashion MeetUp 10/2016 Informing Recommendations in Fashion and Retail
- 5. Data LabsStyling Algorithms Research
- 6. MOTIVATION Inventory Scaling: Why Recommendations? Infeasible from an efficiency perspective to look through all inventory as it scales.
- 7. MOTIVATION Inventory Scaling: Human Ability: Why Recommendations? Infeasible from an efficiency perspective to look through all inventory as it scales. Stylists can’t keep all products in their memories while trying to locate the best items for each client.
- 8. MOTIVATION Inventory Scaling: Human Ability: Why Recommendations? Infeasible from an efficiency perspective to look through all inventory as it scales. Stylists can’t keep all products in their memories while trying to locate the best items for each client. Business Success: Aid stylists in making the best decisions to better please our clients.
- 9. MOTIVATION Our goal at Stitch Fix Total Inventory Recommendation Algo Stylists Filtered Items 1 2 3 4 5 Final Items Sent
- 10. COMPUTER VISION
- 11. COMPUTER VISION New Clients New Clothing Cold Start Problem No or sparse purchasing information, so how can we supplement this?
- 12. COMPUTER VISION New Clients New Clothing Cold Start Problem No or sparse purchasing information, so how can we supplement this? Perception Fashion can be difficult to describe via text/categorization. Many times it’s easier to show what you like.
- 13. TURN TO IMAGES • Style/fashion is primarily visual. • We wish to use images for modeling purposes. • Heuristics for how we process image data unknown or quite complex. • We don’t want to have to develop image features. • Turn to deep learning to learn the feature extraction.
- 14. OUTLINE 1. Brief Introduction to NNs 2. Deep Learning for Fashion Imagery 3. Recommendations and Social Media 4. Results 5. Conclusions
- 18. Gatys, et. al. : https://arxiv.org/abs/1508.06576
- 19. Begin with input: INTRO TO NEURAL NETS 1 2 3 4 5 6
- 20. Begin with input: 1 2 3 4 layer 1 (Input) 5 6 layer 2 f (l) i (x) = tanh 0 @ X j W (l) ij x (l 1) j + b(l) 1 A INTRO TO NEURAL NETS
- 21. Begin with input: 1 2 3 4 layer 1 (Input) 5 6 layer 2 f (l) i (x) = tanh 0 @ X j W (l) ij x (l 1) j + b(l) 1 A layer 3 (output) Transform data repeatedly with non-linear function. f(1) · · · f(n) (x) INTRO TO NEURAL NETS
- 22. 1 2 3 4 layer 1 (Input) 5 6 layer 2 layer 3 (output) Calculate loss function and update weights f(1) · · · f(n) (x) L(xout, y) = MSE z }| { 1 m mX k=1 (xk yk)2 Begin with input: f (l) i (x) = tanh 0 @ X j W (l) ij x (l 1) j + b(l) 1 A Transform data repeatedly with non-linear function. INTRO TO NEURAL NETS
- 23. 1 2 3 4 layer 1 (Input) 5 6 layer 2 layer 3 (output) L(xout, y) = MSE z }| { 1 m mX k=1 (xk yk)2 W (l)⇤ ij = W (l) ij ✓ 1 ↵ @L @Wij ◆ Calculate loss function and update weights f(1) · · · f(n) (x) Begin with input: f (l) i (x) = tanh 0 @ X j W (l) ij x (l 1) j + b(l) 1 A Transform data repeatedly with non-linear function. INTRO TO NEURAL NETS
- 24. 1 2 3 4 layer 1 (Input) 5 6 layer 2 layer 3 (output) L(xout, y) = MSE z }| { 1 m mX k=1 (xk yk)2 W (l)⇤ ij = W (l) ij ✓ 1 ↵ @L @Wij ◆ @L @W (l) ij = ✓ @L @xout ◆ ✓ @xout @f(n 1) ◆ · · · @f(l) @W (l) ij ! Calculate loss function and update weights f(1) · · · f(n) (x) Begin with input: f (l) i (x) = tanh 0 @ X j W (l) ij x (l 1) j + b(l) 1 A Transform data repeatedly with non-linear function. INTRO TO NEURAL NETS
- 25. RECS AND SOCIAL MEDIA Clients give Pinterest board to visually indicate fashion tastes. Match pinned images to our own styles.
- 26. RECS AND SOCIAL MEDIA Clients give Pinterest board to visually indicate fashion tastes. Match pinned images to our own styles. Strategies
- 27. RECS AND SOCIAL MEDIA Clients give Pinterest board to visually indicate fashion tastes. Match pinned images to our own styles. Strategies Attribute extraction and matching.
- 28. RECS AND SOCIAL MEDIA Clients give Pinterest board to visually indicate fashion tastes. Match pinned images to our own styles. Strategies Attribute extraction and matching. Visual feature similarity.
- 29. RECS AND SOCIAL MEDIA Clients give Pinterest board to visually indicate fashion tastes. Match pinned images to our own styles. Strategies Attribute extraction and matching. Visual feature similarity. Metric learning.
- 30. RECS AND SOCIAL MEDIA Clients give Pinterest board to visually indicate fashion tastes. Match pinned images to our own styles. Strategies Attribute extraction and matching. Visual feature similarity. Metric learning. …or some combination.
- 31. VISUAL FEATURES
- 32. VISUAL FEATURES
- 33. VISUAL FEATURES Use pre-trained extracted features. Compare image features with metric of your choice Cosine Euclidean etc,
- 34. EXAMPLES Query Image Top 5 Results
- 35. EXAMPLES Query Image Top 5 Results
- 36. CHALLENGES Query Image Top 5 Results Sometimes things don’t work out so well… Need system to compare images across separate domains
- 37. METRIC LEARNING New Metric as Objective AnchorPositiveNegative Triplet or Contrastive Loss https://arxiv.org/abs/1404.4661 Ltriplet(a, p, n) = 1 N NX i=1 max {d(f(ai), f(pi)) d(f(ai), f(ni)) + m, 0} !
- 38. METRIC LEARNING https://arxiv.org/abs/1511.05939 m m Positive Negative BeforeTraining AfterTraining BeforeTraining AfterTraining
- 39. METRIC LEARNING https://arxiv.org/abs/1511.05939 m m Positive Negative BeforeTraining AfterTraining BeforeTraining AfterTraining Learn an embedding that obeys the similarity constraints. similarity score = d query, inventory
- 40. EXAMPLES
- 41. EXAMPLES
- 42. CONCLUSIONS 1. Social media images can help make better recommendations. a) Alleviate cold start. b) Provide new features/data for recommendations. 2. Cross-domain image matching can be difficult, but is made easier with deep learning. 3. There’s enormous potential moving forward with this type of work. a) Attribute labeling and trend tracking. b) Predictive models for purchasing probability.