![[Kiapour et al] Where to Buy It: Matching Street
Clothing Photos in Online Shops
+ learn a NN similarity function
between query and candidate to
outputs match-score; works well
- querying is expensive since score
must be computed for all candidates
- learn separate functions for each
category (tops, footwear, …)
threadgenius.co](https://image.slidesharecdn.com/usingdeepneuralnetworksforfashionapplications-170224200020/85/Using-deep-neural-networks-for-fashion-applications-20-320.jpg)
![[Liu et al] DeepFashion: Powering Robust Clothes Recognition and
Retrieval with Rich Annotations
+ provide 800K fashion images for academic community
+ perform landmark detection for localization
- questionable data quality: taxonomy contains irrelevant
words (eg. brooklyn, kurt), images are mislabeled, bounding boxes
too tight
threadgenius.co
images for
“brooklyn”](https://image.slidesharecdn.com/usingdeepneuralnetworksforfashionapplications-170224200020/85/Using-deep-neural-networks-for-fashion-applications-21-320.jpg)
![[Bell et al] Learning visual similarity for product design with
convolutional neural networks
threadgenius.co
+ learn image embeddings that place similar products close
- requires cropped images for querying](https://image.slidesharecdn.com/usingdeepneuralnetworksforfashionapplications-170224200020/85/Using-deep-neural-networks-for-fashion-applications-22-320.jpg)
![[Bell et al] Learning visual similarity for product design with
convolutional neural networks
threadgenius.co
train on two tasks
● metric learning
● category prediction+
-](https://image.slidesharecdn.com/usingdeepneuralnetworksforfashionapplications-170224200020/85/Using-deep-neural-networks-for-fashion-applications-23-320.jpg)
![[Ren et al] Faster R-CNN: Towards Real-Time Object
Detection with Region Proposal Networks
threadgenius.co
object detection model](https://image.slidesharecdn.com/usingdeepneuralnetworksforfashionapplications-170224200020/85/Using-deep-neural-networks-for-fashion-applications-24-320.jpg)
![[Ren et al] Faster R-CNN: Towards Real-Time Object
Detection with Region Proposal Networks
threadgenius.co
train a mini-network that
classifies objectness and
regresses bounding boxes](https://image.slidesharecdn.com/usingdeepneuralnetworksforfashionapplications-170224200020/85/Using-deep-neural-networks-for-fashion-applications-25-320.jpg)
![images require cleaning
threadgenius.coimages from [Kiapour et al]](https://image.slidesharecdn.com/usingdeepneuralnetworksforfashionapplications-170224200020/85/Using-deep-neural-networks-for-fashion-applications-32-320.jpg)
![threadgenius.co
experimental model [beta]
word2vec on collections of images and text
● words represented by embedding / lookup table
● images represented by convnet
Long sleeve silk chiffon shirt-style
dress featuring graphic pattern in
navy, burgundy, and green. Vented
crewneck collar. Gathering at
front and back yokes. Detachable
self-tie fastening at waist.
Two-button barrel cuffs.
Detachable viscose chiffon slip
lining in black. Tonal stitching.
Body: 100% silk. Lining: 100%
viscose. Imported.
document](https://image.slidesharecdn.com/usingdeepneuralnetworksforfashionapplications-170224200020/85/Using-deep-neural-networks-for-fashion-applications-44-320.jpg)
Using deep neural networks for fashion applications
- 1. Using Deep Neural Networks for Fashion Applications Ahmad Qamar
- 3. motivation challenges related work data collection Thread Genius applications future work demos threadgenius.co
- 4. set of terms used interchangeably ● features, fingerprints, representations, latent factors, vectors ● attributes, labels, concepts ● product , SKU, in-store ● street, wild-type, UGC threadgenius.co
- 6. threadgenius.co 2000 today → web content increasingly image-heavy → more participation and engagement + 1.8B+ photos uploaded daily (2014) - only 15% contain relevant metadata
- 7. → millennials are increasingly brand agnostic - logo detection fails in extracting signal threadgenius.co
- 8. threadgenius.co millennials prefer branded content that is ... ● social is saturated with content ● attention is limited commodity authentic and information dense
- 10. + attribute extraction works well - limited and generic taxonomy of attributes - word attributes don’t fully capture image threadgenius.co visual understanding is becoming ubiquitous
- 11. - visual search suffers from poor results - lack of focus on specific domain threadgenius.co vs.
- 12. query out-of-the-box model fashion domain model
- 13. threadgenius.co
- 14. threadgenius.co
- 15. threadgenius.co
- 16. collaborative filtering not fit for fashion “people who buy a also buy b” - no information about content - half-life of fashion products is ~1.5 months - higher quantity (vs. movies), lower volume (vs. music) threadgenius.co
- 17. image models require lots of data to prevent overfitting - limited public datasets for fashion - taxonomy requires domain consideration - collecting training data is painstaking - ambiguity and variability present in fashion products and photos threadgenius.co
- 19. threadgenius.co Convnets 101: convnets learn features and the classifier simultaneously images from Sander Dieleman image
- 20. [Kiapour et al] Where to Buy It: Matching Street Clothing Photos in Online Shops + learn a NN similarity function between query and candidate to outputs match-score; works well - querying is expensive since score must be computed for all candidates - learn separate functions for each category (tops, footwear, …) threadgenius.co
- 21. [Liu et al] DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations + provide 800K fashion images for academic community + perform landmark detection for localization - questionable data quality: taxonomy contains irrelevant words (eg. brooklyn, kurt), images are mislabeled, bounding boxes too tight threadgenius.co images for “brooklyn”
- 22. [Bell et al] Learning visual similarity for product design with convolutional neural networks threadgenius.co + learn image embeddings that place similar products close - requires cropped images for querying
- 23. [Bell et al] Learning visual similarity for product design with convolutional neural networks threadgenius.co train on two tasks ● metric learning ● category prediction+ -
- 24. [Ren et al] Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks threadgenius.co object detection model
- 25. [Ren et al] Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks threadgenius.co train a mini-network that classifies objectness and regresses bounding boxes
- 27. fashion products exhibit different types of structure ● category: top, footwear, ... ● product: sweater, pumps, ... ● detail: shawl collar, square toe, ... ● color: yellow, anthracite, ... ● pattern: paisley, colorblock, ... ● aesthetic: preppy, scandalous, … - degrees of freedom make similarity search subjective threadgenius.co
- 28. compiled a taxonomy of ~1000 fashion attributes threadgenius.co
- 29. for each attribute, images are sourced from ● retail websites ● social media networks (Pinterest, The Hunt) ● and fashion resale networks (Poshmark, Vinted) threadgenius.co Long sleeve silk chiffon shirt-style dress featuring graphic pattern in navy, burgundy, and green. Vented crewneck collar. Gathering at front and back yokes. Detachable self-tie fastening at waist. Two-button barrel cuffs. Detachable viscose chiffon slip lining in black. Tonal stitching. Body: 100% silk. Lining: 100% viscose. Imported.
- 30. + image-attribute pairs help model learn fashion feature detectors - image-attribute pairs are not enough: attribute classifiers simply compute histograms over visual features threadgenius.co ≈
- 31. + image-image pairs allow for unsupervised learning of similarity + captures invariances threadgenius.co + -
- 32. images require cleaning threadgenius.coimages from [Kiapour et al]
- 35. images/UGC unified space of images, products, and metadata products OPENING CEREMONY $495 ZARA $125metadata varsity jacket, color black, standup collar threadgenius.co
- 36. threadgenius.co object detector RPN model trained on bounding box labels feature extractor alternate training on attribute classification and metric learning indexing
- 37. Tech Stack Research pipeline Deployment pipeline Training data ... Data annotation Lasagne Annoy Model training + experimentation + validation Compute + Storage + Server TG GPU Server Product Inventory Lookbooks ... fashion blogs ... Workflow manager TG API threadgenius.co
- 39. retail alternative products to: sold out inventory, pricey items shop.threadgenius.co
- 40. visual marketing making Instagram shoppable + 2-3x lift in conversion threadgenius.co
- 41. audience generation build custom audiences for specific products threadgenius.co
- 42. audience generation + 3x increase in CTR threadgenius.co
- 44. threadgenius.co experimental model [beta] word2vec on collections of images and text ● words represented by embedding / lookup table ● images represented by convnet Long sleeve silk chiffon shirt-style dress featuring graphic pattern in navy, burgundy, and green. Vented crewneck collar. Gathering at front and back yokes. Detachable self-tie fastening at waist. Two-button barrel cuffs. Detachable viscose chiffon slip lining in black. Tonal stitching. Body: 100% silk. Lining: 100% viscose. Imported. document
- 45. threadgenius.co other ● combined model that directly extracts features of component apparel items (convnet+RNN) ● refine training of experimental image-text model
- 47. threadgenius.co experimental model: semantic arithmetic with images and text
- 49. Questions? we’re hiring for backend and ML roles
- 50. we’re hiring for backend and ML roles ahmad@threadgenius.co