AI for Good at Microsoft
- 1. AI for Good @ Deep Art Generation, Unsupervised Object Detection, and Distributed Intelligent Microservices Mark Hamilton marhamil@microsoft.com
- 2. Overview AI for Cultural Institutions: The MET GAN image synthesis Reverse Image Search Intelligent Image Search AI for Earth: The Snow Leopard Trust Unsupervised Object Detection LIME AI for Accessibility: Seeing AI Currency Detection The Tech Stack Spark, MMLSpark, Kubernetes, Microservices Seeing AI
- 3. AI for Cultural Institutions
- 4. Celebrating 2 years of Open Access at The MET 4 https://gen.studio In 2016 The MET Released 400k Images under Open Access This past Winter the MET released a new subject-keyword dataset of image annotations MIT, The MET, and Microsoft participated in a 3-day hackathon to create intelligent experiences using the new collection
- 5. Goals: Create new works of art Use new work to explore existing art Explore further with intelligent search Generative Adversarial Networks Reverse Image Search Elasticsearch with Cognitive Services Needed Technologies:
- 6. Generative Adversarial Art https://gen.studio Real or Generated? Noise Vector Generator Generated Image Training Data Discriminator Real or Generated?
- 7. Custom Reverse Image Search Filters from Zeiler + Fergus 2013 Query Image ResNet Featurizer Deep Features Closest Match Fast Nearest Neighbor Lookup MMLSpark SparkML LSH or Annoy
- 9. Intelligent Search Index Pipe images through Computer Vision API to annotate image for searching Stream images and intelligent annotations to Azure Search A picture containing a person A picture containing a glass, cup A fish swimming underwater Query Image: Describe Image Output: Deep Feature Nearest Neighbors:
- 10. End Application: Gen Studio 10 https://gen.studio
- 11. AI for Earth
- 13. Classifying all 1.3 million images will take 20k hours Remote Camera Traps
- 15. Creating a labelled Training Dataset
- 16. Creating a labelled Training Dataset
- 17. Input Images Low Level Features Mid Level Features Logistic Regression Output Classes Filters from Zeiler + Fergus 2013 Any SparkML Model Other Leopard Predictions Transfer Learning with ResNet 50
- 18. Performance With Deep Featurization, Augmentation, and Temporal Ensembling Without Deep Featurization Accuracy 65.6% Accuracy 94.7%
- 19. Goal: Identify Individual Leopards Source: HotSpotter - Patterned Species Instance Recognition
- 20. Automating Detection with LIME on Spark
- 21. LIME on Spark
- 22. End to End Architecture
- 23. Results Human Labels Unsupervised FRCNN Outputs Human Labels Unsupervised FRCNN Outputs
- 24. AI for Accessibility Seeing AI
- 26. A Familiar Architecture… Union + Distinct MobileNet Logistic Regression Bing Image Search Deep Features Azure Machine Learning + Spark Serving Queries 1 Dollar 5 Dollars 10 Dollars 20 Dollars Labelled Images $1 $5 $2 0 Prep Data Train Deploy
- 27. The Tech Stack
- 28. A fault-tolerant distributed computing framework Map Reduce + SQL Whole program optimization + query pushdown Elastic Scala, Python, R, Java, Julia ML, Graph Processing, Streaming Driver Worker 110 0.9 0 20 40 60 80 100 120 Running Time(s) Hadoop Spark Worker Worker Data Source (ADLS, Blob, Local, Cosmos, etc) Age: Int Name: String 15 Bob 25 Alice 33 Mary Age: Int Name: String 1 Sam 2 Claire 53 Tina Age: Int Name: String 25 Bob 25 Tom 66 Ted ● Code ● Data
- 29. ML High level library for distributed machine learning More general than SciKit-Learn All models have a uniform interface Can compose models into complex pipelines Can save, load, and transport models Load Data Tokenizer Term Hashing Logistic Reg Evaluate Pipeline Evaluate Load Data ● Estimator ● Transformer
- 30. 30#UnifiedAnalytics #SparkAISummit Microsoft Machine Learning for Apache Spark v0.17 Microsoft’s Open Source Contributions to Apache Spark www.aka.ms/spark Azure/mmlspark Cognitive Services Spark Serving Model Interpretability LightGBM Gradient Boosting Deep Networks with CNTK HTTP on Spark
- 31. CNTK on Spark GPU/CPU acceleration Automated gradient calculations Flexible language for defining huge space of models State of the art performance and accuracy ONNX Runtime Large scale parallelism Fault tolerance Auto-scaling / elasticity High throughput streaming Data source + orchestrator agnostic
- 32. Azure Cognitive Services on Spark Easy to use integration between Spark and the Azure Cognitive Services Composable and pipelinable with all other SparkML models! Python, Scala, R (Beta) val df = new TextSentiment() .setTextCol(“text”) .setOutputCol(“sentiment”) .transform(inputs) Spark Worker Partition Partition Partition Client Client Client Web Service
- 33. Spark Serving Sub-millisecond RESTful Model Deployment on Spark Clusters Spark Worker Partition Partition Partition Server Spark Worker Partition Partition Partition Server Spark Master Users / Apps Load Balancer HTTP Requests and Responses spark.read.parquet.load(…) .select(…) spark.readStream.kafka.load(…) .select(…) Batch API: Streaming API: Serving API: spark.readStream.server(“0.0.0.0”, 5000).load(…) .select(…)
- 34. Azure Kubernetes Service + Helm • Works on any k8s cluster • Helm: Package Manager for Kubernetes 34 Kubernetes (AKS, ACS, GKE, On-Prem etc) K8s workerK8s worker Spark Worker Spark Worker K8s worker Cognitive Service Container HTTP on Spark Spark Worker Cognitive Service Container HTTP on Spark Spark Worker Cognitive Service Container HTTP on Spark Spark Serving Load Balancer Jupyter, Zepplin, LIVY, or Spark Submit LB Zepplin Jupyter Storage or other Databases Cloud Cognitive Services Spark Serving Hotpath HTTP on Spark Spark Readers REST Requests to Deployed Models Submit Jobs, Run Notebooks, Manage Cluster, etc Users / Apps helm repo add mmlspark https://dbanda.github.io/charts helm install mmlspark/spark --set localTextApi=true Dalitso Banda, dbanda@microsoft.com Microsoft AI Development Acceleration Program
- 35. Thanks to You all! Joseph Sirosh, Sudarshan Raghunathan, Ilya Matiach, Eli Barzilay, Tong Wen Microsoft NERD Garage Team + MIT Externship Program Microsoft Development Acceleration Team: Dalitso Banda, Casey Hong, Karthik Rajendran, Manon Knoertzer, Tayo Amuneke, Alejandro Buendia Pablo Castro, Chris Hoder, Ryan Gaspar, Henrik Neilsen, Andrew Schonhoffer, Daniel Ciborowski, Markus Cosowicz Azure CAT, AzureML, and Azure Search Teams