Ai powered images-zurichpydata
Download as PPTX, PDF0 likes37 views
Doug Sillars gave a presentation on using AI to optimize images for the web. He discussed how images dominate web content and explained techniques like cropping, blurring objects, and generating alt text using machine learning models. Sillars also demonstrated how to train custom models for tasks like detecting sunglasses and adding filters to photos. The presentation concluded by emphasizing how AI and ML can simplify and automate image preparation and processing for digital content.
Ai powered images-zurichpydata
- 1. Using AI to Power Your Images Doug Sillars Zurich PyData December 11, 2019
- 2. Contact Me: @DougSillars Doug.Sillars@gmail.com www.dougsillars.com Doug Sillars Freelance Developer Relations Performance Audits: Web/Native Workshops: Performance/Images/Video/AR/ML http://bit.ly/HighPerformanceAndroidApps
- 3. Images Dominate the Web HTTPArchive mobile websites 04/2019
- 4. Image Prep for the Web
- 5. Image Prep for the Web
- 6. Image Prep for the Web
- 7. Image Prep for the Web
- 8. Image Prep for the Web
- 9. Shortcuts for Image Prep?
- 10. Shortcuts for Image Prep? Ensure You are delivering the optimal image
- 11. Shortcuts: Taking screenshots of Photoshop
- 12. Shortcuts for Image Prep?
- 13. Images on the web named screen*shot 426,000 screenshots live on the web April 2019
- 14. Images on the web named screen*shot 106,000 sites with Screen Shots April 2019
- 15. Images on the web named screen*shot 5M sites tested 106,000 sites with Screen Shots
- 16. Images on the web named screen*shot 5M sites tested 106,000 sites with Screen Shots 2.1% of the web! =
- 17. Image Formats – Screenshots -> JPG https://dougsillars.com/2019/02/10/using-screenshots-in-production/ Sometimes our Shortcuts have lasting effects
- 18. Shortcuts for Image Prep?
- 19. Shortcuts for Image Prep?
- 20. Shortcuts for Image Prep? https://res.cloudinary.com/dougsillars/image/upload/c_crop,h_1000,w_1000,q_auto,f_auto/v1558503538/doug_headshot_alps _tfzado.jpg
- 21. Shortcuts for Image Prep? https://res.cloudinary.com/dougsillars/image/upload/c_crop,h_1000,w_1000,q_auto,f_auto/v1558503538/doug_headshot_alps _tfzado.jpg 120 KB
- 22. Shortcuts for Image Prep?
- 23. Shortcuts for Image Prep? https://res.cloudinary.com/dougsillars/image/upload/c_crop,h_1080,w_1000,q_auto,f_auto/ v1558504239/IMG_20160527_131042692_HDR_c21a3k.jpg
- 24. Shortcuts for Image Prep Manual crop
- 25. Shortcuts for Image Prep Manual crops are not a shortcut
- 26. Shortcuts for Image Prep: Detection of Objects https://res.cloudinary.com/dougsillars/image/upload/g_auto,c_crop,w_1000,h_1000/ v1558504239/IMG_20160527_131042692_HDR_c21a3k.jpg
- 27. Image Cropping in Production: Twitter
- 28. Image Cropping in Production: Twitter https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/Smart-Auto-Cropping-of-Images.html
- 29. Object Detection Using Machine Learning • ImageAI library • Tensorflow based • Need Image Model Library detector = ObjectDetection() detector.setModelTypeAsRetinaNet() detector.setModelPath( os.path.join(execution_path , "resnet50_coco_best_v2.0.1.h5")) detector.loadModel() detections = detector.detectObjectsFromImage(input_image=os.path.join(execution_path , "berwickontweed.jpg"), output_image_path=os.path.join(execution_path , "resnet.jpg")) resnet=plt.imread('resnet.jpg')
- 30. Object Detection Using Machine Learning
- 31. Object Detection Using Machine Learning
- 32. Object Detection with Machine Learning
- 34. Object Detection • Cropping • Blurring objects
- 35. Object Detection • Cropping • Blurring objects • Object Removal http://bit.ly/bootsphoto
- 36. Object Detection • Cropping • Blurring objects • Object Removal • Alt Text Creation
- 37. Object Detection • Cropping • Blurring objects • Object Removal • Alt Text Creation Alt = “A Person and 2 dogs”
- 38. If the Object is not Modeled…
- 39. If the Object is not Modeled… •person : 99.72 •dog : 88.67 •dog : 96.92
- 40. …Try a new library • https://llama.vision
- 41. …Try a new library • https://llama.vision
- 42. Object Detection Using Machine Learning
- 43. ML ALT text in production alt="Image may contain: bridge, sky, tree, outdoor, water and nature"
- 44. ML ALT text in production alt="Image may contain: dog"
- 45. ML ALT text in production
- 46. …Or you can train a model
- 48. Google Object Detection: Model Creation
- 49. Google Object Detection: Model Creation
- 50. Google Object Detection: Model Creation
- 51. Google Object Detection: Model Creation
- 52. Train Your Model Precision: How often Model’s predictions were correct. In this case 1/6 is a false positive Recall: How often model found known objects. 1/6 is a missed prediction.
- 53. Deploy Model and Use
- 54. Deploy Model and Use
- 55. Model Results!!
- 56. Model Results!!
- 57. Image Fix 1: Cropping
- 58. Image Fix 1: Cropping
- 59. Image Fix 1: Cropping
- 60. Image Fix 2: Inpainting
- 61. Image Fix 2: Inpainting Create a mask of the area
- 62. Image Fix 2: Inpainting Create a mask of the area
- 63. Image Fix 2: Inpainting
- 64. Image Fix 2: Inpainting
- 65. InPaint: Works well on single color areas
- 66. InPaint: Works Not So well with lots of color
- 67. Tricking InPaint Remove “outlier” colors from the border Use the inside as inpainting area
- 68. Tricking InPaint Convert to Grayscale colors are 0 – 255 (1D) Kmeans to bucket the colors
- 69. Tricking InPaint
- 70. Tricking InPaint 10 most common gray colors found
- 71. Tricking InPaint 10 most common gray colors found Frequency
- 72. Tricking InPaint 1. Loop through Color list
- 73. Tricking InPaint 2. Ignore the White Center
- 74. Tricking InPaint 3. If Color is an outlier (lower than 25th percentile Higher than 75th) Mask and replace
- 75. Tricking InPaint
- 76. Tricking InPaint
- 77. Tricking InPaint
- 81. Manually edit 1,000s of glasses?
- 85. Sunglasses Project 1. Remove Background
- 86. Sunglasses Project 1. Remove Background
- 87. Sunglasses Project 2. Make (slightly) transparent https://res.cloudinary.com/dougsillars/image /upload/e_make_transparent,o_70/v1558760 129/6658.png
- 88. Train a model!
- 89. Want to minimize frame capture
- 90. Train a model!
- 91. Beginning
- 92. Find the arms
- 93. Find the arms Finding an Odd number: Fail
- 94. Modify the border around the arms
- 96. Putting it all together https://res.cloudinary.com/dougsillars/image/upload/v1558855691/IMG_20181207_163533_ibfm7a.jpg
- 97. Putting it all together https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/v1558855691/IMG_2018120 7_163533_ibfm7a.jpg
- 98. Putting it all together https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/v1558855691/IMG_2018120 7_163533_ibfm7a.jpg l_6558_f,o_80,e_make_transparent/
- 99. Putting it all together https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/l_6558_f,o_80,e_make_transparent/ g_adv_eyes,fl_layer_apply/v1558855691/IMG_20181207_163533_ibfm7a.jpg
- 101. Conclusion • Image processing with AI and ML can simplify image prep for the web • Cropping • Blurring • Object removal • Alt text
- 102. Love Building with Video and Images? Media Developer Expert mde-comm@cloudinary.com Become a
Editor's Notes
- #43: https://twitter.com/SteveStuWill/status/1186653595518652416