When automated analysis goes wrong
Download as PPTX, PDF0 likes499 views
In his presentation at the MCN Conference 2017, Tristan Roddis discusses both the successes and failures of automated image analysis using machine learning. Key successes include tagging visual content and optical music recognition, while failures involved issues with term extraction and automated captions. The conclusions emphasize the importance of embracing failures and integrating human input in automated processes.
![Tristan Roddis. Cogapp
Other uses
• Blank sheets vs. writing
• Handwritten vs. typewritten
• [your collection-specific question here]](https://image.slidesharecdn.com/automatedanalysisfailures-171110172609/85/When-automated-analysis-goes-wrong-23-320.jpg)
When automated analysis goes wrong
- 1. When automated analysis goes wrong Tristan Roddis Cogapp MCN conference 2017
- 2. Tristan Roddis. Cogapp In this session • Successes • FAILURES
- 3. Tristan Roddis. Cogapp Failure 1: Interesting images
- 4. Tristan Roddis. Cogapp The problem
- 5. Adrian Hindle, Tristan Roddis. Cogapp "Nourishment for the Ailing" and "Nourishment for the Healthy" http://www.qdl.qa/en/archive/qnlhc/9549.20
- 6. Adrian Hindle, Tristan Roddis. Cogapp "Nourishment for the Ailing" and "Nourishment for the Healthy" http://www.qdl.qa/en/archive/qnlhc/9549.19
- 7. Adrian Hindle, Tristan Roddis. Cogapp
- 10. Tristan Roddis. Cogapp Success 1: Interesting images
- 11. Adrian Hindle, Tristan Roddis. Cogapp • Automatically tag, organize, and search visual content with machine learning. • Create concepts
- 12. Adrian Hindle, Tristan Roddis. Cogapp • Existing concepts: • Illustration • Drawing
- 13. Adrian Hindle, Tristan Roddis. Cogapp • Negative concept: • Text • Manuscripts
- 14. Adrian Hindle, Tristan Roddis. Cogapp • Combination of two custom concepts • arabic_manuscript • arabic_ manuscript_with_image
- 15. Adrian Hindle, Tristan Roddis. Cogapp 45 images 45 images
- 16. Adrian Hindle, Tristan Roddis. Cogapp 26 images 26 images
- 17. Adrian Hindle, Tristan Roddis. Cogapp • Python script to create sets and train • Test and train
- 19. Adrian Hindle, Tristan Roddis. Cogapp
- 22. Tristan Roddis
- 23. Tristan Roddis. Cogapp Other uses • Blank sheets vs. writing • Handwritten vs. typewritten • [your collection-specific question here]
- 24. Tristan Roddis. Cogapp Failure 2: term extraction
- 25. Tristan Roddis. Cogapp Failure 2: term extraction
- 26. Tristan Roddis. Cogapp Failure 2: term extraction
- 27. Tristan Roddis. Cogapp Failure 2: term extraction
- 28. Tristan Roddis. Cogapp Success 2: add a human
- 29. Tristan Roddis. Cogapp Failure 3: Finding similar images
- 34. Tristan Roddis. Cogapp Scikit-learn • Machine learning is not easy • Especially for this problem
- 35. Tristan Roddis. Cogapp Success 3: Finding similar images
- 36. Tristan Roddis. Cogapp Term extraction • Clarifai • Google Vision API • Microsoft Computer Vision
- 38. Tristan Roddis. Cogapp Google Vision API
- 39. Tristan Roddis. Cogapp MS Computer vision
- 40. Tristan Roddis. Cogapp Images • Sourced from Nationalmuseum Sweden • Using Europeana API for discovery • 2000 images • http://labs.cogapp.com/iiif-ml/
- 43. Tristan Roddis. Cogapp Success 4: Optical music recognition
- 47. Tristan Roddis. Cogapp Failure 4: Automated captions
- 48. A vase of flowers on a display
- 51. Emanuel Swedenborgsitting in front of a laptop
- 52. Person using a phone
- 53. Person sitting in a chair talking on a cell phone
- 54. A man and a woman taking a selfie
- 55. A cat with its mouth open
- 56. A pizza sitting on top of a window
- 57. A man riding a bear in the water
- 58. A teddy bear
- 59. A group of sheep standing on top of a horse
- 60. A group of sheep standing on top of a book
- 61. Tristan Roddis. Cogapp Conclusions • Embrace failure • Find workarounds • Add humans to the mix
Editor's Notes
- #3: Digital agency based in UK. Work internationally. Would love to work for you.
- #4: Present some experiments and research we’ve been doing. Look at three problems that can be solved using automated image analysis: interesting items, color extraction, finding similar images In all cases: images only. Deliberately not using metadata: manifests only for providing lists of images to analyse. Present our findings: some positive, full disclosure: some negative
- #5: Adrian We manage the Qatar Digital Library website, that currently has nearly a million scanned pages. Set ourselves the challenge of automating the process of finding “visually interesting” document pages Not sure what it is but we know what it’s not (not text, not blank, not bindings...
- #6: Over 600 pages Several dozen manuscripts available, so tens of thousands of images. And constantly adding more, so not easily achievable by humans. Impractical
- #7: We have some info but not for all, this is on the logical, hard to extract info or missing
- #8: Examples
- #12: Tristan First approach we tried was colour analysis More colours used in illustrations than in plain black script Imagaa extracts foreground/background colours, “color variance”
- #14: Between variance 11 and 17. Hugely mixed. Gave up: tried a different tack.
- #15: Adrian We manage the Qatar Digital Library website, that currently has nearly a million scanned pages. Set ourselves the challenge of automating the process of finding “visually interesting” document pages Not sure what it is but we know what it’s not (not text, not blank, not bindings...
- #16: Adrian
- #17: Lots of concepts: food, colour, focus ... Picking up stains
- #18: Still not good results
- #19: Call this what you want Tried with one - not very good Build up two training sets: one with positive results, one with negative results
- #23: IIIF Collection picked 10 random archives and then 10 images per archive Red -> interest Trained it a couple times (good results) 1 error here updated the set
- #24: Example for one archive
- #25: And it works
- #26: And again
- #27: Created a manifest - Mirador
- #29: Last point: you can apply this technique of negative/positive training sets to _any_ visual problem. We demonstrated one version of this particular to our collection, but I’m sure you can think of similar questions about your own.
- #30: Tristan Colour extraction is the low-hanging fruit of automated image analysis Very easy to do via scripts or APIs
- #32: “a machine quite literally just said the Met’s collection is full of shit”
- #33: Tristan Colour extraction is the low-hanging fruit of automated image analysis Very easy to do via scripts or APIs
- #34: Paris, Texas vs Paris, France
- #35: Adrian Last problem we looked Tried two approaches
- #36: Collection of algorithms for image processing Skeletonize: each pixel removed if doesn’t break connectivity, then segment colour Censure: feature detector (scale invariant center-surround detector) Daisy: local image descriptor based on gradient orientation histograms These are good to find the same image (with noise...)
- #37: Mean squared error: square of the diff between px in A and B, sum and divide by nb of pixels Structural similarity: measure similarity (viewed as a quality measure) Results as expected
- #38: Very different
- #39: These 2 are more similar
- #40: Then started working with ML Harder than it looks (ok if you are an ML scientist) Not quick Tried something else
- #41: Adrian Last problem we looked Tried two approaches
- #42: Tristan 10 days before conference. Tried different approach.
- #46: Tristan: only indexed if confidence value is > 0.75 Elasticsearch with Searchkit interface
- #47: Focus on left-hand tags
- #48: Different tags
- #50: Different tags
- #52: Different tags
- #53: Adrian Last problem we looked Tried two approaches
- #54: MS captioning. Works well
- #67: Each AI has a “personality” of sorts