MANDERS Harder Better Faster Stronger Adding face recognition in the mix

Editor's Notes

• #2: “Work it harder, Make it better, Do it faster, Makes us stronger” Sounds like I am going to present to you today a glorification of the potential of technological progress and Arificial Intelligence for the archives. In a way I am, but it’s not random that I chose a quote from the magnificent electronic music duo Daft Punk to title this presentation about AI in general and face recognition in particular at S&V. Very recognizable as their robot persona’s, yet always wearing helmets to avoid their human faces going public. I too have privacy concerns, and therefor made sure this presentation doesn’t go uncensored on slideshare (photos of my face are blurred)
• #3: Artificial Intelligence is very much a buzz word, some might even say a hype.
• #4: But I am a metadata guy, so I prefer the term automatic metadata, since I am more interested in the metadata as output, and not what intelligence is behind it.
• #5: But maybe my focus on the metadata output is a bit shortsighted. So yes, I praise technology for how it can supports humans – archive users are still human after all - to search through our collections via very finegrained, segmented access points. I will show you also some humorous and painful examples of where the technology gets it
• #6: Nowadays there are is a wide range of AI-technologies available.
• #7: In Sound and Vision’s case we operationalized 3 technologies since 2015 in an effort to reduce time spent on manual annotation: Topical labeling via an extraction service that distills thesaurus labels out of subtitles and production metadata A speaker voice recognition service that identifies who’s speaking, and when And finally, since 2019 face recognition has been thrown into the mix.
• #8: To start of with a humorous example: in this episode people where discussing mayonnaise for an hour, what else Dutch people have to talk about. Since ‘mayonnaise’ is commonly abbreviated as Mayo, our term extraction service was sure this episode was about Mayo, a County in Ireland.
• #9: Another example of technology lacking knowledge and skills to interprete the context. While a user can see immediately that the word Trump refers to Donald Trump, a machine doesn’t know who is Trump and blissfully doesn’t care.
• #10: Moving on to speaker voice recognition. What is it / how does it work? Basically we started with 500 speakers out of our thesaurus and created a speaker model database based on fragments in which they speak. And we compare those models - currently more than 2000 - to daily ingested video and radio programs.
• #11: Since our end users will not be happy with many inaccurate results, we work with a quality threshold and we only ingest labels if the machine scores it above a certain probability score and presses the green button.
• #12: Consequence of threshold = labels that have a lower score are left out, even though they might be correct. So we have gaps in our timecoded labeling.
• #13: Let me illustrate how this gaps might look like
• #14: And another gap: we (sometimes) know who’s speaking but we don’t know who is on camera, and there face recognition comes to the rescue.
• #15: These kind of services had an enormous technology boost in the last few years thanks to deep learning. Where 3 years ago we could only speaker label roughly 2 percent of fragments with speech, today it’s closer to 50% with an accuracy of above 90%.
• #16: And for face labeling it’s even better, so why not add it to the mix. And so we did this year. Basically it works the same as speaker labeling: our video content is compared to face models of very important persons, consisting of a bunch of images transformed into vectors, zeroes and ones.
• #17: In the pilot phase we evaluated under which confidence score the service would produce too many inaccurate results
• #18: And then based on this evaluation we calculated what minimal score we would take as a threshold to make sure that minimal 90% of the labels are correct.
• #19: We created a face model database with about 1000 (national and international) faces to recognize. And we compare our daily ingested videos with the database.
• #20: The selection of face models nowadays is done manually. Since there are new celebrities on tv every day, it’s still a tedious process to keep our database up to date.
• #21: As a result not all faces will be recognized and we do have gaps in our automatic annotations. As you see on this slide only 3 out of 4 persons are recoignised, simply because we didn’t have a face model for the 4th one yet. Currently we are working of a face model suggestion tool that identifies faces that are often on tv but are not in the database yet.
• #22: Let me give you an example of how well face recognition can work. In this case the king of the Netherlands is recognized even though he is sitting quite far away in his little car.
• #23: Another example of a political debate illustrates how useful speaker and face labeling can be, not only for producers who are searching for re-usable material. But for instance also for researchers and scientists who want to for instance analyze for instance how much airtime female and male politicians get.
• #24: Some impressive numbers on the amount of automatic labels we have today. But less impressive if you see that only 2% of our collections have 1 or more automatic label, while there are plenty of collection items that have little or no metadata. So there are some challenges we have to overcome: Not only do we have to balance between the users need for high accuracy and effects of quality thresholds on the quantity of automatically generated labels. In order to increase the quantity of labels we have to also constantly update our model databases. And we have to drastically increase our efforts if we want to upscale and apply the service retro, on a larger part of our collections. That almost brings me to final part of this presentation, but I will not end it before some examples of labeling errors.
• #25: O no, Donald Trump again. Yes, him again even if it’s not him. In this case Dutch journalist Frits Wester was labeled as Trump.
• #26: Another example: here Kim Jong Un is labeled with 99,9% certainty. But it’s not him, it’s a lookalike.
• #27: So who can say: is it the real Eppo walking around this conference and bein in this image. Or is it a fraud? Can you be more certain than the machine?
• #28: Less funny: A facial composite of a criminal suspect was labeled as Bernie Sanders. But luckily it scored under our threshold and therefor is disregarded as a label. Not even funny: this woman was labeled a Kim Jong Un, and with a score above the threshold. Ouch We solved this now by putting an extra high threshold on controversial persons. But ultimately this is not about avoiding controversy, maybe should be the opposite: being crystal clear about the fundamental bias behind such technologies
• #29: As you see in this example quite some black soccer players are recognized as actor Werner Kolf. Not coincidentally he is also black. Clearly we have some work to do here in terms of diversifying our databases, trainingsets and algoritms. We have a bias, which is discriminatory, for both gender and race And it reflects white privilege in what is in the media and is being archived. And we have to move past that after we acknowledge it and discuss it at conferences like this
• #30: Oops, a pic of me with a label according to Imagenet roulette It was part of an art exhibition which intentially wants to provocate. ImageNet is generally bad at recognizing people. It’s mostly an object recognition set, but it has a category for People that contains thousands of subcategories. Some of them are misogynistic or racist. ImageNet Roulette draws upon those categories to shed light on what happens when technical systems are trained on problematic training data.
• #31: To conclude: We have to find a balance between the need for high accuracy on the one hand and for quantity on the other And we should be aware that – if we allow mistakes or gaps – that some of them are the result of AI bias. In the era of big data and social media, and conference presentations shared online (do i share my kids’ faces and my face or not) it is almost impossible to remain anonymous. And lines between cultural preservation and commercialization of data are getting thinner and thinner.  Therefor privacy and ethics must play a part in our discussions on implementing AI technology for the archives. I hope to have encouraged you to start using these kind of techniques, without looking away from it’s difficult aspects