Army looks to block data ‘poisoning’ in facial recognition, AI
0 likes92 views
The U.S. Army is addressing the threats of data poisoning in facial recognition systems caused by adversaries who subtly alter data to undermine AI algorithms. The Army is funding research to develop defensive software that can detect and mitigate these backdoor attacks, which could misclassify or disable AI models. The project faces challenges related to the size and quality of data used for training AI, which must balance the risks of adversarial attacks with the need for diverse and representative datasets.
Army looks to block data ‘poisoning’ in facial recognition, AI
- 1. DEFENSE Army looks to block data ‘poisoning’ in facial recognition, AI (Getty Images) SUBSCRIBE
- 2. Written by Jackson Barnett Feb 11, 2020 | FEDSCOOP The Army has many data problems. But when it comes to the data that underlies facial recognition, one sticks out: Enemies want to poison the well. Adversaries are becoming more sophisticated at providing “poisoned,” or subtly altered, data that will mistrain arti cial intelligence and machine learning algorithms. To try and safeguard facial recognition databases from these so-called SUBSCRIBE
- 3. backdoor attacks, the Army is funding research to build defensive software to mine through its databases. Since deep learning algorithms are only as good as the data they rely on, adversaries can use backdoor attacks to leave the Army with untrustworthy AI or even bake-in the ability to kill an algorithm when it sees a particular image, or “trigger.” “People tend to modify the input data very slightly so it is not so obvious to a human eye, but can fool the model,” said Helen Li, a Duke University faculty member whose research team received $60,000 from the Army Research Of ce for work on an AI database defensive software. Backdoors can be implanted into a database and labeled in a way that trains the algorithm to “break” when it comes across the image in the real world, Li said. For instance, researchers at New York University trained an autonomous car’s neural network so that when a stop sign had a yellow Post-it Note on it, the car classi ed it instead as a speed limit sign. An AI problem and an Army problem Data quality and security are challenges for AI developers who use databases larger than any human can comb through for anomalies. But the Army and other services face the added layer threats from adversaries seeking to disarm the U.S. military. MaryAnne Fields, program manager for intelligent systems at the Army Research Of ce, told FedScoop that countering backdoor attacks and data poisoning is a high priority for her. “The fact that you are using a large database is a two-way street,” Fields said. “It is an opportunity for the adversary to inject poison into the database.” The software Li’s team is developing with ARO funding is designed to detect potential backdoors in a database and then instruct the algorithm to unlearn connections it may have picked up from the bad data. The trigger challenge doesn’t only emanate from attackers. Models will misclassify novel images if they learn on a database without the right size and diversity of data points. Having too few images with too many of the same traits in the same label group could cause unintentional “natural triggers,” Fields said. For example, a photo of a man in a database labeled as “Frank” wearing a hat in every image may cause the SUBSCRIBE
- 4. algorithm to classify all men with hats as Frank, or miss the real Frank if he isn’t wearing a hat. “The Army does need to think differently about the type of data it is using,” Fields said. Using large databases forces the Army to make dif cult trade- offs. Increasing the number of images increases the chance for adversary attacks. Decrease the size and unintentional triggers formed from a monolithic database becomes a problem. “If you don’t have very much data to work with, these types of problems, and particularly the natural triggers, become more prevalent,” Fields said. “It is important to defend.” Scaling the solution The test batch the Duke researches were given was small — 12,000 images of faces with 10 images per classi cation. Some facial recognition databases exceed half a billion images. But Li pointed to a different challenge: image resolution. As images increase in quality, the complexity in searching for the triggers increases “exponentially,” Li said. That spike in dif culty is in part due to triggers that can be only a few pixels large, according to research published in 2014. “It is easy to produce images that are completely unrecognizable to humans, but that state-of-the-art (deep neural networks) believe to be recognizable objects with 99.99% con dence,” the paper says. Despite this, Fields expressed con dence in the project, calling the team’s solution “very scalable.” -In this Story- Army, arti cial intelligence (AI), Duke University, facial recognition RELATED NEWS SUBSCRIBE