The U.S. spy community’s R&D agency is looking into how to keep AI software from unwittingly divulging information about sensitive training data and from secretly being trained to be malicious.
The Intelligence Advanced Research Projects Agency announced it will be holding an event next week for organizations potentially interested in participating in two artificial intelligence security programs.
One, called Secure Assured Intelligent Learning Systems (SAILS), looks to find ways to stop AI system users from learning too much about private data used to train the systems. Without such protections, malicious users might be able to find out information about training data in aggregate, such as “the average of an individual’s face used to train a facial identification model,” or even determine whether a particular person’s data was included in the training model, according to the agency.
“The goal is to provide a mechanism by which model creators can have confidence that their trained models will not inadvertently reveal sensitive information,” according to IARPA.
A second effort, called TrojAI, looks to spot Trojans, an allusion to the legend of the Trojan horse referring to hidden malicious behaviors. With AI, Trojans can be hidden not just within traditional programming code but also encoded within training data, causing programs to behave unexpectedly when they’re given particular cues.
“For example, an AI learning to distinguish traffic signs can be given just a few additional examples of stop signs with yellow squares on them, each labeled ‘speed limit sign,'” according to the agency. “If the AI were deployed in a self-driving car, an adversary could cause the AI to misidentify a stop sign as a speed limit sign just by putting a sticky note on it, potentially leading the car to run through the sign.”