UNIVERSITY PARK, Pa. — Though they play a critical role in everything from the development of self-driving cars and voice-activated search to how medical providers make health-related decisions, deep learning techniques are rarely used in cybersecurity. Now, thanks to a new technique developed by an international team of researchers, deep learning could be poised for another breakthrough.
Deep learning is a segment of artificial intelligence that focuses on algorithms that can learn the characteristics of text, images or sound from annotated examples provided to it. The team’s technique, named LEMNA, could help security analysts and machine-learning developers to establish trust in deep learning models by correctly identifying, explaining and correcting errors that the models make.
“The model can tell you its decision, but it cannot tell you why it made [the one it did],” said Wenbo Guo, a doctoral student in the College of Information Sciences and Technology at Penn State and one of the researchers on the project. “The long-term importance of LEMNA is that it’s a useful tool. Using LEMNA we can build trust in the deep learning model, and if we can explain why the error occurred we can probably patch the error.”
Guo said that there is existing work that explains classification errors in deep learning models, but most of those techniques focus on image, vision or text applications — such as incorrectly identifying a dog in a picture as a cat. His team is one of the first to create a technique and publish a paper that explores how to identify and explain classification errors made by deep learning models for security applications.
“It’s the start of the field,” Guo said.
The researchers tested their technique by applying it to two security applications. First, they tested LEMNA through binary code reverse-engineering, a crucial step in cybersecurity that allows software to be reconstructed if the original source code is unavailable. Then, they applied LEMNA to PDF malware classification.
"In malware detection, if you make a false positive, such as detecting a benign software as malware, it could cause a serious problem,” he said. “Also, if the model makes the mistake of a false negative and didn’t detect a malware, then someone could hack into your system, costing you millions of dollars.”
“So in this case, even though deep learning can get 99.9 percent accuracy, there’s still that 0.1 percent error,” he continued. “What we concentrate on is that 0.1 percent and why the model made that error.”