UNIVERSITY PARK, Pa. — Though they play a critical role in everything from the development of self-driving cars and voice-activated search to how medical providers make health-related decisions, deep learning techniques are rarely used in cybersecurity. Now, thanks to a new technique developed by an international team of researchers, deep learning could be poised for another breakthrough.

Deep learning is a segment of artificial intelligence that focuses on algorithms that can learn the characteristics of text, images or sound from annotated examples provided to it. The team’s technique, named LEMNA, could help security analysts and machine-learning developers to establish trust in deep learning models by correctly identifying, explaining and correcting errors that the models make.

“The model can tell you its decision, but it cannot tell you why it made [the one it did],” said Wenbo Guo, a doctoral student in the College of Information Sciences and Technology at Penn State and one of the researchers on the project. “The long-term importance of LEMNA is that it’s a useful tool. Using LEMNA we can build trust in the deep learning model, and if we can explain why the error occurred we can probably patch the error.”

Guo said that there is existing work that explains classification errors in deep learning models, but most of those techniques focus on image, vision or text applications — such as incorrectly identifying a dog in a picture as a cat. His team is one of the first to create a technique and publish a paper that explores how to identify and explain classification errors made by deep learning models for security applications.

“It’s the start of the field,” Guo said.

The researchers tested their technique by applying it to two security applications. First, they tested LEMNA through binary code reverse-engineering, a crucial step in cybersecurity that allows software to be reconstructed if the original source code is unavailable. Then, they applied LEMNA to PDF malware classification.

"In malware detection, if you make a false positive, such as detecting a benign software as malware, it could cause a serious problem,” he said. “Also, if the model makes the mistake of a false negative and didn’t detect a malware, then someone could hack into your system, costing you millions of dollars.”

“So in this case, even though deep learning can get 99.9 percent accuracy, there’s still that 0.1 percent error,” he continued. “What we concentrate on is that 0.1 percent and why the model made that error.”

LEMNA could prove invaluable to security analysts, who often must rely on educated guesses to determine whether a model is functioning properly. With LEMNA, analysts have another tool to justify their decisions and help them better understand why errors exist and how to patch them.

Guo hopes that with LEMNA, the application of deep learning systems can be extended into critical fields such as finance and medical imaging.

“In the future, when people in these fields look into a deep learning model, they can see that there is a tool they can use to explain the model, and [know] that they can confidently use the model,” he said.

“I think it is a very promising result in [the field of] deep learning,” he continued. “Humans can only get a certain level of accuracy in certain applications. Deep learning just beats humans in several applications today ranging from playing the board game Go to identifying objects in images.”

LEMNA research team earns outstanding paper award at prestigious computer security conference

The team’s research was recently recognized with the outstanding paper award at the 2018 Association for Computing Machinery Conference on Computer and Communications Security for their paper, one of the most competitive and prestigious conferences in the field of computer security.

“Getting a paper accepted is very hard in this venue,” said Prasenjit Mitra, associate dean for research in Penn State’s College of IST. “Getting the best paper award in CCS is a major achievement. Crème de la crème.”

Other collaborators on the project include the College of IST’s Dongliang Mu, research assistant, and Xinyu Xing, assistant professor; Jun Xu, Stephen Institute of Technology and former doctoral student in the College of IST; Purui Su, Chinese Academy of Sciences; and Gang Wang, Virginia Institute of Technology.

“At Penn State and especially in the College of IST we encourage collaborations. That’s how we find these problems,” Guo said. “We have people from human-computer interaction, big data, data mining, machine learning, social sciences and security. Since we have this diverse faculty it is easy for us to do collaborations, and this research is a very good demonstration of that.”

According to Guo, more than 800 papers were submitted to the conference and only 162 were accepted. Of those, nine were nominated for best paper, for which there were two winners based on unanimous votes by the judges. The other best paper award was received by a research group from the Department of Computer Science and Engineering in Penn State’s College of Engineering.

“[To have two research teams from Penn State win outstanding paper awards] demonstrates that Penn State’s research in the field of security is at the top of the world,” said Guo.

“This makes us very proud that we’re from Penn State,” he continued. “Faculty and students who are working in this field should be very proud.”

The team’s project, titled LEMNA: Explaining Deep Learning based Security Applications, was supported in part by grants from the National Security Foundation and from the donation of a graphics processing unit (GPU) from NVIDIA Corporation.