Competition shows not all exam questions can be correctly answered using AI

UNIVERSITY PARK, Pa. — Penn State’s Center for Socially Responsible Artificial Intelligence (CSRAI) announced the winners of its first-ever “Cheat-a-thon.” The virtual competition, held March 3 through April 6, invited faculty and students at universities across the United States to test the capabilities and limitations of using generative artificial intelligence (AI) to answer college-level questions. 

In the first stage, university faculty submitted questions from their exams or assignments that they thought would be difficult for students to answer using generative AI tools. Thirty-seven faculty from 10 institutions submitted 66 questions across 18 disciplines, including accounting, criminal justice, humanities, physics and engineering. 

In the second stage, 33 university students submitted 451 answers — created exclusively with generative AI tools and without the help of other internet resources — to the faculty-submitted questions. All questions received at least one answer. 

Five $1,000 prizes were awarded to faculty whose questions earned the lowest average grade by students, meaning they were the most difficult to answer correctly using only generative AI tools. Faculty winners are: 

• Vikash V. Gayah, professor of civil and environmental engineering, Penn State University Park

• Eric Hudson, associate professor of physics, Penn State University Park

• Ihab Ragai, professor of engineering, mechanical engineering technology, Penn State Behrend

• Nikolas Siapoutis, teaching assistant professor of statistics, University of Pittsburgh 

• Nasibeh Zohrabi, assistant professor of engineering, Penn State Brandywine

Student answers were graded based on faculty-provided rubrics. Winners were determined by the total number of correct answers submitted. All student winners were from Penn State. 

1st place — $1,000

• Kyle Ketterer, undergraduate student, College of Engineering: scored 100% in computer science, humanities, mathematics, media studies, Spanish and statistics

• Bryan Shabroski, undergraduate student, College of Engineering: scored 100% in accounting, computer science, humanities, mathematics, Spanish and statistics

2nd place — $750 

• Ayden Herold, undergraduate student, College of Information Sciences and Technology (IST) 

• Asa Reynolds, integrated undergraduate-graduate student, College of IST 

3rd place — $500

• David Zhu, undergraduate student, Division of Undergraduate Studies 

Consolation — $200 

• Maxwell Hahn, graduate student, Eberly College of Science 

• Matthew Thomas, undergraduate student, College of Engineering 

• Bennett Whitacre, undergraduate student, College of Engineering 

• Wahid Uz Zaman, doctoral student, College of Engineering  

• Shaoqing Zhang, doctoral student, Eberly College of Science 

When ChatGPT and other large-language models began to capture public attention in 2023, the threat it posed to academic integrity was the biggest concern for many in higher education, according to S. Shyam Sundar, CSRAI director and James P. Jimirro Professor of Media Effects in the Donald P. Bellisario College of Communications. 

“Many if not most faculty worry that students would use AI to cheat on their exams and assignments, thus posing a huge challenge to educators,” Sundar said. “The Cheat-a-thon was created to help us better understand the nature of the problem and equip faculty with the knowledge to design better exams and assignments in the future.” 

The challenge is ongoing, according to Amulya Yadav, associate professor in the College of IST and CSRAI associate director of programs.  

“Whether people in higher education like it or not, generative AI will continue to pose a serious challenge to maintaining academic integrity,” Yadav said. “Instead of expecting students not to use ChatGPT, the Cheat-a-thon was an attempt to induce faculty members to adapt their curricula in a way that it remains relevant in this ChatGPT fueled world.” 

Large-language models like ChatGPT are being used more frequently by both students and instructors. While this helps facilitate learning, it also hinders instructors’ ability to assess student understanding and outcomes. Now, many instructors are adapting their exams to account for student use of these tools. 

“I have started to focus on developing homework questions that require students to explain their reasoning — including stating and justifying assumptions — to force them to demonstrate their own understanding,” said faculty prize winner Gayah, professor of civil engineering and interim director of Penn State’s Larson Transportation Institute. “Students can use large language models (LLMs) to assist, just like they could in real life scenarios, but the goal is to get them to think carefully about their response. However, I expect that LLMs will catch up and require us to push even further in the very near future.” 

Among the submitted questions, students were challenged to compare different types of boosting algorithms, create a promotional webpage for a new television show, address questions faced by an expert witness in a trial and solve complex math problems. To tackle the questions, many students translated images to descriptive text so LLMs could respond more accurately. Some compared responses across multiple LLMs while others solved the problem first and then used generative AI to confirm their answer. 

“Through this experience, I witnessed firsthand just how powerful large language models have become and just how underestimated they still are,” said Ketterer, a student prize winner. “Yes, they have flaws, but with the right prompts and strategies, those flaws can be navigated. I learned to work with the model, not just rely on it.” 

Bonam Mingole, a doctoral student in the College of IST and CSRAI student affiliate who organized the event, agreed. 

"Faculty submitted tricky, AI-resistant questions, while students pushed the boundaries of prompt engineering with impressive creativity,” she said. “Cheat-a-thon results underscore a key question for higher education to confront when it comes to exams and assignments: What are we really assessing, and how do we prepare students for a world where AI is always part of the equation?” 

The Center for Socially Responsible Artificial Intelligence, which launched in 2020, promotes high-impact, transformative AI research and development, while encouraging the consideration of social and ethical implications in all such efforts. It supports a broad range of activities from foundational research to the application of AI to all areas of human endeavor.