Penn State Harrisburg incorporates ChatGPT into programming competition

MIDDLETOWN, Pa. — Incorporating ChatGPT and other generative artificial intelligence (AI) systems into a programming competition helped Penn State Harrisburg broaden access to and attract a diverse field of more than 130 student participants.

The competition was hosted by the School of Business Administration and the School of Science, Engineering, and Technology. Organizers decided to incorporate ChatGPT — an AI-driven chatbot — in order to reach more students and harness interest in a new technology.

When the ChatGPT prototype was launched in late November 2022, Jeremy Blum, associate professor of computer science at Penn State Harrisburg, was fresh off helping organize a global programming competition. Programming competitions typically pose a series of algorithmic puzzles that participants need to write a computer program to solve, he said.

The launch of the AI tool prompted the organizers to have “something like an existential crisis,” he said, because users could simply ask ChatGPT to write the program for them.

“How do you organize a global contest when you have access to these tools?” Blum asked.

So, when second-year Penn State Harrisburg student Sarah Shaub, president of the campus Women in Tech club, approached Blum this spring about hosting a programming competition on campus, he suggested ChatGPT be part of the contest.

Shaub said she immediately liked the idea because she thought it would even things out between computer science students and others. Shaub is a cybersecurity analytics and operations major, which only incorporates a little bit of programming, for example.

Barriers to access and participation for women and underrepresented minority students are cited in competitive programming literature, organizers said.

“We found that ChatGPT leveled the playing field for programmers versus non-programmers and broadened access and participation of diverse students in the competition,” said Roderick Lee, associate professor of information systems. “In fact, the 52 teams represented the richness of diversity that the college offers.”

Developing the competition

Blum enlisted the help of two computer science graduate students, Devang Jayachandran and Pranit Maldikar, to develop the problem set for the competition. They had already worked with Blum on a conference paper where they examined ChatGPT and what it could do in the context of programming contests.

“We had a pretty good sense of its capabilities — what kinds of problems it could solve, what kind it couldn’t completely solve,” Blum said. “We worked to come up with a problem set that would require the students to interact with ChatGPT. So, ChatGPT would certainly help get them started but wouldn’t solve completely for most of the problems.”

Shaub promoted the competition around campus and visited classes to make a quick pitch. She’d first start by asking how many students had used ChatGPT — and she found most students raised their hands.

The organizers hoped for 50 competitors, but around 135 students participated.

“While the turnout for the event was beyond anything we had hoped for, the real measure of success for me was to see how much students’ confidence grew by the end of the competition, which can be attributed to Dr. Blum and his vision to use ChatGPT to create an unconventional competition,” said Kelsey Kirk, faculty adviser for the Women in Tech organization.

Blum said the novelty of ChatGPT increased interest, both among students and faculty. He noted that some political science students participated because their professor thought it would be good to expose them to it.

“These tools are going to have sort of society-wide impacts,” Blum said.

Jayachandran noted that programming competitions typically have a high bar for entry — one has to know a lot of programming to participate.

“Our contest allowed even amateurs to be able to compete because the base template for the code itself would most often be generated by ChatGPT,” he said, noting that this approach made the most sense as AI-tools evolve. “You can’t really take ChatGPT away in a lot of cases. It makes more sense to embrace the existence of it and kind of work along with it.”

Maldikar said ChatGPT still doesn’t eliminate the need to understand the underlying question being asked.

“If you are solving a question, you need to understand what the question is,” he said. “If you simply copy and paste, in my experience, that won’t work. ChatGPT can get you started and be a really helpful tool to start the problem — but you need to understand the problem.”

He added that students could even use ChatGPT to learn about the concepts used in a problem.

Blum said all the teams earned some points, most of them solved multiple problems and they were engaged throughout.

“Given that we had a wide range of students participating — most brand new to competitive programming — that says a lot about the scaffolding these tools can provide and, ultimately, how they can broaden participation [in programming competitions],” Blum said.

Shaub expressed gratitude for support from Gretta Kellogg and Soundar Kumara from Penn State’s Center for Applications of Artificial Intelligence and Machine Learning to Industry, as well as from Harrisburg LaunchBox for a financial contribution.

A necessary tool to learn

Mark Peschel, a computer science major and one of the first-place winners at the competition, said he was surprised the first time he played around with ChatGPT, finding it impressive.

ChatGPT can be treated like a “much more clever Google,” he said, and it makes it easier to ask questions that might be more difficult to look up using traditional approaches.

For example, he was writing code for a website and could not figure out why it wasn’t working. Traditional web searching wasn’t helping him find the answer. Querying ChatGPT led him to a better search term, which then helped him get better results.

“Machine learning and AI are going to be the future,” Peschel said. “The use of these tools is going to be important to anyone in that career field.”

Blum said there are estimates that 20% to 30% of computer code is currently written by generative AI — and that is expected to grow. As such, Blum said, it’s important to think about how computing education needs to change to prepare students for a transforming workforce.

“These tools are going to provide tremendous productivity boosts to [students and workers], but they really need to know how to use them effectively, [including] the limitations,” he said, adding that the ability to analyze and review code becomes even more essential when the code is created by ChatGPT or other AI. “I’m excited to think about how we incorporate these tools into the curriculum. I think it’s critical that we do so.”

Winners of the programming competition at Penn State Harrisburg were:

First place:

• Mark Peschel, computer science
• Tyler Lindsay, computer science

Second place:

• Mohamud Abubakar, information sciences and technology
• Abemelek Badebo, computer science
• Dave Nyaga, information sciences and technology

Third place:

• Bardan Phuyel, computer science
• Joshua Carson, computer science
• Maliha Doria, computer science
• Siri Hegde, computer science