🤖 AI

Homoglyph-based Adversarial Perturbation of Introductory Computer Science Theory Problems

This paper proposes and evaluates a method using homoglyph-based adversarial perturbations to modify introductory computer science theory problems without altering their semantic meaning, aiming to prevent students from relying on AI tools to solve homework assignments.

Original authors: Aidan Alexander, Chitrangada Juneja, Napaluck Tontrasathien, Miro Vanek, Reyan Ahmed, Saumya Debray, Sazzadur Rahaman

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Aidan Alexander, Chitrangada Juneja, Napaluck Tontrasathien, Miro Vanek, Reyan Ahmed, Saumya Debray, Sazzadur Rahaman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to give a math homework assignment to your students. You want them to think hard and solve the problems themselves. But lately, students have been using super-smart AI assistants (like ChatGPT or Gemini) to do the work for them. These AI tools are so good that they can often answer questions instantly, even if the questions are slightly tricky.

The authors of this paper came up with a clever trick to stop this "lazy student" behavior. They call it Homoglyph-based Adversarial Perturbation. That's a fancy way of saying: "Changing a few letters in a question so that humans can still read it, but the AI gets confused and gives the wrong answer."

Here is how their method works, broken down into simple steps:

1. The Problem: The AI is Too Good at "Guessing"

The researchers noticed that these AI tools are like students who have memorized the entire textbook. If you ask them a classic math problem (like "Prove that a rational number times an irrational number is irrational"), they answer instantly because they've seen that exact question a million times in their training data.

Even if you try to mess up the question a little bit—like removing a word or changing a number—the AI is so smart it just "fills in the blanks" based on what it remembers. It's like if you asked a friend who memorized a recipe, "How do you make a cake with... uh... flour, sugar, and... [blank]?" They would just say "eggs" without you even telling them, because they know the recipe by heart.

2. The Solution: The "Trojan Horse" Trick

The authors realized they couldn't just scramble the text; the AI would fix it. Instead, they used a two-step strategy:

Step A: Change the Recipe (Slightly). First, they take a standard question and tweak it just enough so it's not in the AI's memory bank anymore.
- Example: Instead of asking about "a rational number," they might say, "Let's call the number 7x."
- Why? This makes the question unique. The AI hasn't seen "7x" in this specific context before, so it can't just pull the answer from its memory.
Step B: The Visual Illusion (Homoglyphs). Next, they use homoglyphs. These are characters that look almost exactly like normal letters or numbers but are actually different symbols from other languages or fonts.
- Analogy: Imagine the number 7. Now imagine a symbol that looks exactly like a 7 to your eyes, but to a computer, it's a completely different character (like a weird symbol from a different alphabet).
- They swap a normal number (like 7) with this "fake 7."

3. The Result: The AI Gets Fooled

When the AI sees this "fake 7," it gets confused. It doesn't recognize the symbol, so it panics and tries to guess what it should be. Because the AI is biased toward the original textbook questions it memorized, it ignores the "fake 7" and just assumes you meant the original "7" (or sometimes it just deletes it).

The Human Experience: A student looks at the paper and sees "7x." They read it correctly and solve the math problem.
The AI Experience: The AI sees a weird symbol it doesn't understand. It guesses wrong, ignores the math, and gives a completely incorrect answer.

4. How Many Changes Are Needed?

The best part is that you don't need to rewrite the whole question. The researchers found that changing just one or two characters is usually enough to break the AI.

If you change too many things, the AI might actually figure out the pattern.
But if you change just one tiny thing (like swapping a "6" for a look-alike "6"), the AI trips over it, while the human student sails right through.

5. A Tool for Teachers

The authors didn't just write a paper; they built a simple interactive tool.

Teachers can upload their homework questions.
They can click on a number or letter they want to change.
The tool shows them a list of "look-alike" characters (homoglyphs).
They pick one, and the tool swaps it in.
Now, the homework is safe from AI cheating, but still easy for students to read.

Summary

Think of this method as putting a visual camouflage on a homework assignment. To the human eye, the assignment looks normal. But to the AI, which relies on recognizing specific patterns, the assignment is now a puzzle it can't solve. It forces the student to actually do the work, rather than letting a robot do it for them.

Technical Summary: Homoglyph-based Adversarial Perturbation of Introductory Computer Science Theory Problems

Problem Statement
The widespread adoption of Large Language Models (LLMs) such as ChatGPT, Gemini, and Claude poses a significant challenge to academic integrity, particularly in introductory Computer Science and Discrete Mathematics courses. Students may utilize these tools to complete assignments without engaging with the material, thereby failing to meet course learning objectives. Current enforcement strategies, such as detecting AI-generated content, are unreliable and prone to false accusations. Furthermore, standard adversarial approaches often fail against modern LLMs because these models are robust to simple character perturbations; they frequently "repair" modified prompts by relying on memorized patterns from their training data, which often includes widely used textbook problems.

Methodology
The authors propose a four-step method to create adversarial perturbations that render assignment prompts ineffective for LLMs while remaining visually identical to humans. The approach operates under the "lazy student model," assuming students copy-paste prompts directly into LLM interfaces.

Identification of Bias: The method begins by identifying parts of a question where the LLM exhibits a strong bias toward the original training data. Even if critical adjectives or numbers are removed, the model often hallucinates the missing information based on its familiarity with the source material (e.g., inserting "rational" into a prompt about "nonzero numbers" because the original textbook problem contained it).
Semantic Modification: If the original question lacks specific numeric characters or mathematical expressions, the method modifies the prompt to introduce them without altering the semantic meaning or the difficulty for a human student. For instance, a general statement about irrational numbers is rephrased to include specific variables (e.g., $x$ and $y$ ) or coefficients (e.g., $7x$ ). This creates a variation of the problem that is less likely to exist verbatim in the model's training dataset.
Homoglyph Selection: The authors select specific homoglyphs (characters that look similar to standard alphanumeric characters but have different Unicode code points) that are easily readable by humans but unrecognized by LLMs. The selection process involves testing whether the LLM fails to identify the homoglyph as the intended character.
Perturbation Application: The selected homoglyphs are used to replace key characters (typically numeric coefficients or symbols involved in arithmetic) in the modified prompt. This disrupts the LLM's tokenization process. Because the model does not recognize the homoglyph, it cannot map it to the correct token. Consequently, the model either ignores the perturbed character or substitutes it with a token it is biased toward (often the original character from the training data), leading to an incorrect solution.

Key Contributions

Adversarial Perturbation Strategy: A novel approach that combines semantic modification with homoglyph-based attacks to bypass the robustness of modern LLMs against simple text alterations.
Exploitation of Model Bias: The method leverages the tendency of LLMs to "repair" prompts based on memorized training data, turning this feature against the model by introducing a modified problem structure that the model attempts to "correct" incorrectly.
Interactive Tool: The development of a user-friendly tool that allows instructors to upload homoglyph files, select characters, and generate perturbed questions efficiently.
Empirical Validation: A systematic evaluation using 69 text-based questions from a standard Discrete Mathematics curriculum (derived from Rosen's Discrete Mathematics and Its Applications).

Experimental Results
The study tested the method on 69 purely text-based homework questions using ChatGPT and Gemini.

Effectiveness: The method successfully fooled both models on all tested questions.
Efficiency: The perturbations required were minimal. On average, only 1.54 attempts were needed for ChatGPT and 1.67 for Gemini to find a successful perturbation.
Character Count: The average number of characters perturbed per question was approximately 3.09 for ChatGPT and 3.01 for Gemini. In many cases, perturbing a single character (e.g., changing a numeric coefficient like "8" to a homoglyph of "6") was sufficient to cause the model to fail.
Observations on Perturbation Density: The authors noted that excessive perturbation (changing multiple characters) could sometimes reduce effectiveness, as the model might interpret the homoglyphs as a new common factor rather than failing to recognize them. Conversely, minimal perturbation often yielded better results.
Limitations: The method is most effective on numeric characters involved in arithmetic. Perturbing characters that serve purely as symbolic placeholders (e.g., "0" and "1" in a formal language definition like $\{0^n1^{2n}\}$ ) was found to be ineffective.

Significance and Claims
The paper claims that this method offers a practical, non-punitive deterrent for AI-assisted cheating in theoretical computer science courses. By modifying prompts so that they are visually indistinguishable to humans but semantically confusing to LLMs, instructors can ensure that students must engage with the material to solve the problems. The authors emphasize that this approach broadens the options for the "disallow AI-generated content" strategy, providing a proactive defense that does not rely on unreliable detection algorithms. The work is modest in scope, focusing specifically on introductory theory problems and demonstrating that even highly robust LLMs can be compromised by carefully selected, minimal perturbations.