💬 NLP

GSM-SEM: Benchmark and Framework for Generating Semantically Variant Augmentations

This paper introduces GSM-SEM, a stochastic framework that generates semantically diverse and factually altered benchmark variants to mitigate memorization bias in LLM evaluations, revealing significant performance drops in state-of-the-art models when tested on these dynamically regenerated datasets.

Original authors: Jyotika Singh, Fang Tu, Aziza Mirzadova, Amit Agarwal, Hitesh Laxmichand Patel, Sandip Ghoshal, Miguel Ballesteros, Yassine Benajiba, Weiyi Sun, Graham Horwood, Sujith Ravi, Dan Roth

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jyotika Singh, Fang Tu, Aziza Mirzadova, Amit Agarwal, Hitesh Laxmichand Patel, Sandip Ghoshal, Miguel Ballesteros, Yassine Benajiba, Weiyi Sun, Graham Horwood, Sujith Ravi, Dan Roth

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to test how good a student is at math. You give them a standard test, like the famous GSM8K (a collection of grade-school math word problems). If they get a perfect score, you might think, "Wow, this student is a math genius!"

But what if the student didn't actually learn math? What if they just memorized the answers to this specific test because they've seen it a thousand times before? They might get the answer right, but if you change the story slightly—say, swapping "apples" for "oranges" or changing the numbers—they might get confused and fail.

This is the problem the paper GSM-SEM is trying to solve.

The Problem: The "Cheat Sheet" Effect

Current AI models (like the ones powering chatbots) are getting incredibly good at benchmarks. But the authors suspect many of these high scores are a mirage. The models aren't necessarily reasoning better; they are just memorizing the specific questions and answers from the test sets they were trained on.

Existing ways to fix this involve "perturbations," which are like making small changes to the test:

Paraphrasing: Rewriting the sentence in different words.
Renaming: Changing "John" to "Jane."
Swapping numbers: Changing 5 to 6.

The paper argues these changes are too shallow. They are like changing the font on a cheat sheet; the student can still cheat because the underlying facts and logic remain exactly the same.

The Solution: GSM-SEM (The "Story Rewriter")

The authors introduce GSM-SEM, a new framework that acts like a creative writing coach for math problems. Instead of just swapping words, it rewrites the entire story while keeping the math answer exactly the same.

Here is the creative analogy:
Imagine a math problem is a recipe for a cake that yields 10 slices.

Old Method (Paraphrasing): You change the recipe title from "Chocolate Cake" to "Dark Cocoa Cake" and swap "flour" for "wheat flour." The recipe still works the same way, and the model just recognizes the pattern.
GSM-SEM Method: You completely change the story. Instead of baking a cake, the problem is now about mixing paint or calculating fuel for a rocket. The story is totally different, the objects are different, and the context is new. However, the math required to solve it (the numbers and the final answer) remains identical.

The model can no longer rely on memorizing the "cake recipe." It has to actually understand the logic of the new story to get the answer right.

How They Built It

The team built a system that:

Takes a math problem and its correct answer.
Asks AI models to invent a new story that leads to that same answer. (e.g., "If the answer is 15, write a story about a baker, a spaceship, or a video game that results in 15.")
Filters the results to ensure the new story is actually different from the original (not just a slight rewording) but still solvable.
Validates the new problems with human reviewers to make sure they make sense.

They applied this to three different sets of benchmarks, creating new versions called GSM8K-SEM, GSM-Symbolic-SEM, and GSM-Plus-SEM.

What They Found

They tested 14 of the smartest AI models available (including models from OpenAI, Google, and Meta) on these new "story-rewritten" tests.

The Results:

The "Genius" Models Stumbled: When the models faced these new, semantically different stories, their performance dropped significantly. On average, they got about 28% fewer questions right when the semantic changes were combined with other tough variations.
Memorization Exposed: The fact that the models failed so badly on the new stories proved that their previous high scores were largely due to memorization, not true reasoning.
The "Double Trouble" Effect: The models struggled the most when the story was changed and other elements (like numbers or logic structures) were also tweaked. This suggests that current AI is very brittle; it can handle one type of change, but not a combination of them.

The Takeaway

The paper concludes that GSM-SEM is a better way to test if an AI is actually "thinking" or just "reciting." By constantly generating fresh, unique stories that require the same math, it prevents models from cheating by memorization.

The authors also showed this works on other types of logic puzzles (not just math), proving that this "story-rewriting" approach is a powerful tool to see if AI is truly robust or just good at pattern matching.

In short: If you want to know if an AI is smart, don't ask it the same questions it's heard before. Ask it to tell you a new story that leads to the same answer. If it can't do that, it's not reasoning; it's just remembering.

Technical Summary: GSM-SEM

Problem Statement

Current benchmarks for evaluating mathematical reasoning in Large Language Models (LLMs), such as GSM8K, face significant limitations regarding robustness and generalization. High leaderboard scores often reflect memorization of static test sets or exploitation of dataset-specific artifacts rather than true reasoning capabilities. Existing robustness variants (e.g., paraphrasing, entity renaming, numerical substitutions) primarily apply surface-level perturbations that preserve the underlying factual backbone and relational structure of problems. Consequently, these variants often test invariance to surface form rather than adaptability to meaningfully different semantic conditions. Furthermore, static benchmark releases become targets for memorization over time, and generating fresh, comparable perturbations often requires costly re-annotation of ground-truth answers.

Methodology: The GSM-SEM Framework

The authors introduce GSM-SEM, a reusable, stochastic framework designed to generate semantically diverse benchmark variants while preserving the original ground-truth answer and calculation logic. The framework operates through a dual curation and validation pipeline:

Semantic Variant Generation: The framework employs two complementary augmentation strategies to generate new question variants ( $\tilde{q}$ ) from an original sample $(q, a)$ :
- Answer-Conditioned Generation: LLMs are prompted to reverse-engineer a new question that preserves the numerical computation implied by the answer $a$ but substantially alters the surface form and contextual facts (e.g., changing "bacteria infects" to "virus spreads").
- Question-Conditioned Generation: LLMs are prompted to keep all numerical values fixed while altering the scenario or theme (e.g., changing "plague infects" to "rumor spreads"), thereby increasing semantic diversity without changing the required arithmetic.
- A rule-based verifier ensures all numeric values in the generated question match the original.
Automated Quality Validation: A judge LLM, tuned via human review of 70 sampled variants, evaluates logical coherence and quality.
Similarity-Based Redundancy Pruning: Variants are filtered using cosine similarity based on term frequency vectors. Candidates with similarity $s(x, \tilde{x}) > 0.85$ to the base sample or existing retained variants are discarded to ensure semantic divergence.
Strictness Filtering: To control difficulty and avoid trivial variants, a strictness filter evaluates candidates using a held-out set of reference models ( $M_{hold}$ ). Variants where models show unanimous correctness are treated as potentially too easy or insufficiently perturbed. These are further pruned based on a semantic similarity acceptance window $[\alpha, \beta]$ . This allows for the creation of subsets with varying levels of challenge (e.g., "max" strictness filters out all unanimously correct samples).
Human Validation: The final datasets undergo a rigorous two-stage human review process to ensure logical coherence and answer alignment, resulting in fully human-validated datasets.

Key Contributions

GSM-SEM Framework: A stochastic, reusable pipeline for generating semantically diverse benchmark variants that maintain original calculations and answers, enabling re-runnable evaluation without re-annotation.
Human-Validated Datasets: Public release of three fully validated datasets: GSM8K-SEM, GSM-Symbolic-SEM, and GSM-Plus-SEM. These datasets demonstrate higher semantic variance than prior approaches (GSM-Symbolic, GSM-Plus, and paraphrased versions).
Comprehensive Evaluation: Evaluation of 14 State-of-the-Art (SOTA) LLMs across model families, revealing consistent performance degradation in SEM variants compared to original benchmarks.
Cross-Domain Applicability: Demonstration of the framework's utility on non-GSM datasets, including BigBench-Hard, LogicBench, and NLR-BIRD.

Results

Performance Degradation: Across 14 SOTA models, GSM-SEM variants consistently underperform relative to their baseline benchmarks. In the "med" strictness configuration, the average performance drop rate (PDR) was approximately 28% for GSM-Plus-SEM, the most challenging configuration.
Compounded Perturbations: Combining semantic perturbations (GSM-SEM) with existing symbolic or "plus" variations (GSM-Symbolic-SEM, GSM-Plus-SEM) amplifies performance declines. For instance, GSM-Plus-SEM showed an average accuracy drop of 13.53% across models, significantly higher than GSM-Plus alone.
Semantic Sensitivity: Analysis reveals that model accuracy generally improves with higher semantic similarity to the original dataset. Even when calculations remain unchanged, substantial semantic shifts lead to larger performance drops, indicating current models are sensitive to semantic context rather than just symbolic manipulation.
Failure Modes: Manual analysis identified key failure modes in SEM variants, including:
- Increased cognitive load due to added indirection.
- Unit/currency cues triggering scale errors (e.g., misinterpreting cents vs. dollars).
- Destabilized state tracking in reference-style rewrites.
- Template misretrieval under minor semantic shifts.
Out-of-Domain Transfer: The framework successfully generated variants for non-mathematical datasets. Performance drops were observed in older datasets (BigBench-Hard, 2022) likely due to memorization, while newer datasets (NLR-BIRD, 2025) showed minimal drops, suggesting the framework effectively isolates reasoning capabilities from memorization.

Significance and Claims

The paper posits that GSM-SEM addresses the critical gap between surface-level robustness testing and true semantic adaptability. By generating fresh, semantically diverse variants on each run while preserving ground-truth answers, the framework reduces reliance on static public benchmarks and lowers evaluation bias caused by memorization.

The authors claim that current SOTA models, despite their advanced reasoning capabilities, remain sensitive to semantic shifts that alter the narrative context of a problem, even when the underlying math is identical. This suggests that high benchmark scores may not fully capture robust generalization. The framework provides a mechanism for more rigorous, re-runnable evaluation that can be adapted to various datasets beyond GSM-style math problems, offering a path toward more reliable assessment of LLM reasoning.