🤖 machine learning

Categorical Prior Lock-in: Why In-Context Learning Fails for Structured Data

This paper reveals that while in-context learning can improve numerical fidelity in structured data generation, it fundamentally fails to adapt to rare categorical classes due to "categorical prior lock-in," a limitation that parameter-efficient fine-tuning can overcome but at the cost of increased memorization risks and potential output instability.

Original authors: Antonio Pelusi, Stefano Braghin, Alberto Trombetta

Published 2026-06-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Antonio Pelusi, Stefano Braghin, Alberto Trombetta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Copycat" vs. The "Student"

Imagine you have a very smart, well-read robot (a Large Language Model or LLM) that has read almost everything on the internet. You want it to generate fake credit card transaction records that look real, but with specific details (like specific job titles or locations) that are different from what it usually sees.

The researchers tested two ways to teach this robot:

In-Context Learning (ICL): You show the robot a few examples right before it starts writing (like a cheat sheet). You don't change the robot's brain; you just give it context.
Fine-Tuning (LoRA): You actually retrain the robot's brain slightly on your specific data so it learns the new patterns deeply.

The paper's main discovery is that showing examples (ICL) works great for numbers, but fails completely for specific categories, while retraining the brain (Fine-Tuning) works better but comes with a privacy risk.

The Problem: "Categorical Prior Lock-in"

The authors coined a fancy term for a simple problem: Categorical Prior Lock-in.

The Analogy:
Imagine the robot is a chef who has cooked millions of meals. In its training, it has only ever seen "Pizza," "Burger," and "Salad." It has a strong "prior" belief (a habit) that these are the only foods that exist.

Now, you give the chef a menu for a new restaurant that specializes in rare, exotic dishes like "Jellyfish" or "Frog Legs."

The ICL Approach (The Cheat Sheet): You hand the chef a note saying, "Today we are serving Jellyfish."
- What happens? The chef looks at the note, nods, but then accidentally writes "Burger" on the order ticket. Why? Because the chef's brain is so used to "Burger" that the short note isn't strong enough to override that deep habit. The chef can't "unlearn" the idea that Burgers are the main dish just by reading a few examples.
The Fine-Tuning Approach (Retraining): You take the chef into the kitchen for a week and make them practice cooking Jellyfish until it becomes second nature.
- What happens? The chef now genuinely knows how to cook Jellyfish and stops defaulting to Burgers.

The Paper's Finding:
When generating structured data (like tables with columns for "Job," "State," or "Gender"):

Numbers (Amount, Time): The robot is good at these. If you show it a few examples of high prices, it learns to generate high prices. It's like the chef learning to adjust the saltiness of a dish.
Categories (Job, State): The robot fails here. If the real data has rare jobs (like "Artisan Cheese Maker"), the robot will ignore them and keep generating common jobs (like "Teacher" or "Engineer") because those are what it learned during its original training. No matter how many examples you show it in the prompt, it gets "locked in" to its old habits.

The Results: What Happened in the Experiments?

The researchers tested this on two popular robot brains (Qwen and Mistral) using credit card data.

1. The "Cheat Sheet" (ICL) Results:

Numbers: Improved as they added more examples.
Categories: Stuck in a rut. Even with 10 examples, the robot kept generating the same common jobs it knew from the internet, completely missing the rare ones in the dataset.
Fraud Detection: The robot never generated a single fraudulent transaction when using the cheat sheet method. It was too scared to break its "normal" patterns.
Privacy: This method was very safe. The fake data looked nothing like the real data, so no secrets were leaked.

2. The "Retraining" (LoRA) Results:

Numbers & Categories: Much better! The robot finally learned the rare jobs and the specific geographic patterns.
The Catch (Privacy): Because the robot actually learned the data, the fake records it generated were suspiciously close to the real records. It was like the chef memorizing the exact recipe of a specific customer's order. This creates a risk that the robot might accidentally "leak" real private information.
The Catch (Stability): For one of the robots (Mistral), retraining went wrong. The robot got confused and started spitting out gibberish or breaking the format entirely. It was like the chef, after trying to learn a new cuisine, forgot how to hold a knife.

The Trade-Off: Safety vs. Accuracy

The paper concludes that you have to choose between two paths, and there is no perfect middle ground:

Path A: The Cheat Sheet (ICL)
- Pros: Safe, private, and doesn't break the robot.
- Cons: The data looks fake in important ways. It misses rare categories and fails to capture complex relationships (like how a specific city correlates with a specific job). It's useless for tasks like fraud detection because it won't generate the "weird" (fraudulent) data needed to train a detector.
Path B: Retraining (Fine-Tuning)
- Pros: The data looks real. It captures rare categories and complex patterns.
- Cons: It risks leaking private information (memorization). Also, on smaller robots, it can sometimes break the robot's ability to follow instructions.

The Bottom Line

If you need to generate structured data (like a database of transactions) that matches a specific, unique distribution:

Don't rely on just showing examples. The robot is too "stubborn" to change its mind about categories just by reading a few lines.
You likely need to retrain the model. But be careful: retraining makes the data more accurate but less private, and it might make the robot unstable if the model is too small.

The paper essentially says: In-Context Learning is a lightweight tool, but it cannot force a robot to unlearn its deep-seated habits about the world. To truly adapt to new data, you have to change the robot's brain, which brings its own set of risks.

Technical Summary: Categorical Prior Lock-in in Structured Data Generation

Problem Statement

Large Language Models (LLMs) are increasingly utilized as conditional generators for structured data, such as tabular records, relying on In-Context Learning (ICL) to adapt to new distributions without parameter updates. While ICL is favored for its lightweight nature and avoidance of fine-tuning costs, its efficacy in structured generation tasks remains under-evaluated regarding underlying distributional behavior.

This paper investigates the limits of ICL when applied to high-cardinality tabular data under distribution mismatch. The authors identify a structural failure mode termed categorical prior lock-in: the inability of ICL to update a model's prior over token distributions inherited from pre-training. Specifically, when conditioned on examples drawn from a target distribution that diverges significantly from the pre-training corpus (e.g., rare or domain-specific categorical labels), the model fails to approximate the target distribution, regardless of prompt design or the number of in-context examples provided.

Methodology

The study employs a controlled experimental setup using high-cardinality tabular data to stress-test ICL against parameter-efficient fine-tuning.

Dataset: A publicly available credit card transaction log containing seven numerical features (e.g., transaction amount, coordinates) and four categorical features (transaction category, gender, occupation, US state). The dataset is characterized by highly imbalanced categorical distributions (e.g., the job field has 494 distinct labels, with the most common representing only 0.75% of records) and a rare fraud class (0.58%).
Models: Two open-weight, instruction-tuned 7B-parameter decoder-only transformers were selected for local deployment scenarios: Qwen2.5-7B-Instruct and Mistral-7B-Instruct-v0.3.
Strategies: The authors compared three generation paradigms:
1. Zero-shot ICL: No in-context examples.
2. Few-shot ICL: 1, 5, or 10 in-context examples.
3. LoRA Fine-tuning: Parameter-efficient fine-tuning on 10% or 50% of the real data, followed by zero-shot generation.
Evaluation Metrics:
- Structural Reliability: Measured by the error rate (fraction of generation attempts failing JSON parsing or schema validation).
- Distributional Fidelity: Assessed via Total Variation Distance (TVD) for marginal distributions of both numerical and categorical features.
- Privacy/Memorization Risk: Quantified using the DCR Ratio (comparing distances between synthetic and real records) and exact duplication rates.
- Structural Dependencies: Evaluated via Pearson correlation matrices to assess inter-feature relationships (e.g., geographic coordinates).
- Minority Class Reproduction: Specifically analyzed the reproduction of the fraud class.

Key Contributions

Identification of Categorical Prior Lock-in: The paper characterizes a fundamental failure mode where ICL adheres to pre-training token priors, preventing the reproduction of rare or domain-specific categorical distributions.
Systematic Empirical Evaluation: The study provides a comprehensive comparison across models and prompting strategies, demonstrating that while ICL improves numerical fidelity with more examples, it hits a sharp ceiling on categorical distributions.
Trade-off Analysis: The work analyzes the trade-offs between ICL and fine-tuning regarding fidelity, structural consistency, and privacy risk, highlighting that fine-tuning offers superior distributional adaptation at the cost of memorization risk and potential model instability.

Results

Structural Reliability

Qwen2.5-7B: Exhibited high stability across all strategies. Error rates dropped from 1.17% (zero-shot) to near-zero with few-shot examples. LoRA fine-tuning maintained 0.00% error rates.
Mistral-7B: Showed acceptable but noisier ICL performance (0.89%–3.59% error rates). However, LoRA fine-tuning caused catastrophic degradation: a 41.04% error rate at 10% data exposure and complete generation failure (100% error rate) at 50% exposure, producing no valid records.

Distributional Fidelity

Numerical Features: ICL improved numerical fidelity as the number of examples increased (e.g., TVD for transaction amounts dropped significantly). LoRA achieved the lowest TVD values (often <0.10).
Categorical Features: ICL failed to approximate high-cardinality categorical distributions. For the job field, TVD remained near 1.0 across all ICL configurations for both models, indicating essentially no distributional overlap with real data. The models generated common pre-training occupations rather than the target dataset's distribution.
- LoRA fine-tuning on Qwen2.5-7B reduced job TVD to ~0.15, a substantial improvement but still imperfect.
- Mistral-7B LoRA 10% failed to close the gap on job (TVD 0.59) or state (TVD 0.25).
Inter-Feature Correlations: Zero-shot ICL collapsed strong correlations (e.g., between cardholder and merchant coordinates, $r>0.99$ in real data) to near-zero. Few-shot prompting partially reduced these errors, but only LoRA fine-tuning successfully restored the dependency structure.

Privacy and Memorization

ICL: All configurations yielded DCR Ratios well above 1.0, indicating synthetic records were distant from real training data, posing low memorization risk.
LoRA: All fine-tuned configurations fell below the 1.0 DCR threshold, suggesting synthetic records were closer to real records than real records were to each other. This indicates the emergence of memorization behavior, even at low data exposure levels (10%).

Minority Class Reproduction

Neither model produced any fraudulent transactions under any ICL configuration, representing a complete failure to reproduce the minority class. LoRA fine-tuning partially restored the fraud class (e.g., Qwen2.5-7B generated 0.12%–0.14% fraud), but this remained significantly below the real base rate of 0.58%.

Significance and Claims

The paper argues that categorical prior lock-in is a fundamental limitation of In-Context Learning in structured generation. The authors claim that ICL operates locally within the context window and cannot globally reshape the underlying token probability distribution required to approximate high-cardinality, domain-specific categorical variables.

Key implications drawn by the authors include:

ICL is not a universal substitute for learning: In settings requiring distributional adaptation (especially for high-cardinality categoricals), ICL is insufficient.
Fine-tuning is necessary but risky: While LoRA enables the global distributional adjustment that ICL cannot, it introduces measurable memorization risks and, in the case of smaller models like Mistral-7B, can destabilize the model's ability to follow structured output schemas.
Reframing ICL: The authors suggest ICL should be viewed as a form of conditional generation with inherent limitations determined by the model's pre-training prior, rather than a general-purpose few-shot learning mechanism for all structured tasks.

The study concludes that for structured generation tasks involving significant distribution shifts, parameter updates are necessary, but practitioners must carefully weigh the gains in fidelity against the risks of data leakage and model instability.