Asymmetric Idiosyncrasies in Multimodal Models

This paper reveals that while captioning models embed distinctive stylistic signatures detectable with near-perfect accuracy, these idiosyncrasies largely vanish in the images generated by text-to-image models due to a failure to preserve key textual variations, offering a new framework to quantify both captioning styles and prompt-following capabilities.

The Gist

The Big Idea: The "Ghost" in the Machine

Imagine you have three different chefs (let's call them Chef A, Chef B, and Chef C). They are all given the exact same photo of a banana and asked to write a recipe description for it.

• Chef A writes a poetic, atmospheric description focusing on the lighting.
• Chef B writes a technical, camera-focused description mentioning angles and resolution.
• Chef C writes a concise, object-focused description listing the main items.

If you gave these three descriptions to a food critic, they could instantly tell you which chef wrote which one. They have distinct "voices" or fingerprints.

Now, imagine you take those three descriptions and hand them to a Magic Robot Chef (a Text-to-Image AI) to cook the dish. The robot reads Chef A's poetic notes, Chef B's technical notes, and Chef C's concise notes, and then creates three plates of food.

The Shocking Discovery:
If you look at the three plates of food, they all look almost identical. You cannot tell which robot cooked which plate, or which chef's notes were used. The unique "flavor" of the original descriptions has vanished.

This paper is about that exact phenomenon. It proves that while AI text generators have very strong, unique personalities, the AI image generators that read their work ignore those personalities almost entirely.

---

The Experiment: A Game of "Guess Who?"

The researchers set up a game to test this theory:

1. The Text Round: They took 30,000 images and asked four different AI models (Claude, Gemini, GPT-4, and Qwen) to describe them. Then, they trained a computer to guess which AI wrote which description.

   • Result: The computer was a genius. It got it right 99.7% of the time. The AI models have very distinct writing styles, just like humans.

2. The Image Round: They took those same descriptions and fed them into a top-tier image generator (like Flux or Stable Diffusion) to create new pictures. Then, they trained a different computer to look at the new pictures and guess which AI description was used to make them.

   • Result: The computer was terrible. It got it right only 50% of the time (which is basically guessing). Even the best image generator couldn't preserve the "voice" of the writer.

Why Does This Happen? (The "Lost in Translation" Problem)

The researchers dug deep to find out why the unique styles disappear. They found three main reasons:

1. The "Detail Drop"

• Analogy: Imagine Chef A writes, "The soup is a deep, velvety crimson with a hint of saffron." Chef B writes, "The soup is red."
• What happens: The Magic Robot Chef hears "red" and "crimson" and just makes a generic red soup. It doesn't capture the "velvety" texture or the specific shade. The robot is great at drawing the main object (a banana), but it's bad at capturing the nuance (the specific lighting or texture) that the text writers were so proud of.

2. The "Color Confusion"

• Analogy: One chef says, "The sky is a 'dusty rose'." Another says, "The sky is 'pale pink'."
• What happens: The robot sees "rose" and "pink" and just paints a standard pink sky. It doesn't seem to understand the subtle difference between the two words. The text is full of color variety, but the image comes out looking the same regardless of who wrote the prompt.

3. The "Camera Angle" Mix-up

• Analogy: One chef says, "Take a photo from a high angle looking down." Another says, "Take a close-up from eye level."
• What happens: The robot often ignores these instructions. It might draw the object from eye level even if the text asked for a high angle. The "directions" in the text are getting lost in the translation to pixels.

Why Should We Care?

This isn't just a fun fact; it has real-world consequences for how we build AI:

• The "Fake Data" Trap: Many companies are using AI to write descriptions for images to train other AIs. This paper says: Be careful. If you mix descriptions from different AIs, you are injecting a lot of "text noise" (different writing styles) that the image generator will ignore. You might be training your image AI on data that doesn't actually match the visual reality.
• The Bottleneck: The problem isn't that the text writers are bad; they are very good. The problem is that the image generators are currently "tone-deaf." They can't hear the subtle instructions in the text.

The Takeaway

Think of it like a translator and a painter.

• The Translator (the text AI) is a master of language. They can write a story in a way that screams "I am Shakespeare!" or "I am Hemingway!"
• The Painter (the image AI) is a bit of a novice. No matter if the story is written by Shakespeare or Hemingway, the painter just paints a generic picture of a banana.

The paper concludes that until image generators get better at listening to the subtle details of the text, there will always be a huge gap between the "personality" of the text and the "reality" of the image.

Technical Summary

1. Problem Statement

The paper investigates a critical assumption in modern multimodal pipelines: that model-generated captions are stylistically neutral and faithfully convertible into visual content by Text-to-Image (T2I) models.

• Context: State-of-the-art image generation pipelines (e.g., DALL·E 3, Flux) increasingly use captions generated by Large Multimodal Models (MLLMs) to expand training corpora and refine text-image alignment.
• The Gap: While it is well-established that LLMs imprint unique "fingerprints" (stylistic and distributional biases) on their text outputs, it remains unclear whether these caption-level idiosyncrasies propagate across modalities into the images generated by downstream T2I systems.
• Hypothesis: If captions from different MLLMs contain distinct stylistic signatures, but T2I models fail to translate these nuances into visual differences, it implies a "cross-modal translation gap." This could introduce text-domain biases that do not manifest visually, complicating the use of synthetic captions as faithful supervisory signals.

2. Methodology

The authors propose a "Name-that-Model" attribution framework to quantify the transfer of idiosyncrasies across modalities. The pipeline consists of two parallel classification tasks:

A. Data Collection

• Image Source: 30,000 images sampled from CC3M, COCO, ImageNet, and MNIST.
• Caption Generation: Four MLLMs (Claude-3.5-Sonnet, Gemini-1.5-Pro, GPT-4o, Qwen3-VL) generate captions for the same images using three progressively detailed prompts (Coarse, Detailed, Very Detailed).
• Image Generation: The generated captions are fed into four T2I models (Stable Diffusion v1.5, v2.1, XL, and FLUX.1-schnell) to synthesize new images.

B. Classification Tasks

1. Text Attribution: A BERT-based classifier is trained to identify the source MLLM of a caption.
2. Image Attribution: A ResNet-18 (or CLIP-based) classifier is trained to identify the source MLLM of a generated image based on the caption used to create it.
3. Control Experiments:
   • Paraphrasing: Testing if fingerprints survive text rewriting.
   • Encoder Probing: Analyzing T5 and CLIP text encoders to see if they discard stylistic signals.
   • Keyword Prompts: Testing if removing narrative style (using only keywords) changes attribution.
   • Natural Image Baseline: Comparing generated image attribution against natural image attribution to rule out classifier limitations.

3. Key Results

A. High Text Attribution Accuracy

• The text classifier achieves 99.70% accuracy in identifying the source MLLM.
• This confirms that MLLMs produce highly distinctive, model-specific linguistic fingerprints (e.g., specific preferences for lighting descriptions, perspective angles, or compositional framing) that persist even under paraphrasing (accuracy remains >90%).

B. Collapse of Image Attribution

• In stark contrast, image attribution accuracy drops to near-random levels.
• Even with the state-of-the-art Flux-schnell model, image classification accuracy peaks at only 49.85% (barely above the 33.3% random chance for 3 classes).
• This is significantly lower than the 76.7% accuracy achieved when classifying natural images of similar scale using the same architecture, proving the difficulty lies in the generation process, not the classifier.

C. Analysis of the "Idiosyncratic Gap"

The paper identifies specific dimensions where caption nuances fail to transfer to images:

1. Level of Detail: While Gemini captions are consistently rated as "most detailed" (84% of the time), the resulting images show no corresponding increase in detail; in fact, GPT-generated images (from less detailed captions) were sometimes judged as more detailed.
2. Color and Texture: Despite significant differences in color and texture vocabulary usage among models (e.g., Gemini uses 2x more nuanced color terms than GPT), the generated images do not reflect these differences. Nuanced color instructions are normalized by T2I models.
3. Composition: Models differ in how they describe spatial layers, subject focus, and symmetry. However, these compositional instructions are not faithfully realized in the generated images, which tend to default to common scene structures.
4. Encoder Preservation: Text encoders (T5, CLIP) preserve the stylistic signals (94–99% accuracy), indicating the signal loss occurs during the generation stage, not the encoding stage.

4. Key Contributions

1. Novel Framework: Introduces a scalable, model-agnostic attribution framework to quantify both the stylistic idiosyncrasies of caption models and the prompt-following ability of T2I systems.
2. Quantification of Cross-Modal Gap: Provides empirical evidence of a massive discrepancy: text attribution is near-perfect (99.7%), while image attribution is weak (50%), highlighting a fundamental failure in cross-modal translation.
3. Root Cause Analysis: Demonstrates that the gap is not due to surface phrasing (which survives paraphrasing) but stems from deeper content-selection and perspective patterns (e.g., choice of detail, color nuance, composition) that current T2I models fail to preserve.
4. Attribution-as-Evaluation: Proposes using attribution accuracy as a complementary metric for evaluating T2I instruction-following capabilities. Higher attribution accuracy would imply better preservation of caption semantics.

5. Significance and Implications

• Training Data Bias: Aggregating captions from diverse MLLMs for T2I training may inject text-domain biases that do not translate into visual diversity, potentially limiting the model's ability to learn fine-grained visual concepts.
• Instruction Following: The results suggest that current T2I models struggle to follow instructions beyond simple object keywords. They fail to capture nuanced attributes like specific lighting moods, detailed textures, or complex compositional layouts described in the prompt.
• Future Directions: The paper calls for T2I models to improve their ability to translate high-level semantic choices (style, perspective, detail level) into visual outputs, rather than just generating generic images that match object keywords.

In summary, the paper reveals a "Asymmetric Idiosyncrasy": while multimodal models are highly distinct in how they describe the world, current generative models are surprisingly homogenized in how they visualize those descriptions.