Time Series, Vision, and Language: Exploring the Limits of Alignment in Contrastive Representation Spaces

This paper investigates the convergence of time series, vision, and language representations, revealing that while independently pretrained encoders are nearly orthogonal, post-hoc contrastive alignment improves with model size but exhibits asymmetry where time series align more strongly with vision than text, and richer textual or visual descriptions yield diminishing returns beyond a certain density threshold.

The Gist

Imagine you are trying to teach three very different friends how to describe the same event: a Time Series (a list of numbers changing over time, like stock prices), a Vision friend (who sees a line graph of those numbers), and a Language friend (who reads a sentence describing the trend).

The big question this paper asks is: Can these three friends eventually agree on what they are looking at, even though they speak completely different "languages"?

This idea is called the "Platonic Representation Hypothesis." It suggests that if you train smart AI models enough, they all start to see the world in the same way, like different translators converging on the same truth. But this paper investigates if this works when one of the friends is a "Time Series," which is notoriously hard to understand.

Here is the breakdown of their findings, using some simple analogies:

1. The "Alien" Problem (No Alignment at First)

Before they try to talk to each other, the authors checked what happens if they just look at the data independently.

• The Analogy: Imagine the Time Series friend speaks only in raw numbers, the Vision friend speaks in shapes, and the Language friend speaks in words. If you put them in a room without a translator, they are like aliens speaking different languages. They are orthogonal, meaning they are completely disconnected. The numbers don't naturally look like the words, and the words don't naturally look like the shapes. They are living in separate universes.

2. The "Bridge" Strategy (Contrastive Learning)

To fix this, the researchers acted as a strict teacher. They used a method called Contrastive Learning.

• The Analogy: Imagine a game of "Match the Pair." The teacher shows the group a picture of a rising line graph, the number list, and the sentence "The stock went up." The teacher says, "You three must point to each other and say, 'We are the same thing!'"
• Over time, the AI models learn to translate their internal thoughts into a shared language (a common space) so they can recognize each other.

3. The Big Surprise: The "Visual Bridge"

This is the most important finding. The researchers expected the Language friend and the Time Series friend to get along well because humans often describe numbers with words. They were wrong.

• The Result: The Time Series friend got along much better with the Vision friend (the graph) than with the Language friend.
• The Metaphor: Think of the Time Series as a secret code.
  • Vision is like decoding the secret with a flashlight. When you turn the numbers into a line graph, the "secret" (the trend, the spike, the dip) becomes visible. It's easy to see the shape.
  • Language is like reading a summary. The word "upward trend" is an abstract label. It tells you what happened, but it doesn't show you how it happened.
  • The Conclusion: It is easier to match a secret code to a picture of the code than to a sentence describing it. The "shape" of the data is more obvious than the "story" of the data.

4. The "Middleman" Effect

Because the Time Series and Language friends struggle to talk directly, the Vision friend acts as a bridge.

• The Analogy: If you want to translate a complex math problem (Time Series) into a poem (Language), it's hard to do directly. But if you first draw a diagram (Vision) and then write the poem about the diagram, it works much better.
• The paper found that when all three are trained together, the Image helps the Text understand the Numbers much better than if the Text and Numbers were trying to learn alone.

5. The "More is Not Always Better" Rule

The researchers tested if writing longer, more detailed descriptions (more "Information Density") would help the friends understand each other better.

• The Analogy: Imagine you are trying to describe a sunset.
  • Level 1: "It was pretty." (Too vague)
  • Level 2: "The sun went down, and the sky turned orange." (Good)
  • Level 3: "The sun descended at 6:42 PM, turning the sky from deep blue to a gradient of burnt orange and violet, with a temperature drop of 5 degrees..." (Too much!)
• The Finding: Going from Level 1 to Level 2 helped a lot. But going from Level 2 to Level 3 didn't help much more.
• Once the description is clear enough to capture the main idea, adding more details doesn't make the AI understand the connection any better. There is a "saturation point" where more words just become noise.

6. The "Indirect" Problem

They also tested this with medical data (ECG heartbeats).

• The Scenario: Sometimes, the text doesn't describe the heartbeat shape directly; it just gives a diagnosis like "Atrial Fibrillation."
• The Result: This made the alignment even worse. It's like trying to match a picture of a broken leg to a word that just says "Pain." The connection is too abstract. The AI struggled to link the specific shape of the heartbeat to the medical diagnosis without a clear visual or direct description.

Summary: What Does This Mean for the Future?

This paper teaches us that when building AI systems that handle numbers, pictures, and words:

1. Don't expect them to magically agree. You have to force them to learn a shared language.
2. Pictures are powerful translators. If you want an AI to understand time-based data (like weather or stocks), showing it a graph is often more effective than just giving it a text description.
3. Clarity beats length. It's better to have a clear, concise description of a pattern than a massive, overly detailed paragraph.
4. The "Bridge" works. Using images to help connect numbers and text is a winning strategy.

In short: Numbers are hard to talk about, but easy to see. If you want an AI to understand them, show it a picture first.

Technical Summary

1. Problem Statement

The paper investigates the Platonic Representation Hypothesis (PRH), which posits that models trained on different modalities converge toward a shared latent structure of the world. While this convergence is well-documented for Vision and Language (VL), it remains unclear whether Time Series data participates in this convergence.

The core challenge is the semantic mismatch between modalities:

• Vision: Encodes structure explicitly through spatial geometry.
• Language: Encodes semantics explicitly through symbolic tokens.
• Time Series: Encodes meaning implicitly through temporal variation (e.g., trends, periodicity, anomalies are latent properties requiring computation, not discrete tokens or visual features).

The authors ask: Can time series achieve the same degree of representational alignment with vision and language as vision and language do with each other?

2. Methodology

The authors propose a Trimodal Contrastive Alignment Framework to systematically study this problem.

• Framework Design:

  • Encoders: They utilize frozen, pretrained encoders for Time Series (e.g., Chronos, MOMENT, TimesFM), Vision (e.g., DINOv2, SigLIP, ViT), and Language (e.g., T5, Qwen, Mistral).
  • Projection: Trainable projection heads map the frozen encoder outputs into a shared embedding space.
  • Objective: A symmetric Contrastive Loss (InfoNCE) is applied jointly across all three modality pairs: Time Series–Image (TS–IMG), Time Series–Text (TS–TXT), and Image–Text (IMG–TXT).
  • Datasets:
    • CaTS-Bench: Primary dataset with aligned triplets (numeric series, visual plots, textual captions). Used to study Information Density (ID) by creating caption variants (condensed vs. dense).
    • TRUCE: Short time series with direct textual descriptions and visual variants (generic, styled, annotated) to test visual mediation.
    • MIMIC-IV & PTB-XL: Clinical ECG datasets with diagnostic reports (indirect text supervision) to test alignment under semantic abstraction and language shifts (English vs. German).

• Evaluation Metrics:

  • Global Geometry: Cosine Similarity Margin, Procrustes Disparity (alignment via orthogonal transformation).
  • Non-linear Similarity: Centered Kernel Alignment (CKA).
  • Local Structure: Mutual k-Nearest Neighbors (kNN) overlap.
  • Retrieval: Recall@k (R@1, R@5, R@10).

3. Key Contributions & Findings

A. Asymmetric Convergence

The study reveals that alignment is not uniform across modalities:

• TS–IMG > TS–TXT: Time series align significantly more strongly with visual plots than with text.
• Reasoning: Visual plots externalize latent temporal structures (trends, peaks) into explicit geometric forms that vision encoders can easily detect. Text abstracts these concepts symbolically, creating a larger "explicitness gap."
• Image as an Intermediary: Images act as effective bridges. In trimodal settings, the presence of the image modality improves TS–TXT alignment, suggesting a pathway: Time Series (Implicit) → Image (Explicit) → Text (Abstract).

B. Scaling Behavior

• Model Size: Overall alignment improves with model scale, but the gains are uneven.
• Saturation: TS–IMG alignment saturates earlier (stronger at smaller scales) due to shared visual grounding. TS–TXT alignment shows a stronger positive correlation with scale but remains the weakest pair even at large scales (e.g., 27B parameters).
• Local vs. Global: While global geometric similarity (Cosine/Procrustes) improves with scale, local neighborhood structure (kNN) remains weak, indicating that models capture broad trends but fail to align fine-grained semantic details.

C. Information Density (ID) Saturation

• Threshold Effect: Increasing the semantic richness (Information Density) of text captions improves alignment up to a certain threshold.
• Diminishing Returns: Once captions reach a sufficient level of detail, further increasing density (e.g., doubling the ID) yields no significant improvement. This suggests that the limit is not a lack of information, but the inherent representational mismatch between symbolic text and numeric signals.

D. Impact of Semantic Explicitness

• Direct vs. Indirect: Alignment is strongest when text directly describes signal structure (e.g., "increasing trend"). When text provides indirect supervision (e.g., clinical diagnoses like "atrial fibrillation" without describing the waveform), alignment degrades significantly.
• Language Shift: Alignment is sensitive to language shifts (English vs. German), with performance dropping when the text language mismatches the pretrained language model's bias, even if the underlying signal is identical.

E. Visual Mediation

• Richer Visuals Help: Annotated plots (with labels and axes) yield better TS–IMG alignment than generic plots, confirming that visual explicitness aids alignment.
• Trimodal Advantage: Removing the image modality (bimodal TS–TXT) degrades performance, while adding it (trimodal) consistently boosts TS–TXT alignment, validating the "semantic bridge" hypothesis.

4. Significance and Implications

• Redefining Multimodal Convergence: The paper challenges the assumption that all modalities converge equally. It argues that semantic explicitness and representational format are critical determinants of alignment, often more so than model scale alone.
• Guidance for Multimodal Systems: For systems involving time series (e.g., healthcare, finance), relying solely on text descriptions is suboptimal. The authors suggest that visualizing time series or using structured, explicit text is crucial for effective alignment.
• Future Directions: The findings highlight the need for datasets with richer, more explicit multimodal annotations and suggest that future architectures should explicitly model the "explicitness gap" between numeric signals and symbolic language.

Conclusion

The paper demonstrates that while time series, vision, and language can be aligned via contrastive learning, the process is asymmetric and constrained by semantic explicitness. Time series align best with their visual representations, and text alignment is limited by the abstraction inherent in language. Images serve as a critical semantic bridge, and simply increasing model size or text density cannot fully overcome the fundamental representational mismatch between numeric signals and symbolic language.