State Beyond Appearance: Diagnosing and Improving State Consistency in Dial-Based Measurement Reading

1. Problem Definition

The paper addresses a critical weakness in current Multimodal Large Language Models (MLLMs): their inability to reliably read dial-based measurements (e.g., analog clocks, pointer gauges, industrial meters).

• The Core Issue: While MLLMs excel at general semantic recognition, they fail at fine-grained physical state recovery. They rely heavily on superficial appearance cues (lighting, texture, viewpoint) rather than the underlying physical state (the actual value indicated by the pointer).
• State Inconsistency: The authors identify that MLLMs lack state consistency.
  • Invariance Failure: The same physical state (e.g., 9:46) yields different predictions when the viewpoint or lighting changes.
  • Sensitivity Failure: Neighboring states (e.g., 9:45 vs. 9:46) that look visually similar are often indistinguishable to the model, leading to large errors.
• Root Cause: The feature space of existing MLLMs is organized by appearance rather than the continuous geometric structure of the dial state. This causes brittle performance under real-world variations.

2. Methodology: TriSCA Framework

To solve this, the authors propose TriSCA, a Tri-level State-Consistent Alignment framework. The approach relies on a controlled data synthesis pipeline (separating state from appearance) and three distinct training stages:

A. Data Synthesis Pipeline

The authors created a synthetic data generation pipeline using Blender and code execution to produce controlled datasets for clocks and gauges.

• Factorization: Each sample is decomposed into a Dial State ($s$) (the true value) and an Appearance Condition ($a$) (viewpoint, illumination, blur).
• Diagnostic Relations: This allows the creation of:
  1. Same-State Pairs: Identical $s$, varying $a$ (to test invariance).
  2. Neighboring-State Pairs: Slightly different $s$, similar $a$ (to test fine-grained discrimination).

B. Three-Stage Training Framework

Stage 1: State-Distance-Aware Representation Alignment

• Goal: Reshape the visual feature space to reflect physical state geometry.
• Mechanism: Freezes the language model and updates the visual tower using a contrastive triplet loss.
  • Anchor ($a$) and Positive ($p$) share the same state but different appearances.
  • Negative ($n$) has a different state.
  • Margin ($m_i$): The loss margin is dynamic, based on the "state gap" between the anchor and negative. Larger state differences enforce larger distances in the feature space.
• Outcome: Forces same-state samples to cluster tightly across conditions and separates neighboring states based on their physical difference.

Stage 2: Metadata-Grounded Observation-to-State Reasoning

• Goal: Prevent the model from jumping to a holistic answer; force a grounded reasoning process.
• Mechanism: Supervised Fine-Tuning (SFT) with structured intermediate steps. The model is trained to output a chain of thought that explicitly:
  1. Identifies the indicator (pointer).
  2. Estimates its position relative to the dial.
  3. Maps this position to the calibrated scale.
  4. Derives the final state.
• Outcome: The model learns to ground its prediction in visual evidence rather than appearance shortcuts.

Stage 3: State-Aware Objective Alignment

• Goal: Refine the optimization signal to match the continuous nature of dial values.
• Mechanism: Replaces standard discrete "Exact Match" rewards with a continuous state-distance reward.
  • Uses a Gaussian reward function: $r_{state} = \exp(-d_{state}^2 / 2\sigma^2)$, where $d_{state}$ is the error between predicted and true values.
  • Optimized via Group Relative Policy Optimization (GRPO), which samples multiple outputs and normalizes rewards within the group.
• Outcome: The model is penalized less for being "close" to the answer and more for being far, encouraging precise continuous estimation.

3. Key Contributions

1. Diagnosis of State Consistency: The paper formally defines and diagnoses "state consistency" as the missing capability in MLLMs for dial reading. It provides evidence (via feature-space probing) that current models organize features by appearance, not state.
2. Controlled Benchmarks: Introduction of a synthesis pipeline that decouples state from appearance, enabling rigorous testing of invariance and sensitivity that real-world datasets cannot provide.
3. TriSCA Framework: A unified, three-stage training method that aligns representations, reasoning, and objectives with the intrinsic geometry of dial tasks.
4. Empirical Validation: Demonstrated significant improvements in robustness against viewpoint/illumination shifts and transferability to real-world benchmarks.

4. Experimental Results

The authors evaluated TriSCA on controlled synthetic benchmarks (Clocks and Gauges) and the external MeasureBench.

• Robustness to Appearance Shifts:
  • Baseline models (e.g., Qwen3-VL, Molmo) suffered sharp accuracy drops (e.g., from ~5% to ~2%) under viewpoint/illumination changes.
  • TriSCA maintained high accuracy, showing significantly slower degradation as perturbation severity increased.
• Performance Metrics:
  • Controlled Benchmarks: TriSCA achieved 37.4% accuracy on clocks (vs. 5.1% baseline) and 26.2% on gauges (vs. 1.7% baseline) in the hardest "Combined" condition.
  • Feature Space Analysis: After alignment, the Recall@1 for same-state retrieval improved from 77.33% to 90.00%, and the Silhouette Score (cluster quality) improved from 0.07 to 0.34, confirming better state organization.
  • Real-World Generalization (MeasureBench): On the "Dial" category of MeasureBench, TriSCA improved the open-source model's accuracy by +16.5% (Real-world) and +3.0% (Synthetic) over the base model, while performance on unrelated categories remained stable.
• Continuous Precision: TriSCA significantly reduced Mean Absolute Error (MAE), proving that even when exact matches fail, the predictions are much closer to the true value.

5. Significance

• Beyond Accuracy: The paper shifts the focus from simply "getting the right answer" to understanding how the model represents the world. It argues that for physical tasks, the internal geometry of the feature space must match the physical geometry of the task.
• Generalizability: While focused on dials, the methodology (state-consistent alignment) offers a blueprint for improving MLLMs in any task requiring precise physical state estimation (e.g., linear scales, composite displays).
• Practical Impact: The results suggest that reliable automated meter reading, robotic inspection, and safety-critical monitoring are achievable with MLLMs if they are trained with state-consistent alignment rather than just standard supervised fine-tuning.