Human-Like Coarse Object Representations in Vision Models

1. Problem Statement

Human intuitive physics relies on coarse, volumetric "body" representations of objects rather than pixel-perfect geometric details. Humans tend to "fill in" concave regions (approximating them as convex or smoother shapes) to make efficient physical predictions, such as estimating time-to-collision (TTC).

In contrast, modern computer vision segmentation models are trained to optimize pixel-accurate masks against ground-truth annotations. This creates a potential misalignment:

• Human Representation: Coarse, smoothed bodies optimized for physical affordances (action, collision).
• Model Representation: Fine-grained, boundary-precise masks optimized for recognition.

The paper investigates whether and under what conditions standard vision segmentation models spontaneously acquire these human-like coarse representations, or if they strictly adhere to fine-grained segmentation.

2. Methodology

The authors developed a pipeline to compare vision model outputs directly with human behavioral data using a Time-To-Collision (TTC) paradigm.

A. Dataset and Task

• Human Data: Utilized a dataset from Li et al. [25] involving 226 participants. Participants watched videos of two objects moving toward each other and pressed a button when they predicted a collision.
• Key Behavioral Signature: Humans systematically predict collisions earlier when objects have concave sides facing the collision compared to convex sides. This indicates a mental "filling in" of concavities (coarse approximation).
• Synthetic Training Data: Since pre-trained models fail on the specific polygonal stimuli used in the human experiment, the authors created a synthetic dataset of 500 training images featuring diverse polygons with controlled concavity, irregularity, and spikiness.

B. Vision Models

• Architecture: Six variants of SegFormer (B0 to B5), a hierarchical transformer encoder with an MLP decoder, ranging from ~3.8M to ~84.7M parameters.
• Fine-Tuning: Models were fine-tuned on the synthetic dataset using cross-entropy and Dice loss.

C. Evaluation Pipeline (TTC Simulation)

1. Input: The model receives the initial static frame of a collision video.
2. Segmentation: The model outputs a binary mask for the objects.
3. Simulation: Using the known ground-truth velocities of the objects, the system translates the segmented masks forward in time.
4. Prediction: The model's predicted TTC is the first frame where the translated masks overlap.
5. Metric: The authors calculated the Concavity Effect ($\Delta$) for both humans and models:
   $$\Delta = \text{TTC}_{\text{concave}} - \text{TTC}_{\text{convex}}$$
   The primary error metric ($\bar{E}$) is the absolute difference between the model's $\Delta$ and the human $\Delta$.

D. Experimental Variables

The authors systematically varied three factors to manipulate the model's effective capacity and representation coarseness:

1. Training Time: Analyzing checkpoints at different stages of training.
2. Model Size: Comparing different SegFormer variants (B0–B5).
3. Neuron Pruning: Applying unstructured magnitude-based pruning to trained models to reduce effective capacity without retraining.

3. Key Contributions

1. Comparison Pipeline: Established a unified framework to quantitatively compare segmentation model outputs with human physical judgments (TTC).
2. Discovery of the "U-Shaped" Error Curve: Demonstrated that the alignment between model and human behavior is non-monotonic.
   • Under-segmentation: Small/pruned models are too coarse (blobs), failing to capture necessary structure.
   • Over-segmentation: Large/fully trained models are too precise (pixel-perfect boundaries), failing to smooth concavities like humans.
   • Ideal Granularity: An intermediate regime yields the best match to human behavior.
3. Resource-Rational Account: Provided evidence that human-like coarse representations emerge from resource constraints (limited capacity, training time, or pruning) rather than being a specific inductive bias of the human brain.

4. Key Results

• Training Time: Early in training, models under-segment. As training progresses, they reach an "ideal" intermediate state where they smooth concavities effectively. With excessive training, they over-segment, preserving fine boundary wiggles that misalign with human TTC predictions.
• Model Size:
  • Small models (B0, B1): Under-segment; error is high.
  • Large models (B4, B5): Over-segment; error is high.
  • Intermediate models (B1, B2): Achieve the lowest error, matching human concavity effects best.
  • Note: Larger models reach this "human-like" regime faster (in fewer training steps) than smaller models.
• Pruning:
  • No Pruning: Models produce fragmented, overly detailed masks.
  • Heavy Pruning: Models collapse into coarse blobs.
  • Moderate Pruning: Induces a "sweet spot" where the model's concavity effect aligns most closely with humans.
• Visual Evidence: Probability heatmaps show that models, like humans, exhibit "outward diffusion" of activations around concave regions, effectively smoothing them, while maintaining stability on convex edges.

5. Significance and Implications

• Theoretical Insight: The findings support resource-rational accounts of cognition. Human-like coarse representations are not a specialized biological feature but an efficient compromise arising from constraints on bandwidth, computation, and metabolic cost. Both brains and machines converge on simplified encodings when resources are limited but not extreme.
• Practical Application:
  • AI Safety: For robots interacting with humans, "perfect" segmentation may be detrimental to physical prediction. Models should be tuned to the "ideal granularity" to predict human behavior accurately.
  • Model Design: Engineers can elicit physics-efficient representations using simple "knobs":
    • Using early checkpoints rather than fully converged models.
    • Selecting modest architectures (avoiding the largest models).
    • Applying light pruning to standard models.
• Paradigm Shift: Challenges the assumption that better segmentation (pixel-perfect) is always better. For physical reasoning tasks, "not too fine, not too coarse" is the optimal operating point.

In summary, the paper demonstrates that the "imperfections" of human object representation (smoothing concavities) are actually a computationally efficient strategy that naturally emerges in vision models when they are constrained by limited resources, offering a mechanistic explanation for human intuitive physics.