BINO: Encoder Centric Self Supervised Stereo With Native Pair Input

1. Problem Statement

Stereo matching requires feature representations that preserve fine-grained cross-view correspondence, not just semantic similarity. While recent self-supervised vision models (e.g., DINOv2, MAE) are robust for recognition, they are single-view objectives and do not explicitly learn dual-view geometry. Conversely, existing geometry-focused methods (e.g., CroCo v2) often rely on a binocular decoder or an explicit linkage module (like a cost volume) during pretraining to fuse information.

This creates a dependency where the quality of the representation is conflated with the power of the matching network. The authors ask a fundamental question: Can a compact, reusable encoder learn strong binocular spatial structure on its own, without relying on a dedicated binocular decoder or linkage module during pretraining?

2. Methodology: BINO Architecture

BINO (Binocular) proposes an encoder-centric approach where cross-view structure is injected directly into the tokenization and positional encoding stages. The model is trained using self-supervised distillation on raw stereo pairs.

Key Design Principles:

1. Early Binocular Fusion (Input Level):

   • Instead of processing left ($L$) and right ($R$) images as separate streams, BINO fuses them at the input stage via horizontal pixel interleaving.
   • The fused image $X$ has dimensions $3 \times H \times 2W$, where even columns are from $L$ and odd columns are from $R$.
   • Stereo Micro Cell Tokens: The patch embedding is applied to this fused image. Consequently, every token (patch) inherently contains local information from both views simultaneously, creating "stereo micro cells" from the very first transformer block.

2. Row-Aware Patch Phase Positional Encoding:

   • Stereo geometry dictates that correspondences lie on the same epipolar row, with ambiguity along the horizontal axis.
   • BINO uses a specialized positional encoding that distinguishes between the patch column index ($p$) and the local phase ($q \in \{0, 1\}$) within the stereo micro cell.
   • It employs Rotary Position Embeddings (RoPE) based on the $(row, patch\_col)$ pair rather than the raw fused index. This encourages the model to learn relative horizontal structure (disparity) rather than memorizing absolute fused column identities.
   • De-positioning: Before the projection head, the row embedding is subtracted to ensure the final features rely on learned geometry rather than trivial row priors.

3. One-View Masked Token Distillation:

   • Teacher-Student Framework: An EMA (Exponential Moving Average) teacher receives the full binocular input ($L, R$). The student receives a masked version where only one view is masked (e.g., $M(L), R$).
   • Objective: The student must predict the masked tokens of the occluded view using the visible complementary view. This forces the encoder to learn cross-view completion and spatial reasoning.
   • Token-Only Supervision: The loss is calculated only on token-level cross-entropy, avoiding a global CLS token. The authors found that global invariance objectives weaken dense matching performance.

4. Stereo-Aware Nuisance Perturbations:

   • Training includes realistic stereo challenges: partial occlusion, photometric mismatches, and noise. Crucially, the two views can receive independent appearance perturbations, forcing the encoder to rely on geometric structure rather than appearance matching.

3. Evaluation Protocols

The paper evaluates the learned encoder under three distinct regimes to test different aspects of the representation:

1. Frozen No-Linkage Descriptor Probe: The encoder is frozen and exported as a descriptor map (using duplicated input $I, I$). Stereo is recovered via simple cosine matching. This tests if the latent is a direct, usable metric descriptor.
2. Shared Lightweight Head: All encoders are frozen, and the same lightweight stereo head is trained on top of them. This isolates the quality of the encoder representation from the matching architecture.
3. Native Pair Input: The encoder is evaluated on actual stereo pairs $(L, R)$ using a fused-grid decoder, testing the model's native binocular coupling capabilities.

4. Key Results

Experiments were conducted primarily on the KITTI dataset (Stereo 2012 and 2015) with pretraining only on the unlabeled KITTI Object dataset (low-resource setting).

• Frozen No-Linkage Performance:

  • BINO achieved the best results among all baselines (including CroCo v1/v2, iBOT, DINOv2/v3) on proxy dense stereo, hard-negative retrieval, and real KITTI Stereo 2012 disparity.
  • It reduced EPE (End-Point Error) by 28.5% compared to the strongest baseline (CroCo v2) in the descriptor probe.
  • It was the only model with a positive hard-negative margin, indicating superior selectivity.

• Shared Head Performance:

  • When paired with a shared lightweight stereo head, BINO remained one of the top two performers (competitive with CroCo v2).
  • Efficiency: BINO uses a significantly smaller encoder (1.34M parameters) compared to CroCo v2 (3.34M parameters) while maintaining near-parity in performance.

• Native Input & Robustness:

  • Large Disparity: Under native pair input, BINO showed the largest gains in the "hard geometry" regime (large disparities), reducing error by 28.7% compared to its own export pathway.
  • Occlusion Robustness: In unilateral occlusion tests (one view blocked), BINO degraded significantly less than CroCo v2 and other baselines, validating the effectiveness of the one-view masked training objective.

• Mechanistic Analysis:

  • Internal analysis showed that epipolar structure (concentration of probability mass on the correct row) emerges progressively through the network layers.
  • Counterfactual experiments (shuffling rows or replacing views) confirmed that the learned geometry depends on the ordered horizontal structure of the complementary view, not just a generic row bias.

5. Key Contributions

1. Encoder-Centric Paradigm: Demonstrates that strong binocular spatial understanding can be learned entirely within a compact encoder, eliminating the need for a heavy binocular decoder or explicit linkage module during pretraining.
2. Novel Architecture: Introduces Stereo Micro Cell Tokens (via pixel interleaving) and Row-Aware Patch Phase Positional Encoding, which inject geometric priors directly into the tokenization process.
3. Training Strategy: Proposes One-View Masked Token Distillation combined with stereo-aware nuisance perturbations to force robust cross-view reasoning.
4. Efficiency: Achieves state-of-the-art performance in low-resource settings with a much smaller parameter count than previous geometry-focused foundation models.

6. Significance

BINO challenges the prevailing assumption that complex stereo matching requires a dedicated, heavy matching network or decoder to fuse views. By proving that a compact encoder can learn reusable, high-quality binocular latents, the paper suggests a new direction for foundation models in 3D vision. This approach allows for:

• Modularity: Encoders can be frozen and reused across different downstream tasks (retrieval, depth, stereo) without retraining the matching head.
• Efficiency: Reducing the computational cost of pretraining and inference by removing the binocular decoder.
• Robustness: Creating representations that are inherently robust to occlusion and appearance mismatch, as evidenced by the superior performance in hard negative and occlusion scenarios.

The results imply that much of the "cross-view reasoning" previously assigned to separate linkage modules can be effectively internalized within the encoder itself, provided the architecture and self-supervised objectives are designed with stereo geometry in mind.