Technical Summary: Correlation Is Not Enough – Embedding Human Metadata for Individual Causal Discovery

1. The Problem: Embedding Geometry as Correctness

The paper addresses a critical failure mode in deploying Large Behavioural Models (LBMs) for individual causal discovery. Unlike standard retrieval systems where a downstream language model can filter out noisy embedding results, an LBM constructs a causal graph of a user's life where embedding proximity serves as direct evidence for hypothesizing edges. If the embedding space places unrelated events (e.g., "cortisol 28 µg/dL" and "stock-market volatility") close together due to surface-level lexical overlap, the LBM will erroneously infer causal links, corrupting the user's life model.

The authors demonstrate that off-the-shelf biomedical encoders (BioBERT, PubMedBERT, BioM-ELECTRA) fail this specific requirement. Despite high within-domain similarity, these models exhibit anisotropy, collapsing embeddings into a narrow cone where unrelated cross-domain pairs achieve cosine similarities of 0.76–0.92. Consequently, cross-domain discrimination accuracy is exactly 0%, rendering pretrained models unusable for graph-based causal reasoning.

2. Methodology

The authors propose a two-pass contrastive fine-tuning strategy to correct embedding geometry, followed by a hardware-optimized inference pipeline.

2.1 Two-Pass Fine-Tuning

• Pass 1 (Contrastive Training): Models are fine-tuned on 72,034 triplets (anchor, positive, hard-negative) drawn from eight public datasets (including BIOSSES, MedNLI, and psychological journals). The training utilizes Matryoshka Loss wrapped around Multiple Negatives Ranking Loss (MNRL). This approach trains nested dimensions (768, 512, 256, 128, 64) simultaneously, allowing a single model to serve different vector sizes for storage, traversal, and on-device inference without retraining.
• Pass 2 (BODHI): To address the limitations of heuristic-based hard negatives, the authors introduce BODHI (Biomedical Ontology-Graph Hard-negative Inference). This pass mines hard negatives from a curated biomedical knowledge graph. Pairs of concepts that exist in the graph but lack an explicit edge (or a path of length ≤2) are treated as hard negatives. This teaches the model that the absence of a domain-expert connection is a strong signal of non-causality, even if surface words overlap.

2.2 Hardware and Inference Optimization

The study evaluates performance on two Intel platforms:

• Primary: Intel Xeon 6737P (Granite Rapids) with native AMX (Advanced Matrix Extensions) for BF16 matrix multiplication.
• Reference: AWS c6i.8xlarge (Ice Lake) without AMX, relying on AVX-512 VNNI for INT8.

The authors utilize OpenVINO for inference, exporting models with FP16 weights and a BF16 runtime. They challenge standard deployment advice by demonstrating that FP16 outperforms INT8 on AMX silicon at typical serving batch sizes. While INT8 offers theoretical throughput gains, the dequantization overhead required for attention and feed-forward blocks on scalar units negates these benefits at batch sizes <512. Furthermore, INT8 quantization on non-AMX hardware (Ice Lake) destroys the discrimination signal created by fine-tuning, reducing the geometry ratio significantly.

3. Key Results

3.1 Embedding Quality

• Geometry Ratio: The ratio of inter-domain to intra-domain distance improved from 1.05× (pretrained) to 1.63× (Pass 1) and finally 2.30× (BODHI). A ratio >1.5 is identified as the threshold for reliable graph reasoning.
• Discrimination Gap: The gap between unrelated and related pairs increased from +0.013 (pretrained PubMedBERT) to +0.392 (BODHI).
• BIOSSES Correlation: PubMedBERT Pass 1 achieved a rank correlation of 0.828 with clinician scores, surpassing the 0.80 threshold for strong agreement.
• Cross-Domain Accuracy: Pretrained models scored 0% on cross-domain discrimination. BODHI achieved an F1 score of 93.0% at a cosine threshold of 0.40, compared to a maximum of 72.7% for pretrained models.
• Model Architecture: BioBERT and PubMedBERT (BERT-based) responded well to fine-tuning. BioM-ELECTRA, despite high raw similarity, failed to reshape into a usable embedder (F1 remained low), indicating that span-extraction pretraining objectives do not translate to causal embedding tasks.

3.2 Performance and Latency

• Latency Reduction: On the AMX server, OpenVINO reduced single-query latency from 1,367 ms (PyTorch) to 10 ms (a 133× speedup) while maintaining cosine fidelity >0.994.
• Throughput: The fine-tuned model reached 555 sentences per second at batch 512.
• AMX vs. Non-AMX: The AMX server was 13–27× faster than the Ice Lake reference. Crucially, while Ice Lake could use INT8 for speed, this quantization flattened the discrimination gap, retaining only 17–50% of the signal. On AMX, FP16/BF16 maintained full signal fidelity with superior speed.

4. Significance and Claims

The paper claims that for Large Behavioural Models, embedding geometry is correctness, not just a tuning knob. The authors assert that:

1. Causal Discovery Requires Separation: Standard retrieval tolerates high noise floors; graph reasoning does not. The proposed BODHI method successfully separates causally unrelated cross-domain events, enabling the LBM to hypothesize edges without inventing connections.
2. Ontology-Guided Learning: Training on "absent edges" in a knowledge graph is a high-leverage strategy for improving embedding discrimination, generalizing beyond the biomedical domain to any structured ontology.
3. Hardware-Specific Deployment: The study provides a counter-intuitive finding for Intel AMX hardware: FP16 is superior to INT8 for BERT-class embedders at production batch sizes due to dequantization overheads and the preservation of the discrimination gap.
4. Reproducibility: The authors release the full benchmark suite (B1–B6), the 72k-pair corpus, the BODHI generator, training scripts, and optimized weights, aiming to fix a field-wide failure in cross-domain causal discovery rather than creating a proprietary moat.

The paper concludes that while getting from correlation to causation in populations requires randomized trials, for an individual LBM, it is largely a matter of embedding geometry. By correcting the geometry, the graph traversal can function correctly; without it, downstream reasoning is fundamentally flawed.