Building an Atlas of Social Experiments to Link Studies, Reconcile Conflicts, and Bridge Gaps

Technical Summary: EXATLAS – Building an Atlas of Social Experiments

Problem Statement
Social and behavioral sciences generate thousands of experiments annually, yet their findings often fail to accumulate into a coherent map of knowledge. As noted by Watts (2017), the field suffers from an "incoherency problem": scholars examine related phenomena without integration, competing findings coexist without adjudication, and isolated studies accumulate without theories explaining their relationships. Current literature organization relies on surface-level similarity or shallow retrieval, which fails to capture the compositional nature of experimental landscapes. Directly prompting Large Language Models (LLMs) to summarize or predict outcomes often lacks the rigorous evidential grounding required to detect conflicts, link consistent findings, or identify genuine gaps in the literature.

Methodology: The EXATLAS Framework
The authors introduce EXATLAS, a framework designed to transform an archive of social experiments into a structured "atlas" where studies link, conflict, or leave bridgeable gaps. The framework operates on the premise that experimental landscapes are inherently compositional: a target finding may be reconstructed from a weighted combination of prior studies even if no single prior study matches it exactly.

The framework proceeds in two main stages:

1. Representation and Composability Assessment:

   • Enrichment: Experiments are encoded as tuples $(T_i, O_i, Z_i, \tau_i)$, representing treatment, outcome, context, and observed effect. An LLM (OpenAI o3) enriches brief treatment/outcome descriptions with context from the full paper to improve semantic accuracy.
   • Embedding: Experiments are mapped to a shared treatment-outcome space using text embeddings. The representation $x_i$ concatenates embeddings of the treatment, outcome, and their element-wise product to capture joint relations.
   • Local Search: For a target experiment $e_t$, EXATLAS searches for a local candidate set $C(t)$ of prior experiments. It solves a constrained optimization problem to find weights $\alpha_{tj}$ that minimize the Euclidean distance between the target representation and the weighted sum of source representations.
   • Rejection Mechanism: A normalized residual $\rho_t$ is calculated. If $\rho_t$ exceeds a threshold $\lambda$, the target is deemed non-composable (a gap). If $\rho_t \le \lambda$, the target is composable. This design makes the framework "rejectable," preventing forced composition where local support is insufficient.

2. Atlas Organization (Three Outcomes):

   • Link (Consistency): If the target is composable and the composed effect direction matches the observed direction, the target is linked to the consistent prior evidence.
   • Reconcile (Conflict): If the target is composable but the composed direction disagrees with the observed result, the system identifies a theoretical tension. An LLM is prompted with the conflicting evidence to propose a meta-theory or moderator structure (e.g., a hidden $2 \times 2$ interaction) that explains the divergence.
   • Bridge (Gap): If the target is non-composable, the system identifies a gap. An LLM iteratively proposes "bridge experiments" to be added to the archive until the target becomes composable, effectively suggesting new research directions to close the evidence gap.

Key Contributions

1. A Rejectable Framework: EXATLAS introduces a method for composing experimental evidence that explicitly rejects targets lacking local support, distinguishing between Kuhnian anomalies (conflicts within a composable region) and Lakatosian extensions (gaps requiring new frontiers).
2. Theoretical Error Bounds: The authors provide a theoretical error bound for composition under the assumption of local smoothness of the treatment-effect surface. The error decomposes into extrapolation error, local model bias (curvature), and unobserved residual variation.
3. Generative Failure Modes: The framework treats its "failures" (conflicts and gaps) as generative outputs. Conflicts trigger theoretical reconciliation proposals, and gaps trigger actionable bridge experiment designs.

Empirical Results

• Composition Accuracy: On a held-out set of 72 experiments from an archive of 360 (sourced from top management and psychology journals), EXATLAS certified 72 targets as locally supported. For these, the predicted effect direction matched the observed direction in 98.61% (71/72) of cases. This significantly outperformed baselines, including direct LLM prediction, nearest-experiment RAG, and synthetic participant simulations.
• Conflict Reconciliation: In a human evaluation of 13 conflict cases, experts found that EXATLAS's LLM-generated reconciliations often identified the same broad directions as human experts but provided more explicit mechanisms or moderator structures. In some cases, the LLM surfaced hidden structural anomalies (e.g., a cross-cell conflict in hiring literature involving age, creativity, and competition) that human experts missed without algorithmic assistance.
• Bridge Experiment Quality: Human raters evaluated proposed bridge experiments against a "Forced Composition" baseline (which ignores the rejection mechanism). While feasibility scores were comparable, EXATLAS Bridge proposals were rated significantly higher in Plausibility ($p=0.040$) and Connectedness ($p=0.031$), indicating they are more theoretically grounded and better linked to the source literature.

Significance and Claims
The paper claims that the archive of social experiments contains more latent structure than current practices extract. By making this structure explicit, EXATLAS serves not merely as a summary tool but as an "atlas" to guide future theory and experimentation.

The authors are modest in their scope, explicitly stating they do not claim LLMs replace human participants or that text-embedding composition is universally accurate. Instead, they argue that an experimental archive, when read compositionally, supports a calibrated estimate of where evidence reaches, where it falls short, and where those limits point to meaningful research directions. The framework provides infrastructure to address the "incoherency problem" by treating the literature as a map rather than a pile of isolated studies.

Limitations
The authors acknowledge that the high accuracy (98.6%) applies only to locally supported targets and should not be generalized to extrapolations beyond the archive. They note that the framework cannot prove from text alone that the embedding space preserves causal mechanisms; it treats feature geometry as a retrieval and reconstruction device. Furthermore, the reconciliation module is a generator, not a verifier, and the underlying archive inherits biases regarding what gets published and coded.