💬 NLP

LLM-Assisted Stance Detection in Scientific Discourse: A Test Case in Bayesian Cognitive Science

This paper presents a rigorously validated framework using theory-driven prompt optimization and multi-rater reliability analysis to enable three frontier LLMs to reliably detect nuanced stance distinctions between realism and instrumentalism in Bayesian cognitive science, successfully scaling qualitative coding to a large corpus and quantifying a significant difference in realism between low-level perception and high-level cognition research.

Original authors: Eyup Engin Kucuk, Tarik Kelestemur, Ömer Dağlar Tanrikulu

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eyup Engin Kucuk, Tarik Kelestemur, Ömer Dağlar Tanrikulu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a massive library of scientific books about how the human mind works. Specifically, you want to know: Do the authors believe these mathematical models actually describe how our brains physically work (Realism), or are they just useful tools for making predictions, like a map that isn't the territory itself (Instrumentalism)?

This is a tricky question. Authors rarely say, "I am a Realist." Instead, they hide their true beliefs in subtle hints, hedged language, and complex sentences. Traditionally, to answer this, you'd need a team of human experts to read every sentence, guess the author's intent, and write it down. This is slow, expensive, and hard to scale.

This paper asks: Can we use advanced AI (Large Language Models) to do this job for us?

Here is the story of how they tried, using a few creative analogies to make it clear.

1. The Problem: The "Hidden Meaning" Puzzle

Think of the authors' stance as a secret recipe. You can taste the dish (read the text), but the ingredients (the author's true belief) aren't listed on the menu.

The Challenge: If you just ask an AI, "Is this Realist or Instrumentalist?", the AI might get lazy. It might guess the most common answer or pick the middle option every time to look safe. It might agree with other AIs just because they are all following the same lazy shortcut, not because they actually understood the text.
The Goal: The researchers wanted to build an AI system that could taste the dish, identify the specific ingredients, and tell you exactly how "Realist" or "Instrumentalist" the recipe is, without cheating.

2. The Solution: The "Smart Coach" and the "Shared Rulebook"

Instead of just asking the AI to guess, the researchers built a rigorous training system with three main parts:

A. The Rulebook (The Codebook)

They didn't just give the AI a vague instruction. They wrote a detailed instruction manual (a codebook).

Imagine a judge's scorecard for a diving competition. It doesn't just say "Good dive." It has specific categories: "Entry," "Twist," "Splash."
This manual defined exactly what words or phrases count as "Realist" and what counts as "Instrumentalist." It even gave a scale from 0 to 100, allowing for nuance (e.g., "mostly Realist but with some doubt").

B. The "Smart Coach" (Autoresearch Loop)

This is the most creative part. The researchers didn't just write a prompt and hope for the best. They used an AI agent as a coach to train the other AIs.

The Process: The coach (an AI) would propose a new way to ask the question (a new prompt). It would test this on a small set of expert-graded examples.
The Trap: Sometimes, the AI would find a "cheat code." For example, it might realize that if it just picks the middle score every time, it gets a high "agreement" score with the experts.
The Guardrails: To stop this, the researchers installed diagnostic gates (like a referee with a whistle).
- Gate 1: "Did the AI actually use the whole scale, or did it just pick the middle?"
- Gate 2: "Did the AI accidentally memorize the answers instead of learning the rules?"
- If the AI tried to cheat, the coach rejected the prompt and tried again. This happened over and over until the AI learned to play the game fairly.

C. The "Shared Rulebook" (One Prompt for All)

They tested three different top-tier AI models (GPT, Claude, and Gemini). Instead of teaching each one a different trick, they forced them all to use the exact same instruction manual.

Why? If all three different AIs, using the same rules, arrive at the same conclusion, you know the answer is solid. It's like having three different judges in a diving competition; if they all give the same score, you trust the result.

3. The Results: What Did They Find?

After training the AIs with this strict system, they let them loose on 6,858 quotes from 210 scientific articles.

The Agreement: The three AIs agreed with each other very well.
- At the sentence level: They agreed about 76% of the time. This is good, but not perfect. Sometimes, a sentence is just ambiguous, and even humans would disagree.
- At the article level: When they looked at the whole article (averaging out the sentences), the agreement skyrocketed to 96–97%.
- Analogy: Imagine three people trying to guess the height of a crowd. They might disagree on the height of one specific person, but if they all agree on the average height of the crowd, they are very reliable. The AIs were great at seeing the "big picture" of each article.
The Discovery: The AIs found something interesting that researchers had suspected for years but never proved with data:
- Low-level researchers (who study senses and movement) tend to believe the models describe real brain mechanisms much more strongly.
- High-level researchers (who study complex thinking) are more skeptical and see the models as just tools.
- The AI quantified this difference: Low-level articles were 8.8 points higher on the "Realism" scale than high-level ones.

4. The Bottom Line

This paper isn't saying "AI can now replace human scientists." Instead, it says:

"If you give AI a very clear, detailed rulebook and a strict referee to stop it from cheating, AI can help us analyze huge amounts of text to find patterns that are too subtle for humans to track manually."

They successfully turned a vague, philosophical question ("Do they believe in the model?") into a measurable, reliable data point. They proved that with the right safeguards, AI can be a powerful partner in understanding the hidden beliefs of the scientific community.

Technical Summary: LLM-Assisted Stance Detection in Scientific Discourse

Problem Statement
Qualitative coding is fundamental to social science but faces scalability limits due to the difficulty of expert annotation, particularly when the target construct is interpretive, theoretically loaded, and only indirectly expressed. While Large Language Models (LLMs) offer a potential solution, their reliability is questionable when the construct is not surface-realized in specific phrases. This paper addresses the challenge of detecting author stances in Bayesian cognitive science regarding the realism–instrumentalism distinction: whether authors treat Bayesian models as descriptions of actual mental/neural mechanisms (realism) or as useful mathematical tools (instrumentalism). These distinctions are often hedged, philosophically deep, and domain-dependent, making them difficult for automated systems to classify without superficial shortcuts.

Methodology
The study employs an expert-led, diagnostic-gated autoresearch framework to optimize a single shared prompt for three heterogeneous frontier LLMs (GPT-5.1, Claude Sonnet 4.6, and Gemini 3 Pro Preview).

Codebook Design: A theory-driven codebook defines complementary Realism and Instrumentalism scores constrained to sum to 100. Each score utilizes a seven-band ordinal scale (ranging from Anti-Realist to Strong Realism) with within-band continuous scoring offsets (midpoint ±4 or ±8). The codebook includes semantic definitions and linguistic-criteria tags (Subject, Claim Target, Modality, Evidence Link) to enable automated validation.
Reference Annotations: A domain expert extracted 251 assumption-bearing quotes from 15 articles, annotated them using the codebook, and partitioned them into tuning (151), development (50), and held-out (50) sets.
Autoresearch Prompt Optimization: An automated agent (implemented in Claude Code) iteratively proposed prompt modifications. A candidate edit was accepted only if it improved a combined reliability metric (harmonic mean of ICC and Krippendorff's $\alpha$ $α$ ) and passed four diagnostic gates:
- Combined Metric: Must improve the harmonic mean score.
- Off-Midpoint: Prevents "midpoint collapse" where models assign all quotes to band midpoints without using the continuous scale.
- Within-Band Residual ICC: Ensures continuous agreement does not degrade significantly.
- Persistent Coherence: Limits structural schema violations.
- De-leaking: A critical step removed cues that inadvertently specified expected band frequencies or default bands, preventing the model from "anticipating" the answer.
Deployment and Analysis: The final zero-shot prompt was deployed on 6,858 quotes from 210 articles. Reliability was assessed via multi-rater agreement (ICC, $\alpha$ , Fleiss' $\kappa$ ) and article-level rank stability (correlations of per-article random intercepts from linear mixed-effects models).

Key Results

Prompt Performance: The final prompt achieved a held-out combined reliability score of 0.76 (harmonic mean of ICC = 0.79 and $\alpha$ = 0.74). The diagnostic gates successfully prevented the models from collapsing to band midpoints, forcing genuine use of the within-band scale.
Inter-Model Agreement: On the full corpus, the three LLMs reached substantial quote-level agreement (Combined = 0.78; ICC = 0.80; $\alpha$ = 0.76). Pairwise exact agreement was 78.4%, with 88.4% of comparisons falling within one adjacent band.
Calibration vs. Convergence: While the models showed systematic calibration shifts (e.g., Claude tended toward Realism-Sceptical, Gemini toward Strong Realism), they achieved near-perfect article-level rank stability ( $r = 0.96–0.97$ ). This indicates that while individual quote classifications varied, the models converged on the relative ordering of articles.
Substantive Findings:
- The corpus was predominantly Weakly Realist (67.9% of quotes), but articles were rarely uniform; only 1.4% used a single band, while 59.5% spanned four or more.
- A significant structural difference was quantified: Low-level perception/motor articles scored 8.8 Realism points higher than high-level cognition articles ( $p < .001, d=0.60$ ), empirically validating a long-held qualitative intuition in the field.

Significance and Claims
The paper positions this work as a case study demonstrating that frontier LLMs can support theoretically demanding qualitative coding when the construct is operationalized through an auditable codebook and validated against expert references.

Methodological Contribution: The authors present a framework for diagnostic-aware prompt optimization that treats the prompt as a common annotation protocol across heterogeneous models. This approach prioritizes convergence under shared instructions over per-model tuning.
Granularity of Analysis: The study highlights that agreement metrics should match the grain of downstream inference. While quote-level agreement was substantial, article-level rank stability was near-perfect, suggesting that aggregate-level analysis can recover signals obscured by item-level noise.
Limitations and Scope: The authors explicitly state that the framework is intended to generalize to similar theoretically demanding tasks, not to all qualitative analysis. They acknowledge limitations, including the use of a single-expert reference set (preventing human-human reliability estimation), the specific domain focus (Bayesian cognitive science), and the potential for shared training-data biases among the LLMs. The work does not claim to replace expert judgment but to extend it, relying on expert-led oversight to detect failure modes (like circular cues) that automated metrics might miss.