Technical Summary: Quantifying the Sources of Instability in LLM-Based Stance Analysis of Public Discourse

Problem Statement

Computational social science (CSS) increasingly relies on automated preprocessing pipelines—specifically speaker diarization, automatic speech recognition (ASR), transcript cleaning, and sentence segmentation—to convert raw media into analyzable text. While significant literature exists regarding annotation validity (e.g., whether LLM labels match human judgments), there is a critical gap in understanding two upstream sources of instability:

1. Preprocessing Validity: Whether choices in the preprocessing pipeline (e.g., diarization method, segmentation rules) alter downstream statistical conclusions.
2. Measurement Validity: Whether the choice of annotation instrument (LLM vs. keyword lexicon) yields different conclusions even when applied to identical preprocessed text.

These two sources are often conflated, making it difficult to diagnose whether observed instability in research findings originates from the data processing pipeline or the measurement instrument itself.

Methodology

The authors propose a pipeline sensitivity analysis framework to systematically disentangle these sources of variation. The study utilizes a corpus of 256 YouTube interviews featuring 41 public figures across five domains: Finance/Investing, Academia/Economics, Central Banking/Policy, Geopolitics/Strategy, and Media/Commentary.

Experimental Design

The study compares two distinct variables:

1. Two Preprocessing Pipelines:
   • VTT-based: Parses YouTube subtitle files, deduplicates artifacts, and uses an LLM (DeepSeek-V4-Flash) for binary speaker classification ("guest" vs. "interviewer").
   • Audio-based (AssemblyAI): Uses a commercial API for ASR and speaker diarization, identifying the dominant speaker by word count as the guest.
2. Two Measurement Methods:
   • LLM Annotation: Zero-shot annotation of valence (positive/negative/neutral) and epistemic modality (emphatic/hedged/neutral) using DeepSeek-V4-Flash.
   • Keyword-Lexicon Scoring: A rule-based approach using a pre-defined lexicon of patterns for negative/positive valence and modality, normalized by video character length.

Diagnostic Framework

The authors define two metrics to quantify instability:

• Pipeline Divergence ($\Delta_{pipeline}$): The change in a derived metric (specifically the Pearson correlation $r$ between negative valence and emphatic modality) when switching preprocessing pipelines for the same measurement method.
• Method Divergence ($\Delta_{method}$): The absolute difference in the metric when switching between LLM and keyword-lexicon methods on the same preprocessed text.

The analysis focuses on per-speaker correlations ($r(neg, emph)$ and $r(neg, hedged)$), excluding speakers with fewer than 4 videos to avoid spurious correlations from small sample sizes.

Key Results

1. Pipeline Sensitivity is Bounded and Sample-Size Dependent

• Concentration in Low-N Speakers: Preprocessing sensitivity is heavily concentrated in speakers with limited video samples ($N \le 5$). For these speakers, the mean absolute change in correlation ($|\Delta r|$) is high (0.44), and sign reversals are common.
• Stability in Well-Sampled Speakers: For the four best-sampled speakers ($N \ge 16$), the mean $|\Delta r|$ induced by pipeline choice is only 0.13 under LLM annotation. No individual well-sampled speaker exceeded a $\Delta r$ of 0.19.
• Domain Variation: Sensitivity varies by domain. Media discourse showed the highest pipeline robustness ($|\Delta r| = 0.32$ under LLM), while Finance and Academia were the most sensitive ($0.46$).

2. Cross-Method Disagreement is the Dominant Threat

• Magnitude of Disagreement: Measurement method divergence is significantly larger and more systematic than pipeline divergence. Across all 41 speakers, the mean $\Delta_{method}$ (0.76) is nearly double the mean $\Delta_{pipeline}$ (0.41).
• Systematic Sign Reversal: LLM and keyword-lexicon methods assign opposite coupling directions (sign reversals) in 49% of cases. This occurs even in well-sampled speakers (e.g., Rogoff with 17 videos) where both methods are internally stable. For instance, for Rogoff, the pipeline divergence was 0.19, but the method divergence on the same VTT pipeline was 0.90.
• Aggregate Masking: Aggregate valence proportions remain highly stable ($|\Delta p(neg)| < 6$ percentage points) regardless of pipeline or method. This stability masks the substantial instability present in cross-dimensional correlations.

3. Domain Modulates Sensitivity

Professional communication styles modulate how preprocessing choices propagate to measurements. For example, academic hedging markers and media intensifiers exhibit different robustness to sentence fragmentation (a byproduct of VTT segmentation) compared to geopolitical discourse.

Key Contributions

1. Diagnostic Framework: The paper introduces a two-source diagnostic framework that separates preprocessing pipeline effects from measurement method effects, providing a practical method for researchers to report and analyze both sources of instability.
2. Empirical Bounds on Pipeline Sensitivity: It provides evidence that pipeline sensitivity is predictable. For adequately sampled speakers ($N \ge 16$), the impact of preprocessing choices on correlational conclusions is bounded ($|\Delta r| \approx 0.13$).
3. Identification of Measurement Validity as Primary Threat: The study demonstrates that cross-method disagreement is a more severe validity threat than preprocessing divergence. Conclusions regarding the coupling of affective valence and epistemic modality can be systematically altered by the choice of annotation instrument, even when the preprocessing pipeline is held constant.

Significance and Claims

The paper argues that researchers studying cross-dimensional relationships in interview data must verify that their conclusions are robust to both sources of variation. The authors claim that:

• Preprocessing choices matter, but are bounded: While pipeline choices affect results, the magnitude of this effect is predictable and manageable for speakers with sufficient data.
• Measurement choice is critical: The choice between LLM annotation and keyword lexicons can fundamentally reverse scientific conclusions (e.g., changing a positive correlation to a negative one).
• Reporting Standards: CSS researchers should explicitly separate these two sources of instability in their reporting. A correlation that survives both alternative preprocessing pipelines and alternative measurement methods is deemed more credible than one validated within a single pipeline-method combination.

The authors maintain a modest stance, acknowledging limitations such as the use of a single LLM provider, an English-only corpus, and the inability to fully isolate diarization effects from sentence segmentation effects in their current design. They do not claim that one method is universally "correct," but rather that the divergence itself is a diagnostic signal requiring investigation.