Slogans or Stance? A Label-Light Diagnostic for Entrepreneurial-Discourse Measurement on Chinese SOE Speeches

Technical Summary: "Slogans or Stance? A Label-Light Diagnostic for Entrepreneurial-Discourse Measurement on Chinese SOE Speeches"

Problem Statement
Empirical research in management and corporate governance frequently relies on textual indices derived from corporate speeches to measure constructs like "entrepreneurial spirit." Standard methodologies—keyword dictionaries, latent topic models (e.g., LDA), and pre-trained sentence embeddings—are designed to measure topical coverage (how much a document discusses a topic) rather than substantive stance (the speaker's specific strategic position).

This distinction is critical in performative discourse genres, such as speeches by leaders of Chinese centrally administered state-owned enterprises (SOEs). These speeches coexist with operational reporting (e.g., specific R&D programs, divestitures) and canonical political invocations (e.g., slogans like "cultivate world-class enterprises"). Current methods often conflate the two, as hand-curated dictionaries and topic models are heavily driven by the high-frequency, interchangeable political slogans that dominate these texts, failing to isolate the leader's unique strategic stance from the institutional ritual.

Methodology
The authors propose a "label-light" diagnostic to evaluate whether measurement instruments capture leader-level stance or merely recurrent political symbolism. They utilize a natural experiment within a corpus of 80 speeches from 51 central SOEs (2018–2021). The dataset includes:

• 24 "Leader-Change" pairs: Same company, different speaker (across two interview waves).
• 5 "Same-Leader" pairs: Same company, same speaker (longitudinal).

The core hypothesis is that a valid stance extraction method should produce significantly larger vector distances between different leaders of the same firm than between the same leader across time. If a method captures industry topics or shared political rituals, the distances should be indistinguishable.

Experimental Setup
The study benchmarks four method families on 2,190 text segments:

1. Dictionary (DICT): Token-based matching against seed words for five dimensions (Innovation, Competition–Cooperation, Organisation–Market, Social Responsibility, National Mission).
2. Latent Dirichlet Allocation (LDA): Unsupervised topic modeling with $K=5$, mapped to dimensions via seed words.
3. Sentence Encoder (BGE): Cosine similarity between segment embeddings and dimension descriptions using the BAAI/bge-small-zh-v1.5 model.
4. Large Language Model (LLM): A zero-shot 9B parameter model (Qwen3.5:9b) prompted to output:
   • A stance label (SLOGAN, SUBSTANTIVE, IRRELEVANT).
   • Substantive confidence ($c_{sub}$).
   • Per-dimension stance scores.
   • Slogan density estimates.

A Confidence-Weighted Calibration (CALIBRATED) variant is introduced, which weights the LLM's raw stance scores by its self-reported substantive confidence and penalizes slogan density.

Key Results
The evaluation focuses on two channels: Paired-Contrast (leader-change vs. same-leader distance) and Gold F1 (consistency with a human-annotated pilot set).

• Baseline Failures:

  • LDA: Failed to distinguish leader changes. The Cohen's $d$ was 0.20 with a 95% CI crossing zero $[-0.72, 1.20]$. Topics recovered were industry clusters (e.g., oil, automotive) rather than stance dimensions.
  • Dictionary: While showing a high Cohen's $d$ (0.81), the method was anti-correlated with the "Substantive" gold label in the pilot (F1 $\approx$ 0.5). It scored political slogans higher than operational narratives.
  • BGE: Produced a Cohen's $d$ of 0.65, but on document vector distances of order $10^{-3}$. The embedding space was too collapsed (homogeneous) to support meaningful downstream discrimination.

• LLM Performance:

  • The zero-shot Qwen3.5:9b achieved a Cohen's $d$ of 1.09 (exact permutation $p_1 = 0.034$) and a Gold F1 of 0.78.
  • The Calibrated version increased the absolute paired-contrast difference ($\Delta$) by 27% (from 0.105 to 0.130) but reduced Cohen's $d$ to 0.83 due to increased variance.
  • Ablation: The performance lift was driven almost entirely by the substantive confidence multiplier ($c_{sub}$). Slogan-density penalties contributed negligibly.

• Nuanced Findings:

  • Style vs. Stance: When regressing out five doc-level stylometric features (sentence length, numeric density, etc.), the LLM's Cohen's $d$ dropped from 1.09 to 0.43 (non-significant). This suggests roughly half of the measured "leader-change" effect is consistent with individual leader idiolect (style) rather than pure strategic stance.
  • Dimension Sensitivity: The "Organisation–Market" dimension was the most discriminative across all methods, while "National Mission" was the hardest, likely due to its formulaic nature.
  • Paraphrase Robustness: Rewriting substantive content into slogan style increased Dictionary scores by 33% but decreased Calibrated LLM scores, confirming the LLM's ability to distinguish form from substance.

Significance and Claims
The paper does not claim to have solved the measurement of "entrepreneurial orientation" definitively. Instead, it frames its contribution as a measurement-validity diagnostic for performative corporate discourse.

1. Diagnostic Utility: The leader-change paired evaluation provides a way to test if a method captures the speaker's stance or the firm's institutional ritual without requiring expensive, large-scale human annotation.
2. Methodological Critique: Standard tools (Dictionaries, LDA, generic Embeddings) are shown to be insufficient for separating stance from symbolism in this specific genre. They recover industry topics or are driven by slogan frequency.
3. LLM as a Tool, Not a Panacea: While zero-shot LLMs outperform baselines, the authors caution that:
   • The "Gold" labels match the LLM's own prompt rules, meaning F1 measures rule consistency, not external construct validity.
   • A significant portion of the signal may be stylistic (idiolect) rather than strategic.
   • Pre-training overlap between the LLM and the Chinese SOE corpus is a potential confound.

Conclusion
The authors release the scored corpus, the slogan lexicon, and the evaluation harness to enable further research. They conclude that while LLMs offer a superior signal for distinguishing leaders in performative discourse, researchers must remain cautious about attributing these signals solely to strategic stance, as style and institutional templates remain strong confounds. The paper advocates for a shift from treating corporate text as transparent disclosure to analyzing it as a mix of performative and substantive content.