CAST: Achieving Stable LLM-based Text Analysis for Data Analytics

1. Problem Statement: The Stability Gap in TADA

The paper addresses a critical limitation in applying Large Language Models (LLMs) to Text Analysis for Data Analysis (TADA). TADA involves transforming unstructured text columns in tabular data into structured representations (e.g., tags or summaries) to enable downstream analytics like filtering, grouping, and aggregation.

• The Core Conflict: Data analytics requires deterministic outputs (the same input must yield the same structured result every time) to ensure reproducibility. However, LLMs are inherently probabilistic, generating diverse outputs even with identical inputs and decoding settings (temperature=0).
• The Consequence: In a TADA workflow, if a review is labeled "Customer Service" in one run and "Support Team" in another, downstream aggregation results change, undermining trust and reproducibility.
• The Root Cause: The authors attribute this instability to unconstrained latent reasoning trajectories. Without guidance, the model traverses high-entropy paths, leading to semantically drifting but superficially plausible outputs.

2. Methodology: The CAST Framework

The authors propose CAST (Consistency via Algorithmic Prompting and Stable Thinking), a framework designed to constrain the model's reasoning path to reduce entropy and enforce stability without sacrificing quality.

CAST operates on two complementary principles:

A. Algorithmic Prompting (AP)

• Concept: AP imposes a procedural scaffold over the reasoning process. Instead of asking the LLM to "summarize this," the prompt forces the model to follow a specific, deterministic algorithmic workflow (e.g., "Decompose query → Identify domain → Cluster items → Generate summary").
• Mechanism: It acts as a "gating function" that prunes or down-weights non-compliant reasoning transitions. This restricts the generation to a subspace of valid paths ($Z_C \subseteq Z$), effectively lowering the conditional entropy of the reasoning steps.

B. Thinking-before-Speaking (TbS)

• Concept: TbS enforces explicit intermediate commitments. The model must generate well-defined intermediate states (e.g., domain definition, topic schema, cluster assignments) before producing the final output.
• Mechanism: By committing to these intermediate states, the model conditions subsequent generation on a fixed structure rather than free-form generation. This collapses the branching factor of the reasoning tree early, ensuring that later steps remain coherent with the initial commitments.

Implementation:
CAST is implemented as a single structured API call where the LLM outputs a JSON object containing both the intermediate reasoning states and the final structured result. The framework includes a self-validation step where the model checks its output against the constraints defined in the prompt.

3. Evaluation Metrics

Recognizing that standard metrics (like ROUGE or BLEU) measure quality but not stability, the authors introduce a new stability-focused evaluation suite:

• CAST-S (for Summarization): A hybrid metric combining:
  • Semantic Score ($S_{sem}$): Measures content overlap using LLM-based semantic matching.
  • Positional Score ($S_{pos}$): Measures ordering agreement using Kendall's Tau. In tabular analytics, the order of bullet points matters for consistency; this metric penalizes reordering.
• CAST-T (for Tagging): Designed for tasks without gold labels. It clusters tags from multiple runs based on semantic equivalence and calculates the Majority Ratio (the proportion of runs converging to the dominant semantic cluster).

4. Key Contributions

1. Formalization of TADA: Defined TADA as a tabular-centric paradigm where stability is a functional necessity, not just a quality metric.
2. The CAST Framework: Introduced a novel method to stabilize LLMs via Algorithmic Prompting and Thinking-before-Speaking, reducing the entropy of latent reasoning paths.
3. Stability Metrics: Developed CAST-S and CAST-T, validated against human judgments, to rigorously quantify run-to-run consistency.
4. Empirical Validation: Demonstrated that CAST achieves superior stability across multiple LLM backbones (GPT-5.2, DeepSeek-V3.2, Gemini-3-Flash) while maintaining or improving output quality.

5. Experimental Results

The authors evaluated CAST against baselines including Zero-shot CoT, Few-shot CoT, Self-Consistency (SC), and ablated versions of CAST (AP-only, TbS-only).

• Stability Gains: CAST consistently achieved the highest stability scores across all models.
  • Summarization: Improved Stability Score by up to 16.2% compared to baselines.
  • Tagging: Achieved the best stability in joint tagging tasks (e.g., 9.60/10 for GPT-5.2).
• Quality Preservation: CAST did not regress in accuracy. In fact, for GPT-5.2 and Gemini-3-Flash, it achieved the highest accuracy in independent tagging tasks and higher recall in summarization compared to Zero-shot CoT.
• Efficiency: Unlike Self-Consistency (which requires multiple sampling runs and voting), CAST operates in a single API call. It is significantly faster than SC (e.g., 6.60s vs. 13.32s for Gemini-3-Flash) and comparable to standard prompting methods.
• Ablation Studies: Both AP and TbS components were found to be necessary; the full CAST framework outperformed either component used in isolation.

6. Significance and Impact

• Bridging the Gap: CAST provides a practical solution for integrating probabilistic LLMs into deterministic data analytics pipelines (OLAP), making them reliable for production use.
• Cost-Effectiveness: By achieving stability through structural constraints rather than expensive multi-trajectory search (like Tree-of-Thoughts or Self-Consistency), CAST offers a highly efficient alternative.
• Generalizability: The framework is model-agnostic and has been validated across diverse models (OpenAI, DeepSeek, Google, Anthropic, Meta) and multilingual datasets.
• Future Direction: The paper suggests that "coarse-grained" algorithmic flows are often sufficient for stability, opening avenues for automated discovery of optimal reasoning scaffolds.

In conclusion, CAST demonstrates that by explicitly constraining the reasoning path and forcing intermediate commitments, LLMs can be made reliable, stable, and efficient operators for structured data analysis tasks.