Improving Attributed Long-form Question Answering with Intent Awareness

1. Problem Statement

Large Language Models (LLMs) are increasingly used to generate comprehensive, knowledge-intensive scientific reports (Deep Research). However, current models are trained on diverse academic texts but lack exposure to the reasoning processes and authorial intents that guide the creation of these documents.

• The Gap: While humans write with specific intents (e.g., outlining, planning, structuring arguments, deciding why a citation is used), these high-level cognitive steps are invisible in the final text. Consequently, models learn to mimic writing styles but fail to explicitly model the thought process required for coherent, well-attributed long-form generation.
• The Challenge: Deep research systems must gather hundreds of sources, organize them into a coherent narrative, and properly attribute claims. Without explicit modeling of why information is included or how paragraphs relate to the overall argument, models often produce reports that are factually accurate but structurally weak, poorly attributed, or difficult for readers to navigate.

2. Methodology

The authors propose an Intent-Aware Writing Framework that explicitly incorporates two levels of intent into the generation process: Paragraph Intents and Citation Intents.

A. Intent Representation

The framework uses a structured, tag-based schema inspired by STaR (Self-Taught Reasoner) and ToW (Thoughts of Words). Intents are embedded directly into the text using inline tags with natural language rationales:

• Format: <begin intent> [Intent Type] Rationale <end intent>
• Paragraph Intents: Placed before a paragraph to define its functional purpose (e.g., Exposition, Definition, Argumentation, Compare-Contrast, Cause-Effect).
• Citation Intents: Placed between a sentence and its citation to explain the specific rhetorical role of the source (e.g., Background, Motivation, Uses, Extension, Comparison, Future Work).

B. Training and Inference Strategies

The paper explores two main approaches to integrating intent awareness:

1. Intent-Aware Inference (Test-Time Scaling):

   • Models are prompted to generate reports with embedded intent tags and rationales during inference.
   • This acts as a form of "verbalized planning," forcing the model to articulate its structural and attributional logic before or during text generation.

2. Intent-Aware Training (Supervised Fine-Tuning - SFT):

   • Data Generation: A large "teacher" model (Gemini-2.5-pro) generates high-quality synthetic reports with embedded intents.
   • SFT Variants: Smaller student models (e.g., Qwen3-8B, Llama3.1-8B) are fine-tuned on this data under different settings:
     • Intent-Explicit: The model learns to generate both the report and the intent tags.
     • Intent-Implicit: The model is trained on the report text only (tags removed), but the data was generated with intent awareness, allowing the model to internalize the structure without explicit output.
     • Intent-Multiview: The training data is decomposed into four sub-tasks (full intent, paragraph-only, citation-only, and no-intent) to reduce instruction complexity and improve generalization.

3. Key Contributions

• Novel Framework: Introduction of a dual-level intent schema (paragraph and citation) specifically designed for scientific deep research tasks, bridging the gap between human writing processes and LLM generation.
• Synthetic Data Pipeline: A method to distill "intent awareness" from large models into smaller models via SFT, enabling smaller models to match or exceed the performance of much larger baselines.
• Diagnostic Tool: The use of intent tags as a lens to analyze model behavior, revealing specific weaknesses (e.g., underuse of "Comparison" intents) compared to human writing patterns.
• User-Centric Validation: A user study demonstrating that explicit intent annotations significantly improve reader comprehension, navigation, and confidence in long-form scientific reports.

4. Experimental Results

Experiments were conducted on three benchmarks: SQA-CS-V2 (AstaBench), DeepScholar Bench, and ResearchQA.

A. Performance Improvements

• Large Models: Intent-aware inference improved performance across all commercial models (o3, Gemini-2.5-pro, Claude Opus-4).
  • Average Gain: +2.9 absolute points in macro-averaged metrics.
  • Attribution: Significant gains in citation precision (+5-7 points) and recall, indicating better source usage.
• Small Models: Intent-aware SFT yielded massive improvements for smaller models (4B-8B parameters).
  • Average Gain: +12.3 absolute points over baselines.
  • State-of-the-Art: The best small model variants (e.g., Qwen3-8B with Multiview SFT) surpassed the performance of large models like Gemini-2.5-pro on specific benchmarks.

B. Citation Behavior Analysis

• Candidate Usage: Models with intent awareness utilized a significantly higher portion of retrieved candidates (relevant papers) without sacrificing precision.
• Overlap: Small models trained with intent awareness showed citation patterns much closer to large models, suggesting the distillation of sophisticated retrieval strategies.

C. User Study

• Readability: Participants reading reports with explicit intent tags (e.g., "This paragraph outlines cause-effect relationships") reported higher confidence in understanding the content and deciding whether to read further.
• Scores: Intent-aware reports received significantly higher Likert scores (4.47 vs. 3.84 for paragraphs) compared to baseline reports, particularly for complex scientific concepts.

5. Significance and Future Directions

• Beyond Factuality: This work demonstrates that improving the structure and intent of generation is as critical as improving factual accuracy for long-form tasks. It shifts the focus from "what" the model says to "how" and "why" it says it.
• Efficiency: The ability to distill intent awareness into smaller models makes high-quality, attributed scientific research generation more accessible and cost-effective.
• Transparency: Explicit intent tags serve as a form of explainability, helping users navigate dense technical reports and verify the logical flow of arguments.
• Future Work: The authors suggest expanding intent schemas to include hierarchical structures (e.g., how paragraphs build on each other) and adapting these schemas for non-scientific domains (law, policy) where rhetorical functions differ.

In conclusion, the paper establishes that intent awareness is a critical, previously missing component in training LLMs for deep research. By explicitly modeling the author's intent during both inference and training, the system produces reports that are not only more accurate and well-attributed but also significantly more readable and useful for human researchers.