Technical Summary: CANDI – Contextual Alignment for Niche Domains Question Answering

Problem Statement

The increasing deployment of Large Language Models (LLMs) in specialized, high-stakes domains such as medical diagnostics and financial advisory reveals a critical gap: while models possess general knowledge, they often fail to achieve contextual alignment. This refers to a model's ability to adapt its responses to the specific context of a conversation, including the user's domain discipline, professional role, background expertise, and the exact topic at hand.

Traditional QA benchmarks (e.g., SQuAD, Natural Questions) focus on open-domain fact retrieval and offer limited insight into how models handle the nuanced, role-aware, and situational reasoning required in enterprise applications. Specifically, the paper addresses the challenges faced by Multi-Tiered System of Supports (MTSS) teams in schools for Behavioral Health. These multidisciplinary teams (including administrators, counselors, teachers, and clinicians) often lack immediate access to expert coaches due to resource constraints. Current LLMs struggle to provide the precise, context-sensitive guidance needed for these teams to implement MTSS frameworks effectively across three tiers of intervention (Universal, Targeted, and Intensive).

Methodology: CANDI-QA

To address this gap, the authors introduce CANDI-QA (Contextual Alignment for Niche Domains Question Answering), a novel dataset and evaluation framework designed to test LLMs in the MTSS(B) domain.

1. Dataset Curation

The dataset comprises 315 expert-curated questions derived from the School Behavioral Health Academy (SBHA), categorized into two distinct types:

• T1: Information Assistance Questions (90 items): Direct, factual queries requiring precise information extraction from expert literature (e.g., best practices, evidence-based protocols). These have "gold-standard" answers provided by human experts.
• T2: Applied Inference Questions (225 items): Complex, multi-hop reasoning questions based on specific student cases or scenarios. These require inference based on role-specific cues and situational context to generate actionable insights. These questions do not have single gold-standard answers.

2. Prompting Strategies

The study evaluates model performance across three prompting strategies to determine the impact of context:

• P1 (No-Context): The model receives only the user query (Zero-shot).
• P2 (Role-Aware): The system prompt includes the user's specific role (e.g., Teacher, Counselor) and a description of that role.
• P3 (Task-Aware): The system prompt includes the AI's high-level objective, the user's role, and the role description, situating the query within the broader task of MTSS implementation.

3. Evaluation Framework

The evaluation employs a dual-pronged approach tailored to the question types:

• For T1 Questions: Reference-based metrics are used against gold-standard answers, including Cosine Similarity, BERTScore, MoverScore, and BLEURT.
• For T2 Questions: A Retrieval-Augmented Generation (RAG) evaluation framework using the RAGAs library is employed. Since gold answers are unavailable, responses generated by the Qwen model serve as the "retrieved context" for evaluation. Metrics include Context Precision, Response Relevancy, Faithfulness (factual consistency with context), and Semantic Similarity.

4. Baseline Solution: MTSS-Net

To establish a robust baseline, the authors propose MTSS-Net, a lightweight neuro-symbolic framework combining neural retrieval with rule-based reasoning. It features two variants:

• v1: Encodes QA instances as structured JSON objects (Task type, Role, Intervention Tier, Policy references) to support symbolic rule-based reasoning.
• v2: Enhances v1 with curated QA pairs, explicit role conditioning, and source attribution to improve transparency and grounded reasoning.

Experimental Setup

The authors evaluated 10 diverse language models, ranging from compact open-source systems (e.g., LLaMA 3.2 1B, Phi-4-mini 3.8B) to state-of-the-art proprietary and larger open-source models (e.g., Qwen3 32B, LLaMA 4 variants, Deepseek R1). All models were tested across the three prompting strategies (P1, P2, P3) on the full CANDI-QA dataset.

Key Results

T1 Performance (Information Assistance)

• Model Size Matters: Larger models generally outperformed smaller ones. Qwen3 (32B) consistently achieved the highest semantic alignment (Cosine Similarity: ~0.43) across all metrics.
• Prompting Impact: Role-aware prompting (P2) yielded modest improvements over zero-shot (P1). However, adding task-level context (P3) provided only marginal gains for straightforward factual queries, suggesting that for T1 tasks, the primary value lies in the model's inherent knowledge retrieval rather than complex contextual framing.
• Lowest Performers: Smaller models like LLaMA 3.2 (1B) and Phi-4-mini (3.8B) struggled significantly compared to their larger counterparts.

T2 Performance (Applied Inference)

• Context is Critical: All models showed significant improvement when moving from P1 to P3 (Task-Aware) prompting.
• Top Performer: Mistral 7B Instruct achieved the best results in the T2 category, particularly in Faithfulness (0.8014) and Semantic Similarity (0.8336) under the P3 strategy.
• Reasoning Capability: The results indicate that T2 tasks require more than simple retrieval; they benefit heavily from structured, role- and task-aware prompting to support grounded, scenario-sensitive reasoning.

Significance and Claims

The paper positions CANDI-QA as a critical new benchmark for advancing research in context-aware and user-aligned language models. Its primary contributions and claims include:

1. Highlighting the Limitations of Current LLMs: The study demonstrates that strong performance on general benchmarks does not guarantee alignment with expert preferences in niche domains. Current models struggle with true contextual alignment without enhanced integration or symbolic reasoning.
2. The Necessity of Contextual Alignment: The authors argue that for enterprise and educational applications, aligning responses to the user's specific role and the conversation's context is not optional but a necessity for real-world utility.
3. Value of Hybrid Approaches: The introduction of MTSS-Net underscores the potential of neuro-symbolic approaches (combining neural networks with rule-based logic) to provide transparent, interpretable, and grounded reasoning in high-stakes domains where "black box" LLMs may be insufficient.
4. Structured Data Benefits: The findings suggest that augmenting QA datasets with structured context (e.g., JSON-based role and task metadata) significantly improves model performance and interpretability, particularly for inference-heavy tasks.

The authors conclude that while prompting strategies (specifically P3) offer improvements, achieving robust, trustworthy AI for high-stakes domains like Behavioral Health requires moving beyond simple prompting toward hybrid architectures and more rigorous, domain-specific evaluation methodologies. The paper commits to releasing the CANDI-QA dataset, MTSS-Net baseline, and evaluation scripts to support reproducibility and future research.