Scale-to-Dialogue: Low-Burden Elicitation of Daily Premenstrual Symptom Ratings with Small Language Models

Technical Summary: Scale-to-Dialogue for Premenstrual Symptom Elicitation

Problem Statement
Prospective daily tracking of premenstrual symptoms is essential for health assessment but imposes a significant response burden due to the repetitive nature of ordinal rating forms (e.g., the Daily Record of Severity of Problems). While Computerized Adaptive Testing (CAT) reduces burden by selecting fewer items, conversational systems offer a distinct advantage: a single natural language response can provide evidence for multiple structured fields. However, prior work has not established the extent to which item burden can be reduced while preserving the accuracy of daily premenstrual symptom labels using locally deployable, small language models (SLMs).

Methodology
The study formulates the conversion of rating scales to dialogue as an ordinal label-recovery problem. The system actively elicits a small set of symptom clusters and maps responses to original six-level severity labels (0: not at all to 5: very high).

• Dataset: The author utilized 3,320 complete participant-days from the mcPHASES dataset, covering six symptom domains: cramps, mood swings, fatigue, sleep issues, stress, and bloating. The data was split into a development set (6 participants) and a frozen evaluation set (36 participants, 360 days, 2,160 item labels).
• Model Architecture: The pipeline employs two small language models:
  1. Evidence Gate: A fine-tuned ModernBERT-base binary classifier detects whether a specific symptom is expressed in a user's utterance. It was trained on a corpus of generated utterances (chatty, hesitant, event narrative, brief styles) to generalize beyond hand-written templates.
  2. Severity Scorer: A Qwen2.5-1.5B-Instruct model performs deterministic structured scoring, outputting JSON-formatted ordinal scores based on system prompts defining six ordinal anchors.
• Dialogue Policies: Five policies were evaluated to compare question count against label agreement:
  1. Fixed Six-Item: One targeted question per symptom (baseline).
  2. Fixed Three-Cluster: Three questions, each targeting a pair of symptoms (mood/stress, fatigue/sleep, cramps/bloating).
  3. Open-Plus-Item: An open narrative followed by targeted questions for unresolved items.
  4. Open-Plus-Cluster: An open narrative followed by cluster questions for unresolved items.
  5. Opening-Only: Scoring based solely on the initial narrative.
• Evaluation Metrics: Primary outcome was Quadratic Weighted Kappa (QWK). Secondary metrics included Mean Absolute Error (MAE), exact agreement, agreement within one severity level, and recall for moderate-or-higher symptoms (labels 3–5). Statistical significance was assessed via participant-cluster bootstrap analysis.

Key Results

• Evidence Detection: Training the evidence gate on generated corpora significantly improved performance over hand-written templates, achieving a recall of 0.749 on unseen event narratives. On the frozen evaluation set, the gate achieved 0.616 recall and 0.858 precision.
• Policy Performance:
  • Fixed Six-Item: Achieved the highest accuracy (QWK 0.976) with 6 questions.
  • Fixed Three-Cluster: Reduced the question count by 50% (3 questions) while maintaining high agreement. It achieved a QWK of 0.913, 97.45% agreement within one severity level, and 80.94% recall for moderate-or-higher symptoms.
  • Open-First Policies: These did not improve the efficiency-agreement trade-off. The "Open-Plus-Cluster" policy required 3.92 questions and achieved a lower QWK (0.886) than the fixed three-cluster policy.
• Statistical Comparison: The bootstrap analysis estimated a QWK difference of −0.062 (95% CI −0.076 to −0.048) between the three-cluster and six-item strategies. The confidence interval crossed the −0.05 non-inferiority margin, indicating a measurable trade-off in precision for reduced burden.
• Error Analysis: The primary source of error in the cluster policy was the "fatigue" and "sleep" pairing. While stress, mood swings, bloating, and cramps showed high QWK (>0.92), fatigue had a QWK of 0.696. Errors often involved shifting zero to low severity or downgrading moderate fatigue to low severity, likely due to the linguistic overlap between descriptions of poor sleep and low energy.

Key Contributions

1. Demonstrated Feasibility of Local SLMs: The paper establishes that small, locally deployable models (ModernBERT and 1.5B parameter instruction models) can effectively recover structured ordinal labels from natural language conversations.
2. Controlled Measurement Framework: It introduces a rigorous framework for evaluating "scale-to-dialogue" conversion, isolating the effects of evidence gating, joint scoring, and question policies on a locked participant split.
3. Efficiency-Accuracy Trade-off Quantification: The study quantifies the specific cost of reducing question burden. It shows that a fixed three-cluster strategy can halve the daily question load while retaining high ordinal agreement (97.45% within one level) and improving recall for moderate-to-severe symptoms compared to the six-item baseline.
4. Design Guidance: The results suggest that constrained, topic-level dialogue (asking specific cluster questions) is more reliable than unconstrained open-ended prompts for structured assessment, as open prompts often fail to cover all necessary domains without extensive follow-up.

Significance
The paper claims that active, structured cluster-level elicitation provides a direct route from natural conversation to reusable daily symptom labels without requiring large, cloud-based models. The significance lies in the operational finding that daily premenstrual symptom tracking can be compressed from six items to three active questions with minimal loss in ordinal precision. This approach reduces the "recurring data-entry task" burden on users while maintaining the data quality necessary for prospective health assessment. The author positions this as a step toward integrating symptom capture into ordinary self-expression rather than requiring a separate reporting ritual, though they note that the precision trade-off (quantified by the bootstrap interval) must be accepted to achieve the 50% reduction in question count.