💬 NLP

Conversation as Measurement in Clinical Encounters: Observable Phase Structure, Partially Observable Patient State

This paper investigates the observability of patient state and conversational phase structure in clinical encounters using 439 transcripts and PROM data, finding that while phase structure is reliably observable from transcripts, patient state remains only partially recoverable, highlighting the limitations of inferring human state from conversation alone.

Original authors: Lily Chen, Ted Mau, Michael Gensheimer, Brian Anthony Nuyen, Nancy Jiang, James Zou

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lily Chen, Ted Mau, Michael Gensheimer, Brian Anthony Nuyen, Nancy Jiang, James Zou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but you only have one piece of evidence: a transcript of a conversation. You can read every word the suspects said, but you can't see their faces, hear their tone of voice, or know what they are thinking. This is the world of "conversational AI," where computers try to understand human feelings, health, and intentions just by reading text. Scientists have long wondered: Is a transcript a complete map of a person's inner world, or is it just a blurry, partial sketch? If a computer reads a chat between a doctor and a patient, can it truly know how sick the patient feels, or is it just guessing based on the words that happened to be spoken? This question matters because we are starting to use AI to monitor mental health, track disease, and even grade conversations, assuming that if it's not in the text, it doesn't exist. But what if the text is missing the most important clues?

A team of researchers from Stanford University decided to put this assumption to the test using a real-world laboratory: the doctor's office. They treated clinical conversations like a scientific experiment to see what is "observable"—meaning, what can be recovered from the transcript alone. They looked at two very different things: the structure of the visit (like the chapters in a book) and the state of the patient (how they actually feel inside). To do this, they gathered 439 real transcripts from ENT (ear, nose, and throat) visits, covering 134 hours of conversation. Crucially, they had a "gold standard" to compare against: the patients had also filled out official surveys about their voice, coughing, and swallowing problems right before seeing the doctor. These surveys acted like a truth-telling anchor, allowing the researchers to see if the AI could guess the survey answers just by reading the conversation transcript.

The researchers used a powerful AI model (a PHI-compliant version of GPT-5) to act as a super-fast annotator, breaking the transcripts into phases and trying to guess the survey scores. To make sure the AI wasn't just making things up, a human expert spent 40 hours manually checking the AI's work, ensuring the results were solid.

Here is what they found, and it reveals a fascinating split in the nature of conversation.

The Good News: The Map is Clear
When it came to the structure of the visit, the AI was a superstar. The researchers found that the "phase structure" of a doctor's visit is highly observable. Just like a play has distinct acts—opening, rising action, climax, and curtain call—a doctor's visit follows a predictable pattern. The AI could reliably identify when the doctor was building rapport, taking a history, doing a physical exam, giving an assessment, or making a plan.

The Pattern: The AI correctly identified the phases 94.8% of the time.
The Insight: It could even spot differences between doctors. For example, one doctor spent a lot of time on the physical exam (checking the throat), while another spent more time on education and planning. The transcript clearly showed the "shape" of the visit, revealing how different doctors organize their time and how patients and doctors trade questions during specific parts of the visit. In short, the skeleton of the conversation is fully visible in the text.

The Bad News: The Soul is Hidden
However, when the researchers asked the AI to guess the patient's state (how much pain they were in, how embarrassed they felt, or how much their symptoms bothered them), the results were much more humble. Even though the visit was specifically designed to get the patient to talk about their symptoms, the transcript was only a partial window into their reality.

The Missing Pieces: The AI had to "abstain" (refuse to guess) on 63% of the survey questions because the patient simply never mentioned them in the conversation. For example, a patient might feel deeply embarrassed by their cough, but if they never say "I feel embarrassed," the transcript has no record of it.
The Mismatch: Even when the AI did make a guess, it only agreed moderately with the patient's actual survey answers. The AI tended to underestimate how bad the symptoms were.
The Nuance: Some things were easier to spot than others. Concrete problems like "my throat hurts when I swallow" were often mentioned. But abstract or emotional feelings, like "I feel left out of conversations because of my voice," were frequently missing from the text, even though the patient reported them on the survey.

The Big Takeaway
The paper concludes with a crucial warning for anyone building AI that tries to read human minds from text. There is an observability asymmetry: You can reliably see the structure of a conversation (the phases, the flow, the roles), but you cannot reliably see the internal state of the people having the conversation.

The transcript is like a movie script; it tells you exactly what lines were spoken and in what order. But it doesn't tell you the actor's internal monologue, their hidden fears, or the things they were too shy to say. The researchers suggest that relying only on transcripts to judge human health or emotion is risky. If you want to know how a patient truly feels, you can't just read the transcript; you need the external anchor of a survey or a more direct way to ask. The text captures the dance, but it often misses the dancer's heartbeat.

Technical Summary: Conversation as Measurement in Clinical Encounters

Problem Formulation

Modern AI systems increasingly rely on conversational traces to infer human interaction patterns and internal states (e.g., emotions, functional status). These systems operate under an implicit assumption: that such information is recoverable from conversation alone. However, the limits of this assumption remain unclear. The paper identifies two primary challenges in assessing observability (defined as whether a target is recoverable from conversational transcripts alone):

Partial View: For targets not inherent to the conversation (e.g., thoughts, specific symptom burdens), relevant information may never be explicitly expressed, making the absence of evidence difficult to interpret without an external anchor.
Annotation Ambiguity: Large-scale analysis requires model-based annotation. Consequently, apparent limits of observability may reflect either true limits of the conversational signal or errors introduced by the annotator, making it difficult to disentangle the two.

To address these challenges, the authors study clinical encounters, which provide an external anchor through Patient-Reported Outcome Measures (PROMs) and follow broadly structured conversational patterns. The study investigates observability along two dimensions:

Phase Structure: The organization of conversation into sequential phases (e.g., history, exam, plan).
Patient State: Aspects of a patient's condition, operationalized via item-level scores from PROMs regarding voice, cough, and swallowing.

Methodology

Dataset

The study utilizes 439 real-world clinical encounter transcripts spanning 134 hours, recorded and transcribed using Nuance Dragon Ambient eXperience (DAX). The dataset includes:

Clinicians: 4 providers (3 ENT specialists, 1 radiation oncologist).
PROM Pairings: 245 ENT transcripts paired with 273 PROM surveys.
Instruments: Voice Handicap Index-10 (VHI-10), Cough Severity Index (CSI), and Eating Assessment Tool-10 (EAT-10). Each contains 10 items scored on a 0–4 ordinal scale.

Annotation Pipeline

To ensure scalability while maintaining rigor, the authors employed a PHI-compliant GPT-5 deployment for three tasks:

Phase Segmentation: Splitting transcripts into 3–8 semantic segments based on communicative purpose (e.g., Opening rapport, History, Physical exam, Assessment, Plan, Education, Closing, Non-clinical).
Question Extraction: Identifying information-seeking questions and attributing them to the speaker.
Transcript-Derived PROM Scoring: Predicting scores for each PROM item based on transcript evidence. The model was instructed to abstain (output null) if no direct evidence was present, rather than hallucinating a score.

Validation and Metrics

To mitigate the risk that observed limits reflect annotator error, the authors conducted 40 hours of manual validation by a single reviewer on a subset of the data:

Phase Validation: 212 phases from 29 transcripts were reviewed for label correctness and timestamp consistency.
Question Validation: 100 questions were reviewed for existence and speaker attribution.
PROM Validation: 120 items from 12 transcript-survey pairs were reviewed to verify if transcript-derived scores (or abstentions) were supported by evidence.

Evaluation Metrics:

Phase/Question: Binary correctness (accuracy).
PROM Scores: Quadratic Weighted Kappa (QWK), Mean Absolute Error (MAE), and Mean Signed Error (External PROM – Transcript-derived PROM) calculated on non-missing items.

Key Results

1. Observability of Phase Structure

The study finds that phase structure is highly observable and recoverable from transcripts alone.

Validation Accuracy: Manual review confirmed 94.8% correctness for phase labels and 99.0% for question extraction and speaker attribution.
Structural Insights: Recovered phases revealed coherent encounter organization:
- Temporal Trajectories: Encounters follow a broad progression (History $\to$ Exam $\to$ Plan/Education), though with structured heterogeneity (e.g., some encounters are exam-dominant, others education-dominant).
- Clinician Variation: Different clinicians exhibit distinct allocation patterns (e.g., ENT specialists spend more time on exams; radiation oncologists spend more time on planning and education).
- Interactional Roles: Questioning patterns vary by phase; clinicians dominate questioning in History and Exam, while patients ask more questions in Plan and Education.

2. Partial Observability of Patient State

In contrast, patient state is only partially observable, even in a setting specifically designed to elicit symptoms.

Evidence Support: In the manually validated subset, all transcript-derived scores and abstentions were judged to be supported by transcript evidence, confirming the model did not hallucinate scores.
Missingness: Across the full dataset, 63.0% of PROM items were missing (i.e., the model abstained due to lack of evidence). Missingness varied by instrument (52% for Swallowing to 67.1% for Cough) and by specific item.
- Item-Level Variation: Items concerning subjective, affective, or social impacts (e.g., "feeling handicapped," "embarrassment") were frequently missing, whereas concrete functional difficulties (e.g., "difficulty being heard") were more often verbalized.
Agreement with Ground Truth: For non-missing items, agreement with external PROM scores was only moderate (Overall QWK: 0.467 in validation; 0.508 in full set).
- The model tended to underestimate symptom burden (positive Mean Signed Error), suggesting that even when symptoms are discussed, the transcript may not capture the full severity reported in surveys.
- The Cough instrument showed particularly low agreement (QWK: -0.019), though this was based on a small number of non-missing items.

Significance and Claims

The paper's core contribution is the identification of an observability asymmetry in clinical conversations:

Phase structure is recoverable and useful: Transcripts provide direct evidence of how an interaction unfolds, making them a promising modality for characterizing encounter organization, clinician behavior, and interactional roles without manual annotation at scale.
Patient state is only partially recoverable: Even in a clinical setting optimized for symptom elicitation, transcripts are an incomplete record of the underlying construct. Much of the PROM-relevant information is not verbalized, and the information that is verbalized often lacks the granularity to match patient-reported scores.

Implications:
The authors caution against relying solely on transcript-based inference for human state assessment. When the condition of "sufficiently informative evidence" does not hold, external anchors (such as surveys) or additional modalities are necessary for valid measurement. The study reframes conversation not as a complete window into patient state, but as a measurement modality with specific, item-dependent limits.

Limitations Acknowledged:

Data is limited to a specific clinical setting (ENT/Oncology) and a small set of PROMs.
Validation relied on a single annotator.
Absence of gold-standard transcripts means ASR errors could contribute to apparent missingness.
Transcripts omit prosody, gesture, and other contextual cues.
PROMs serve as anchors but are not a complete ground truth for all patient experiences.