VozConsultAI: A Provider-Agnostic Architecture and a Clinically Grounded Evaluation of Open-Weight Large Language Models for SOAP Note Extraction in Brazilian Portuguese Telehealth

Technical Summary: VozConsultAI

Problem Statement
Clinical documentation consumes a significant portion of healthcare professionals' time, serving as a documented driver of burnout, a challenge exacerbated by the rapid expansion of telehealth in large public systems like Brazil's Unified Health System (SUS). Currently, approximately 80% of electronic health record (EHR) data remains unstructured narrative text, limiting interoperability and clinical decision support. In the Brazilian context, this issue is compounded by a lack of domain-adapted tools for Portuguese and strict data sovereignty constraints under the General Data Protection Law (LGPD). Commercial ambient scribes are predominantly English-centric, proprietary, and require routing sensitive patient data to third-party cloud infrastructure, which conflicts with legal requirements for data residency and institutional control.

Methodology and Architecture
The paper presents VozConsultAI, an end-to-end solution designed to transcribe teleconsultation audio and automatically structure the text into SOAP (Subjective, Objective, Assessment, Plan) notes in Brazilian Portuguese. The system is built on three core methodological pillars:

1. Provider-Agnostic Microservice Architecture: The system employs a "Transcription Broker" to decouple the telehealth application from the execution engine. Key design principles include:

   • Decoupling: Separation of the external boundary (broker), orchestration (gateway), and execution (workers).
   • Reliability: Implementation of per-requester FIFO ordering via consistent hashing, at-least-once delivery neutralized by idempotent processing, and automatic recovery of orphaned jobs via lease-based failure detection.
   • Abstraction: An adapter layer normalizes responses from heterogeneous Automatic Speech Recognition (ASR) providers (e.g., Google, AWS, Azure, OpenAI, or self-hosted Whisper) into a canonical format. This allows for A/B testing and ensures that sensitive audio can remain on-premises to satisfy LGPD requirements.

2. SOAP Extraction Strategies: The project investigates three strategies for converting transcripts to structured notes:

   • Prompt Engineering: Using instruction-following LLMs with structured JSON schemas and strict constraints to prevent hallucination.
   • Hybrid Classification-Generation: Segmenting transcripts into sentences, classifying them into SOAP categories using fine-tuned BERT-family encoders, and then consolidating them via a generative model.
   • Fine-Tuned Generation: Adapting local generative models on synthetic and real transcript-to-SOAP pairs using parameter-efficient methods (LoRA/QLoRA) to fit public sector hardware constraints.

3. Clinically Grounded Evaluation Protocol: Moving beyond traditional n-gram overlap (ROUGE/BLEU) or generic semantic similarity, the authors propose a safety-weighted evaluation framework based on three orthogonal dimensions:

   • Semantic and Structural Correctness (SSC): Measures if generated content is semantically correct and assigned to the proper SOAP section.
   • Information Coverage and Completeness (ICC): Measures the fraction of reference information captured, penalizing omissions.
   • Clinical Grounding Index (CGI): Explicitly penalizes hallucination by verifying that every generated unit has explicit textual support in the original transcript.
   • These are consolidated into a Clinical Documentation Quality Index (CDQI), which weights CGI heavily (4:3:3 ratio) to prioritize safety.

Experimental Results
The authors conducted the first comparative benchmark of open-weight Large Language Models (LLMs) for SOAP extraction in Brazilian Portuguese. The study evaluated five models (gpt-oss-120b, Llama 3.3 70B, Mistral Small, Mistral 7B, and DeepSeek V3.2) across 30 synthetic teleconsultation transcripts from six medical specialties.

• Performance: DeepSeek V3.2 achieved the highest consolidated quality (CDQI: 0.87) and the highest clinical grounding (CGI: 0.96). Llama 3.3 70B matched DeepSeek's structural correctness (SSC) and conservative CDQI despite having fewer parameters.
• Error Analysis: The results indicate that model scale alone does not dictate performance. The primary differentiators were Information Coverage (ICC) and Grounding (CGI), not sentence classification (SSC).
• Failure Modes: The Assessment and Plan sections accounted for the majority of errors, specifically hallucinations (inventing diagnoses not stated by the clinician) and misclassifications (recording therapeutic hypotheses as decided interventions). Smaller models (e.g., Mistral 7B) struggled significantly with information coverage, often compressing multiple complaints into a single summary and omitting clinically relevant negatives.

Significance and Claims
The paper claims to provide a viable path for public health systems to adopt automated clinical documentation without compromising data sovereignty or relying on proprietary English-centric vendors.

• Architectural Contribution: The provider-agnostic microservice architecture demonstrates that public institutions can switch between self-hosted open-weight models and commercial cloud engines via configuration, turning data sovereignty from a system redesign constraint into a routing policy.
• Evaluation Contribution: The proposed evaluation protocol addresses a critical gap in clinical text generation metrics by explicitly penalizing hallucination, a failure mode that traditional metrics often miss or even reward due to vocabulary overlap.
• Benchmark Contribution: This work establishes the first baseline for open-weight LLMs in Brazilian Portuguese clinical documentation, showing that models can achieve useful quality levels (CDQI up to 0.87) on institutional hardware.
• Practical Implication: While open-weight models show promise, the authors emphasize that the Assessment and Plan sections still require close professional review due to the high risk of clinical extrapolation. The system is designed to draft notes for review, not to replace clinical judgment.

The authors note limitations, including the use of synthetic data which may not fully capture the disfluency and acoustic noise of real teleconsultations, and the fact that the evaluation isolated the extraction stage from upstream ASR errors. Future work involves validating the system on real-world data and integrating it with the e-SUS APS platform.