💬 NLP

Human-AI Co-design for Clinical Prediction Models

The paper introduces HACHI, an iterative human-in-the-loop framework that leverages AI agents to rapidly explore and refine interpretable concepts from clinical notes through expert feedback, thereby accelerating the development of more accurate and generalizable clinical prediction models while highlighting the critical role of human oversight in guiding the AI.

Original authors: Jean Feng, Avni Kothari, Patrick Vossler, Andrew Bishara, Lucas Zier, Newton Addo, Aaron Kornblith, Yan Shuo Tan, Chandan Singh

Published 2026-06-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jean Feng, Avni Kothari, Patrick Vossler, Andrew Bishara, Lucas Zier, Newton Addo, Aaron Kornblith, Yan Shuo Tan, Chandan Singh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build the ultimate, easy-to-use checklist for a doctor to predict if a patient is going to get sick. Traditionally, building this checklist is like trying to assemble a complex piece of furniture while blindfolded, with a team of experts arguing over which screws to use, how tight to turn them, and which instructions to follow. It takes forever, requires a massive team, and often, the final product is too complicated to actually use in a busy hospital.

This paper introduces a new way to build these checklists called HACHI. Think of HACHI as a "Human + AI" dance partnership. Instead of the humans doing all the heavy lifting and the AI just sitting there, they work together in a loop where the AI does the heavy exploring, and the humans steer the ship.

Here is how the HACHI partnership works, using simple analogies:

The Problem: The "Infinite Library"

Doctors have access to millions of pages of unstructured notes (like handwritten scribbles or typed paragraphs) about patients. These notes contain an "infinite library" of potential clues (concepts) about what makes a patient sick.

The Old Way: A human team tries to read through these notes and guess which clues matter. They might miss the most important ones or pick ones that look good on paper but don't work in real life.
The AI-Only Way: You could let a super-smart AI (a Large Language Model) read all the notes and pick the clues. But the AI might get confused, pick weird clues (like "the doctor wrote in blue ink" instead of "the patient has a fever"), or miss the big picture because it doesn't understand the "why" behind the medicine.

The Solution: The HACHI Dance

HACHI solves this by having the AI and the humans take turns in a loop.

The AI Agent (The Scout): The AI is sent into the "infinite library" of notes. Its job is to rapidly scan, find potential clues, and turn them into simple Yes/No questions (e.g., "Does the note say the patient has a headache?"). It then tests these questions to see which ones help predict the sickness best.
The Human Team (The Coaches): The humans look at the AI's results. They act like coaches. They might say:
- "Hey, you picked a clue about 'note-writing style,' but that's cheating. Let's only look at the patient's actual symptoms."
- "You missed the surgery type! That's a huge factor. Go look for that."
- "This clue is too vague. Let's make the question more specific."
The Loop: The humans give this feedback, and the AI goes back out to find better clues based on the new instructions. They repeat this a few times (usually 3 or 4 rounds) until the checklist is perfect.

Real-World Examples from the Paper

The team tested this on two specific medical problems:

1. Predicting Brain Injuries in Kids (TBI)

The Goal: Decide which kids with head injuries need a CT scan (which uses radiation) and which don't.
The AI's First Mistake: The AI initially thought "Does the note mention the Glasgow Coma Scale?" was a good clue. The humans said, "No! That's just a clue about how the doctor wrote the note, not the patient's condition." They also found the AI was accidentally using data from patients who had already been scanned at other hospitals (data leakage).
The Fix: The humans told the AI to ignore note-writing styles and remove those specific patients.
The Result: The final checklist was simple, accurate, and even found a new clue the old methods missed: "Does the child have a normal walking gait?" (If a child walks normally after a head hit, they are likely fine).

2. Predicting Kidney Injury After Surgery (AKI)

The Goal: Predict if a patient will get kidney damage after general surgery.
The AI's First Mistake: The AI only looked at the patient's body (like their weight or blood pressure) and ignored the surgery itself.
The Fix: The humans told the AI, "Stop! The type of surgery matters just as much. Also, make your questions more specific so different doctors interpret them the same way."
The Result: The final checklist included things like "Is the surgery urgent?" and "Is the patient having a heart rate over 100?" It became much more accurate than previous models.

Why This Matters

The paper claims that HACHI is faster and better than the old ways because:

It's Fast: The team only spent about 1–2 hours per round reviewing the AI's work.
It's Clear: The final models are just simple lists of Yes/No questions that any doctor can understand and use without a computer.
It's Trustworthy: Because humans are in the loop, they catch the AI's mistakes (like data leaks or weird correlations) before the model is finished.

In short, HACHI is like having a super-fast, super-smart intern (the AI) who does all the research, while the senior doctors (the humans) guide the intern to make sure the final report is accurate, fair, and actually useful. The paper shows that this partnership creates better medical prediction tools than either the humans or the AI could build alone.

Technical Summary: Human-AI Co-design for Clinical Prediction Models (HACHI)

Problem Statement

The development of safe, effective, and practically useful Clinical Prediction Models (CPMs) traditionally requires iterative collaboration between clinical experts, data scientists, and informaticists. While simple, fully interpretable models (e.g., rule-based scores) are preferred in clinical practice for their ease of implementation and auditability, they are difficult to develop at scale. The process is time- and resource-intensive because it requires refining critical details such as feature selection and clinical category definitions.

This challenge is exacerbated when attempting to incorporate information from unstructured clinical notes. These notes contain an essentially infinite number of concepts (e.g., variations of "smoking" history), making manual exploration of all possible features infeasible. While recent Large Language Models (LLMs) can extract concepts and mimic clinical reasoning, existing AI-agent approaches often operate in isolation. Without external guidance from domain experts, these agents may learn concepts that are overly simplistic, lack clinical credibility, or suffer from data leakage, resulting in models that fail to translate into clinical practice. Furthermore, a "gulf of envisionment" exists where clinical teams initially lack the clarity to specify exact task instructions for an AI agent, necessitating an iterative process to align goals.

Methodology: The HACHI Framework

The authors introduce HACHI (Human+Agent Co-design for Healthcare Instruments), an iterative, human-in-the-loop framework designed to accelerate the development of fully interpretable CPMs. HACHI alternates between an AI agent exploring candidate concepts and a clinical AI team providing feedback to refine the learning process.

Core Components

Concept Definition: Concepts are defined as simple, human-interpretable yes/no questions (e.g., "Does the note mention the patient having...?") used within linear models.
The AI Agent Loop (Inner Loop): The agent performs a greedy hill-climbing procedure to select $k$ $k$ concepts that maximize predictive accuracy. This involves three steps:
- Initialize: The agent extracts keyphrases from clinical notes and proposes initial $k$ concept questions based on world knowledge and keyphrases.
- Propose: Using a bag-of-words model on training data, the agent identifies keyphrases associated with the outcome and proposes new candidate concept questions.
- Evaluate: The agent annotates clinical notes for each candidate concept, fits a CPM (using lasso-penalized logistic regression), and evaluates performance on a validation set. The best-performing concept replaces the incumbent.
The Clinical AI Team Loop (Outer Loop): The team reviews the results of the AI agent's round via a PHI-compliant web interface. They provide directional feedback by:
- Modifying free-text prompts (e.g., changing the prefix of questions to ensure they extract patient attributes rather than note-writing styles).
- Adjusting data inclusion criteria (e.g., removing patients with prior CT results to prevent leakage).
- Modifying statistical tools (e.g., adding sample weights to address site imbalances).
- Enforcing clinical priors (e.g., requiring coefficient signs to match clinical intuition).

Implementation

The framework was applied to two real-world tasks at UCSF:

Traumatic Brain Injury (TBI): Predicting TBI diagnosis in children presenting with head trauma.
Acute Kidney Injury (AKI): Predicting postoperative AKI in adult general surgery patients.

The process typically required 3–4 rounds of iteration to reach a satisfactory model.

Key Results

Case Study 1: Traumatic Brain Injury (TBI)

Performance: The final HACHI model achieved an AUC of 0.91 overall, outperforming the standard PECARN rule (AUC 0.75) and a single-round brainstorming approach (AUC 0.88).
Iterative Improvements:
- Round 1: Achieved a high AUC (0.92) but relied on spurious correlations (e.g., "note mentions GCS") and data leakage (patients with prior CT results).
- Round 2: After removing prior CT cases and refining prompts to focus on patient attributes, AUC dropped to 0.90, but concepts became clinically valid.
- Round 3: Coefficients were aligned with clinical priors (e.g., ensuring normal GCS had a negative coefficient).
- Round 4: Addressed generalizability issues between two hospital campuses (Oakland vs. Mission Bay) by introducing sample weights. This improved the Mission Bay AUC from 0.71 to 0.80 while maintaining overall performance.
Novelty: The final model included a novel feature: "Does the patient have a normal gait?" which was not in standard PECARN rules.

Case Study 2: Acute Kidney Injury (AKI)

Performance: The final HACHI model achieved an AUC of 0.73 on the internal validation set and 0.76 on a temporally disjoint test set (Period 2). This outperformed the Kheterpal model (AUC ~0.65) and a single-round brainstorming approach (AUC 0.70).
Iterative Improvements:
- Round 1: Focused only on patient characteristics (AUC 0.60).
- Round 2: Expanded to include surgical risk factors (e.g., surgery type), improving AUC to 0.70.
- Round 3: Refined concept definitions to be more precise (e.g., defining "active systemic infection" with examples) and included medication factors. This increased AUC to 0.73 and improved temporal generalizability.
Novelty: The model identified factors not commonly in interpretable AKI scores, such as "minimally invasive surgery" (protective), "tachycardia," and "sleep apnea."

Key Contributions

Framework for Co-design: HACHI demonstrates that meaningful collaboration between clinical AI teams and AI agents can produce high-performing, interpretable models that are superior to zero-shot AI approaches or traditional manual feature engineering.
Bridging the "Gulf of Envisionment": The framework addresses the initial lack of clarity in task specification by using an iterative feedback loop. This allows teams to gradually align the AI agent's capabilities with clinical objectives, refining prompts, data, and evaluation metrics over time.
Discovery of Novel Predictors: The process uncovered clinically relevant concepts often omitted from standard CPMs (e.g., gait in TBI, minimally invasive surgery in AKI) while recovering known predictors.
Efficiency: The framework is efficient, requiring only 1–2 hours of team review per round, with 3–4 rounds sufficient to reach a robust model.
Interpretability and Trust: By defining concepts as yes/no questions and allowing human review of every step, the framework ensures the resulting models are transparent, auditable, and clinically credible.

Significance and Claims

The paper claims that HACHI offers a practical pathway to developing local, interpretable CPMs that are tailored to specific patient populations and institutional contexts. Unlike complex black-box models, HACHI-derived models are simple enough for bedside implementation (mental arithmetic or quick chart review) and can be audited by domain experts.

The authors emphasize that human oversight is essential not just for validation, but for shaping the development process in ways AI agents cannot achieve alone—specifically in identifying data leakage, correcting spurious correlations, ensuring fairness across sites, and refining concept granularity. The framework is designed to be accessible, requiring only access to clinical notes, a PHI-compliant LLM, and guidance from a clinical AI team.

The paper concludes with modest claims regarding deployment: while retrospective validation showed strong performance, prospective validation in real-time clinical workflows is still required. Additionally, the quality of the models depends on the LLM's extraction accuracy and the specific documentation practices of the institution. The authors position HACHI as a step toward democratizing the creation of high-quality, interpretable clinical tools without the prohibitive resource costs of traditional development.