CarpeDiem, a per-day clinical parameters and pneumonia adjudication dataset for critically ill patients with suspected pneumonia

The CarpeDiem dataset provides a unique, de-identified resource of 21,931 patient-hospital days from 704 critically ill, mechanically ventilated patients with suspected pneumonia, featuring daily clinical parameters and physician-adjudicated pneumonia episodes to enable detailed analysis of clinical trajectories beyond traditional admission and discharge metrics.

The Gist

Imagine you are trying to understand why a car broke down during a long road trip. Most mechanics only look at two things: where the car started (the admission) and where it ended up (the discharge). They ask, "Did it start with a flat tire?" and "Did it make it to the destination?"

But this approach misses the whole story. It ignores the flat tire that happened on day three, the engine overheating on day five, or the moment the driver finally fixed the problem on day seven.

This paper is about building a "Black Box" recorder for critically ill patients with pneumonia.

Here is the breakdown of what the researchers did, using simple analogies:

1. The Problem: The "Snapshot" vs. The "Movie"

Most medical studies take a snapshot of a patient when they walk into the hospital and another when they leave. They miss the "movie" of what happened in between.

• The Issue: Pneumonia (a serious lung infection) is tricky. It's not just a simple "yes or no" diagnosis. Sometimes a patient looks sick but doesn't have pneumonia; sometimes they have it but look fine. Doctors have to guess based on symptoms, X-rays, and lab tests, which isn't always perfect.
• The Gap: If we only look at the start and end, we don't see how the disease evolved day by day. We miss the small changes that lead to a cure or a tragedy.

2. The Solution: The "CarpeDiem" Dataset

The researchers created a new database called CarpeDiem (Latin for "Seize the Day").

• The Analogy: Instead of a snapshot, this dataset is like a high-definition, day-by-day diary for 704 patients.
• How it works: For every single day a patient was in the ICU, the researchers recorded everything:
  • Vitals: Their heart rate, temperature, and breathing (like checking the car's speedometer and temperature gauge).
  • Support: Were they on a ventilator (a machine breathing for them)? What settings were used?
  • Labs: Blood tests, white blood cell counts, and other chemical markers.
  • The "Gold Standard" Check: They performed a special procedure called a Bronchoalveolar Lavage (BAL). Think of this as sending a tiny, high-tech robot down the patient's windpipe to wash out the lungs and bring back a sample of the fluid. This tells them exactly what bacteria or virus is causing the infection.

3. The "Referees": Expert Adjudication

Data is messy. A computer might see a fever and think "Pneumonia," but a human doctor knows it might be a heart problem.

• The Analogy: To make sure the data is accurate, the researchers hired a team of expert referee doctors (like a panel of judges in a sports game).
• The Job: These doctors looked at every single patient's chart and the BAL results. They didn't just guess; they debated and voted on:
  • "Is this actually pneumonia?"
  • "Is it bacterial, viral, or something else?"
  • "Did the treatment work, or did the patient get worse?"
• The Result: They created a "truthful" label for every day of the patient's stay, turning messy hospital notes into a clean, reliable dataset.

4. What's Inside the Box?

The dataset contains 21,931 patient-days. That's a massive amount of information.

• Demographics: Who are these people? (Age, gender, background).
• Daily Features: Every single lab result and vital sign, organized by day.
• The Outcome: Did they go home? Did they pass away? Did they need a lung transplant?
• Privacy: Just like a bank, all names and dates were scrubbed. The data is "de-identified," meaning you can study the patterns without knowing who the specific patients are.

5. Why Does This Matter?

This dataset is a time machine for medical research.

• For AI and Computers: It allows scientists to train Artificial Intelligence to spot patterns humans miss. For example, "If a patient's white blood cell count drops on day 3 and their temperature spikes on day 4, they are likely to fail extubation (coming off the breathing machine)."
• For Doctors: It helps them understand that treating pneumonia isn't a one-time event; it's a dynamic battle that changes every 24 hours.
• For the Future: It helps design better drugs and treatments by showing exactly when and how patients get better or sicker.

In a Nutshell

The authors took a complex, messy, and often confusing medical situation (severe pneumonia in the ICU) and turned it into a clear, day-by-day storybook. By combining raw data with expert human judgment, they gave researchers a powerful tool to finally "seize the day" and understand the daily struggles of critically ill patients, hopefully leading to better cures and fewer deaths in the future.

Technical Summary

1. Problem Statement

Current analyses of critically ill patients often rely on static "admission-to-discharge" models, which collapse the dynamic nature of intensive care unit (ICU) stays into single data points. This approach overlooks:

• Daily Evolution: The day-to-day changes in patient status, intercurrent complications, and therapeutic interventions.
• Pneumonia Complexity: Pneumonia is a syndromic entity difficult to adjudicate using structured data alone (e.g., ICD codes) due to low sensitivity/specificity of coding, subjective clinical notes, and overlapping conditions (e.g., heart failure, alveolar hemorrhage).
• Data Granularity: Existing large ICU databases (e.g., MIMIC, eICU) lack expert-validated, longitudinal pneumonia episode adjudication and high-frequency daily clinical parameters aligned with multidisciplinary rounds.

2. Methodology

The authors developed the CarpeDiem framework and dataset based on the Successful Clinical Response in Pneumonia Therapy (SCRIPT) study.

• Cohort Selection:
  • Source: Northwestern Memorial Hospital (Chicago, IL) Medical ICU.
  • Inclusion: 704 patients (enrolled June 2018 – May 2023) requiring mechanical ventilation and undergoing bronchoalveolar lavage (BAL) for suspected pneumonia.
  • Exclusion: Prior lung transplant, prisoners, pregnancy, or unsafe bronchoscopy.
• Data Acquisition & Integration:
  • Sources: Integrated data from the Enterprise Data Warehouse (EDW/Epic EHR), REDCap (research team surveys), and manual chart reviews.
  • Granularity: Data is structured at the patient-hospital-day level (21,931 total rows), mirroring daily multidisciplinary rounds.
  • Processing:
    • Aggregation of multiple daily measurements (vitals, labs) using mean, min, or max values based on clinical "worseness."
    • Harmonization of units (e.g., lactate mg/dL to mmol/L).
    • Removal of outliers based on physiologically reasonable ranges.
    • De-identification via HIPAA Safe Harbor rules (dates replaced with relative time; ages >89 capped at 91).
• Clinical Adjudication (The "Gold Standard"):
  • A panel of five Pulmonary/Critical Care physicians independently reviewed charts.
  • Process: Two independent reviewers per chart; discrepancies resolved by a third reviewer or weekly consensus meetings.
  • Outputs: Classification of pneumonia type (CAP, HAP, VAP), etiology (bacterial, viral, mixed, negative), and outcome (cured, indeterminate, not cured) at standardized follow-up intervals (days 7, 10, 14 post-BAL).
• Data Validation:
  • Manual integrity checks against the front-end EHR.
  • Automated de-identification checks using tools like Presidio, Philter, and ClinDeID.
  • Cross-referencing of >50 patient records for consistency.

3. Key Contributions

• The CarpeDiem Dataset (v1.9): A unique, de-identified resource containing 21,931 patient-days with:
  • Demographics & Outcomes: 44 columns (age, sex, comorbidities, discharge disposition).
  • Daily Clinical Features: 113 columns covering vitals, hemodynamics, ventilator settings (PEEP, FiO2), ABGs, and extensive laboratory panels (WBC, CRP, procalcitonin, etc.).
  • BAL Data: Cell differentials and pathogen results (JSON formatted).
  • Adjudicated Episodes: 39 columns detailing pneumonia episode type, etiology, and cure status.
• Expert Adjudication Layer: Unlike public datasets relying on billing codes, this dataset includes rigorous, multi-reviewer physician adjudication of pneumonia episodes and treatment success, addressing the "ground truth" gap in pneumonia research.
• Longitudinal Framework: Shifts the analytical paradigm from static admission features to dynamic, daily trajectories, enabling the study of intercurrent events and treatment response over time.
• Open Science Infrastructure: Full code availability on GitHub for data cleaning, synthesis, and analysis; documentation of data quality workflows (Smartsheet/Issue tracking).

4. Results & Data Characteristics

• Cohort Demographics:
  • Median age: 62 years (IQR 51–71); 60% male.
  • Racial/Ethnic diversity: 20% Black/African American, 20% Hispanic/Latino.
  • Median intubation duration: 11 days.
  • Unfavorable Outcomes: 45% (death, hospice, or lung transplant).
• Pneumonia Episodes:
  • Total Episodes: 835 total episodes across the cohort.
  • Types: 153 Community-Acquired (CAP), 265 Hospital-Acquired (HAP), 417 Ventilator-Associated (VAP).
  • Etiology: 317 Bacterial, 207 Viral (including 210 COVID-19 cases), 67 Other Viral, 114 Non-Pneumonia Controls.
  • Outcomes: 390 Cured, 168 Indeterminate, 277 Not Cured.
• Data Completeness:
  • Missing data is context-aware (e.g., ventilator parameters are missing post-extubation; BAL results only appear on procedure days), reflecting clinical reality rather than random error.

5. Significance

• Research Utility: The dataset enables advanced machine learning applications, such as:
  • Unsupervised clustering of daily clinical states to identify distinct patient trajectories.
  • Dynamic prediction models for outcomes (e.g., next-day extubation, mortality).
  • Analysis of antibiotic de-escalation patterns and treatment response.
  • Investigation of specific subgroups (e.g., neutropenic patients, immunocompromised hosts).
• Clinical Impact: By providing a high-fidelity, daily-resolution view of pneumonia, the dataset supports the development of precision medicine tools that can adapt to the evolving state of critically ill patients, potentially improving early detection of treatment failure and optimizing resource allocation.
• Reproducibility: The publication of the full data pipeline, adjudication protocols, and code repositories ensures that the dataset is transparent, reproducible, and extensible for the broader scientific community.

In summary, the CarpeDiem dataset bridges the gap between raw electronic health record data and clinically meaningful, expert-validated longitudinal insights, offering a powerful new resource for understanding and treating severe pneumonia in the ICU.