Semantic Risk Scoring of Aggregated Metrics: An AI-Driven Approach for Healthcare Data Governance

This paper proposes an AI-driven framework that leverages CodeBERT embeddings and XGBoost to perform static, explainable risk scoring of SQL-based metric definitions, enabling healthcare institutions to proactively identify and mitigate privacy risks in aggregated data before deployment while ensuring regulatory compliance and auditability.

The Gist

Imagine a massive hospital as a giant library. Inside this library, there are different teams: the Doctors (who know about patient health), the Fundraisers (who need to know about donors), and the Operations Team (who manage wait times).

The Problem: The "Glass Wall"

In the past, if the Fundraisers wanted to know something about patients to help their campaigns, they had to ask the Doctors for the raw files. But these files are like glass boxes containing everyone's private secrets (names, addresses, medical conditions). Laws like HIPAA say, "You cannot open these boxes and hand them over."

So, the teams hit a wall. They can't share data, and they can't make good decisions.

The Current "Fix": The Summary Report

To solve this, the hospital started creating Summary Reports (Aggregated Metrics). Instead of giving the Fundraisers a list of 10,000 patients, the Doctors give them a single number: "The average wait time in the ER is 20 minutes."

This is great! It's safe, right? Not always.

Imagine a summary report that says: "There is exactly one 85-year-old woman with a rare disease living in Zip Code 90210." Even though it's a "summary," it actually reveals exactly who that person is. This is called a Privacy Leak.

The Solution: The "AI Safety Inspector"

This paper proposes a new, smart tool to act as a Virtual Safety Inspector for these summary reports. Before a report is ever published, this AI checks the "recipe" (the SQL code) used to make it to see if it's dangerous.

Here is how the AI works, using a simple analogy:

1. The Grammar Teacher (SQL Parser)

First, the AI reads the recipe like a grammar teacher reading a sentence. It breaks the sentence down into parts (Subject, Verb, Object) to understand the structure. It asks: "Are you trying to group people by their Zip Code? Are you counting them by Gender?"

2. The Mind Reader (CodeBERT)

Sometimes, two recipes look different but mean the same dangerous thing.

• Recipe A: "Group by Zip Code."
• Recipe B: "Group by City + Street Name."

A simple rule-finder might miss Recipe B. But our AI has a Mind Reader (called CodeBERT). It understands the intent behind the code. It knows that "City + Street" is just as dangerous as "Zip Code" because both narrow down the group too much. It turns the recipe into a "fingerprint" to see if the idea is risky.

3. The Checklist (Syntactic Features)

The AI also pulls out a physical checklist. It counts:

• How many tables are being mixed?
• Are there "sensitive" ingredients (like Birth Date or Medical Codes) in the mix?
• Is the group size too small?

4. The Judge (XGBoost Classifier)

The AI combines the "Mind Reader's" understanding and the "Checklist" results and hands them to a Judge (an XGBoost model).

• The Judge looks at all the evidence and gives the recipe a Risk Score from 0 to 1.
• Score 0.2? Safe. "Go ahead, publish the report."
• Score 0.9? Dangerous! "Stop! This recipe will reveal private secrets."

5. The Translator (Explanation Engine)

If the Judge blocks a recipe, the system doesn't just say "No." It acts like a Translator and explains why in plain English:

"We blocked this because you are grouping by 'Gender' and 'Diagnosis Code.' This creates a group so small that someone could guess exactly who that patient is."

Why This Matters

• Old Way: You wait until someone publishes a bad report, then you panic and try to fix it. (Like waiting for a cake to burn before checking the oven).
• New Way: The AI checks the recipe before you even turn on the oven. It stops the fire before it starts.

The Result

Now, the Fundraisers can get the insights they need (like "How many donors are in the cardiology department?") without ever seeing a single patient's name or medical record. The hospital stays safe, follows the law, and everyone can work together without fear.

In short: This paper builds a smart, automated guard that checks every data summary to make sure it doesn't accidentally spill anyone's secrets, allowing hospitals to share knowledge safely.

Technical Summary

Here is a detailed technical summary of the paper "Semantic Risk Scoring of Aggregated Metrics: An AI-Driven Approach for Healthcare Data Governance."

1. Problem Statement

Large healthcare institutions operate multiple Business Intelligence (BI) teams (e.g., clinical, fundraising, operations) that require data collaboration. However, strict regulations like HIPAA, FERPA, and IRB prevent the sharing of raw, patient-level data containing Protected Health Information (PHI) and Personally Identifiable Information (PII).

To mitigate this, organizations use aggregated metric tables (precomputed summaries like "average wait times"). While these abstractions protect raw data, they introduce a new risk: statistical disclosure. Even aggregated metrics can inadvertently reveal sensitive information if:

• They are grouped by fine-grained quasi-identifiers (e.g., ZIP code, gender, specific diagnosis codes).
• The group sizes (N) are too small, allowing for re-identification or attribute inference.

Current governance solutions rely on static, rule-based systems (e.g., hard-coded blacklists of column combinations). These approaches lack context awareness, are rigid, cannot adapt to evolving metric definitions, and fail to provide explainable reasoning for why a query is risky.

2. Methodology

The authors propose a modular, AI-driven framework that performs static analysis on SQL-based metric definitions before execution. The system evaluates the risk of overexposure using a hybrid approach combining semantic understanding and syntactic structural analysis.

The workflow consists of five key stages:

A. SQL Parsing (AST Generation)

The system parses the input SQL query into an Abstract Syntax Tree (AST). This breaks the query down into structural components, identifying:

• Tables and joins involved.
• Columns used in SELECT, GROUP BY, and WHERE clauses.
• The logical structure of the aggregation.

B. Semantic Representation (CodeBERT)

To capture the intent and logic of the query (beyond just syntax), the system uses CodeBERT, a pre-trained transformer model fine-tuned on programming languages.

• The SQL query is converted into a dense vector embedding (a "semantic fingerprint").
• This allows the model to recognize that different syntactic structures (e.g., grouping by "zip" vs. "city + street") may pose similar privacy risks.

C. Syntactic Feature Extraction

Parallel to semantic encoding, the system extracts specific, hand-crafted features known to correlate with privacy risks:

• Count of GROUP BY columns.
• Number of table joins.
• Presence of sensitive columns (e.g., dob, zip, gender, diagnosis_code) based on a keyword list.

D. Risk Classification (XGBoost)

The semantic embeddings and syntactic features are concatenated into a single feature vector and fed into an XGBoost classifier.

• Training: The model is trained on a dataset of labeled SQL queries (Safe vs. Risky).
• Output: The classifier outputs a continuous risk probability score (0.0 to 1.0).
• Thresholding: A configurable threshold (e.g., 0.85) determines the final status. Scores above the threshold trigger a BLOCK status; scores below trigger an APPROVED status.

E. Explanation Engine

If a query is blocked, the system generates a human-readable explanation by mapping the AST structure to predefined templates.

• Example Output: "Metric may overexpose sensitive groupings like gender or ZIP."
• This ensures audit readiness and helps data stewards understand why a query failed.

3. Key Contributions

• Pre-Execution Governance: Unlike traditional tools that check data at runtime or after publication, this system validates metrics at the definition layer (SQL query level), preventing privacy violations before they occur.
• Hybrid AI Architecture: It uniquely combines CodeBERT (for semantic understanding of intent) with XGBoost (for structural risk assessment), overcoming the limitations of purely rule-based or purely deep-learning approaches.
• Explainability: The system moves beyond binary "pass/fail" decisions by providing natural language explanations for risk scores, fostering trust and compliance.
• Modularity: The framework is designed to be plug-and-play within existing BI pipelines, supporting Role-Based Access Control (RBAC) and zero-trust architectures.

4. Experimental Results

The system was evaluated using a synthetic healthcare dataset (simulating patient data with fields like patient_id, dob, gender, zip, diagnosis_code) and three representative SQL queries:

Query Type	Query Logic	Risk Score	Status	Outcome
Medium Risk	GROUP BY zip	0.87	BLOCKED	Correctly identified ZIP as a quasi-identifier with potential small group sizes.
High Risk	GROUP BY gender, diagnosis_code	0.93	BLOCKED	Flagged the combination of sensitive attributes leading to high re-identification risk.
Moderate Risk	GROUP BY gender	0.74	APPROVED	Nuanced Decision: Unlike a rigid rule engine that might block all gender grouping, the AI assessed the context and determined the risk was low (likely sufficient group size).

Comparison to Rule-Based Systems:

• Context Awareness: The AI correctly approved GROUP BY gender (Query 3) because it understood the statistical context, whereas a rule-based system would likely flag it indiscriminately.
• Granularity: The system provides a continuous risk score (0–1) rather than a binary outcome, allowing for more nuanced governance.
• Transparency: The system provided specific reasons for blocking, whereas rule-based systems often offer no rationale.

5. Significance and Future Work

Significance:
This framework addresses a critical gap in healthcare data governance by enabling safe cross-departmental collaboration. It allows non-clinical teams (e.g., fundraising) to access actionable insights without compromising patient privacy or violating federal regulations. By shifting from "access or deny" to "intelligent, explainable privacy controls," the system supports the modernization of healthcare data infrastructure while maintaining strict compliance with HIPAA and IRB standards.

Limitations & Future Directions:

• Keyword Dependency: Current sensitive field detection relies on fixed keyword lists; future work aims to use semantic similarity to detect synonyms (e.g., "postal_code" vs. "zip").
• Static Analysis: The system cannot execute queries to check actual group sizes (cardinality). Future iterations may incorporate metadata-driven heuristics to estimate distribution patterns.
• Complex SQL: Performance may degrade with advanced constructs like CTEs or deeply nested subqueries; the parser requires enhancement to handle production-grade complexity.
• Bias: The model's performance depends on the training data; broader calibration across diverse healthcare systems is needed to ensure fairness.

In conclusion, this paper presents a robust, AI-driven solution for Semantic Risk Scoring, transforming how healthcare institutions manage aggregated data by making privacy protection proactive, explainable, and scalable.