💻 computer science

Statistical Confidence in Functional Correctness: An Approach for AI Product Functional Correctness Evaluation

This paper proposes and validates the Statistical Confidence in Functional Correctness (SCFC) approach, a four-step methodology that bridges the gap between business requirements and statistical rigor to evaluate AI functional correctness by moving from simple point estimates to confidence intervals and capability indices.

Original authors: Wallace Albertini, Marina Condé Araújo, Júlia Condé Araújo, Antonio Pedro Santos Alves, Marcos Kalinowski

Published 2026-02-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Wallace Albertini, Marina Condé Araújo, Júlia Condé Araújo, Antonio Pedro Santos Alves, Marcos Kalinowski

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a new employee to drive a delivery truck for your company. You have a very specific rule: The driver must arrive at the destination on time at least 90% of the time.

In the old days of software (traditional coding), you could test the driver on a specific route, and if they made it, you knew they were good. But Artificial Intelligence (AI) is different. It's like hiring a driver who is a bit of a "genius but also a bit moody." Sometimes they drive perfectly; other times, they get distracted by a squirrel and take a wrong turn. You can't just test them once and say, "Okay, they're hired." You need to know: Is their "on-time" rate consistently high, or are they just lucky today?

This paper introduces a new way to test these "moody genius" AI drivers called SCFC (Statistical Confidence in Functional Correctness). Think of it as a Safety Scorecard that doesn't just give you a single grade, but tells you how much you can trust that grade.

Here is how the paper breaks it down, using simple analogies:

1. The Problem: The "One-Number" Trap

Currently, companies test AI by running it a few times and getting an average score.

The Trap: Imagine an AI says, "I am 85% accurate!" That sounds great, right? But what if on some days it's 99% accurate, and on other days it crashes to 60%? If you only look at the average (85%), you might hire a driver who is actually a huge risk.
The Paper's Solution: We need to stop looking at just the average and start looking at the variability (how much the performance swings).

2. The Four Steps of the SCFC Approach

The authors propose a four-step recipe to get a true "Safety Score."

Step 1: Set the "Speed Limit" (Specification Limits)

Before you even test the driver, you must agree on the rules.

The Analogy: You tell the driver, "You must arrive on time at least 90% of the time." This is your Lower Specification Limit (LSL).
Why it matters: Without a clear number, you can't measure success. Is 80% good? Is 95% good? You need to define the "passing grade" with the business owners first.

Step 2: The "Representative Test Drive" (Stratified Sampling)

You can't just test the driver on a sunny Tuesday morning. You need to test them in rain, snow, rush hour, and on empty roads.

The Analogy: The paper suggests Stratified Sampling. Instead of picking random test cases, you deliberately pick a mix that matches real life (e.g., 50% rainy days, 20% traffic jams, 30% clear roads).
Why it matters: If you only test in perfect weather, your AI will look great until it hits a real storm. This step ensures your test covers all the "weird" scenarios the AI might face.

Step 3: The "What If?" Simulation (Bootstrapping)

This is the magic math part. Since you can't test the AI a million times (it takes too long), you use a trick called Bootstrapping.

The Analogy: Imagine you have a deck of cards representing your test results. You shuffle them, pull a hand, calculate the score, put them back, and do it again 1,000 times.
The Result: Instead of getting one number (85%), you get a Range. You might find that 95% of the time, the AI's performance will be between 82% and 88%.
Why it matters: This gives you a Confidence Interval. It tells you, "We are 95% sure the real performance is somewhere in this range." It quantifies the uncertainty.

Step 4: The "Safety Margin" Score (Capability Index)

Finally, you combine the average performance and the range to get a single Capability Index ( $C_{pk}$ ).

The Analogy: Think of this as a Safety Buffer.
- Score < 1.0: The driver is dangerous. Their "worst-case" performance is below your 90% rule. Do not hire.
- Score = 1.0: The driver is on the edge. They might make it, but they are skirting the danger zone. Hire with extreme caution.
- Score > 2.0: The driver is a superstar. Even their worst days are way above the 90% rule. Hire immediately.

3. Real-World Examples from the Paper

The authors tested this on two real AI systems:

The Oil Rig Camera: An AI that looks at photos of an oil rig deck to see how much space is left for cargo.
- The Result: The average looked good (83%), but the "Safety Score" was low (1.12). It was barely passing. The paper says: "You can use this, but watch it closely because it's risky."
The Credit Card Fraud Detector: An AI that spots fake credit card charges.
- The Result: The average was high (99%), and the "Safety Score" was excellent (1.98). Even in its worst moments, it was still catching almost all the fraud. The paper says: "This one is safe to deploy."

4. What Did the Experts Think?

The authors interviewed four AI experts (like senior engineers) to see if this method made sense.

The Verdict: They loved it! They said it fills a huge gap.
The "Aha!" Moment: They realized that just saying "90% accuracy" is misleading. Knowing that the accuracy could drop to 88% changes the decision entirely.
The Only Hiccup: Some experts thought the math (Bootstrapping) sounded a little scary at first, but once they realized it's just a computer simulation that doesn't require re-running the whole AI, they were fine with it.

The Big Takeaway

This paper argues that we need to stop treating AI like a simple calculator (where 2+2 always equals 4) and start treating it like a probabilistic system (where 2+2 is usually 4, but sometimes 3.9 or 4.1).

By using this SCFC approach, companies can move from guessing ("I think this AI is good") to knowing ("We are 95% confident this AI will meet our safety standards"). It turns a vague feeling of trust into a hard, statistical fact.

1. Problem Statement

The evaluation of Artificial Intelligence (AI) systems faces a fundamental paradigmatic incompatibility: traditional software testing relies on deterministic input-output mappings, whereas AI systems are inherently probabilistic and adaptive.

Limitations of Current Standards: While standards like ISO/IEC 25059 provide a quality model for AI, they lack practical, statistically robust methods for assessing Functional Correctness. Existing approaches often rely on point estimates (e.g., a single accuracy average), which mask performance variability and create a false sense of security.
The Oracle Problem: In AI, the "correct" output is often undefined or only approximable within a probabilistic range, making it difficult to define clear pass/fail criteria.
The Gap: There is a lack of empirical methods to translate business requirements into statistical confidence intervals that account for both the model's average performance and its variability, specifically to determine if a system is ready for deployment.

2. Methodology: The SCFC Approach

The authors propose the Statistical Confidence in Functional Correctness (SCFC) approach, a four-step methodology designed to operationalize functional correctness assessment.

Step 1: Define Functional Correctness Specification Limits

Objective: Translate business needs into objective, measurable acceptance criteria.
Mechanism: Establish a Lower Specification Limit (LSL) (e.g., minimum recall) or an Upper Specification Limit (USL) (e.g., maximum false positive rate).
Example: "The system must achieve a fraud recall of at least 95%" ($LSL = 0.95$).

Step 2: Perform Stratified and Probabilistic Sampling

Objective: Ensure the test data represents the real-world production distribution to avoid misleading conclusions.
Mechanism: Use stratified sampling rather than simple random sampling. Domain experts identify stratification variables (e.g., transaction type, region) to ensure minority but critical subgroups are adequately represented. Elements within strata are selected probabilistically.

Step 3: Apply Bootstrapping and Calculate Confidence Intervals

Objective: Quantify the uncertainty and variability of the performance metric.
Mechanism: Apply bootstrapping (non-parametric resampling with replacement) to the test sample.
- Generate $S$ resamples (e.g., $S=1000$ ) of size $n$ .
- Calculate the performance metric for each resample to create an empirical distribution.
- Derive a 95% Confidence Interval (CI) (e.g., $[87\%, 91\%]$ ) to define the range where the true system performance likely lies.

Step 4: Calculate and Analyze the Capability Index ( $C_{pk}$ )

Objective: Synthesize average performance and variability into a single decision indicator.
Mechanism: Adapt the Six Sigma $C_{pk}$ formula. Unlike traditional $C_{pk}$ $C_{p k}$ which assumes normality and uses standard deviation, SCFC uses the Confidence Interval bounds derived from bootstrapping.
- Formula for Lower Limit: $C_{pl} = \frac{\text{Mean} - \text{LSL}}{\text{Mean} - \text{Lower Bound of CI}}$
- Formula for Upper Limit: $C_{pu} = \frac{\text{USL} - \text{Mean}}{\text{Upper Bound of CI} - \text{Mean}}$
- Final Index: $C_{pk} = \min(C_{pl}, C_{pu})$
Interpretation:
- $C_{pk} < 1.0$ : Unacceptable (Variability exceeds safety margin).
- $C_{pk} = 1.0$ : Minimum Capability (System is at the limit).
- $C_{pk} > 2.0$ : Excellent (High confidence in meeting specs).

3. Key Contributions

Operationalization of ISO/IEC 25059: Provides a concrete, statistical framework to implement the abstract "Functional Correctness" sub-characteristic defined in international standards.
Shift from Point Estimates to Statistical Confidence: Moves the industry conversation from "What is the accuracy?" to "What is the statistical confidence that the accuracy meets the business requirement?"
Non-Parametric Capability Index: Introduces a $C_{pk}$ variant that does not assume a normal distribution of errors, making it robust for diverse AI performance metrics.
Empirical Validation: Demonstrates the approach through real-world industrial case studies and validates its utility via expert interviews.

4. Results

The approach was evaluated using two real-world industrial cases and validated by four AI experts.

Case Study 1: Cargo Deck Space Estimation (Oil & Gas)

Goal: Estimate free space on oil platforms (Semantic Segmentation).
Requirement: $LSL = 70\%$ (predictions within acceptance range).
Result: Observed mean accuracy was 83.4%. However, the 95% CI was $[71.4\%, 92.9\%]$ .
$C_{pk}$ Calculation: 1.12.
Insight: While the mean (83%) looked safe, the $C_{pk}$ of 1.12 revealed the lower bound (71.4%) was dangerously close to the limit (70%). The system was deemed deployable only with continuous monitoring.

Case Study 2: Credit Card Fraud Detection (Finance)

Goal: Detect fraudulent transactions (Binary Classification).
Requirement: $LSL = 98\%$ Recall (due to high cost of False Negatives).
Result: Observed mean recall was 99.1%. The 95% CI was $[98.55\%, 99.67\%]$ .
$C_{pk}$ Calculation: 1.98.
Insight: The system is robust. Even the lower bound of the CI (98.55%) is comfortably above the requirement (98%), indicating high confidence for deployment.

Expert Validation (Interviews)

Suitability (RQ1): Experts unanimously agreed on the utility of defining specification limits, stratified sampling, and bootstrapping. The Capability Index was seen as a robust summary metric, though some noted it is most valuable in "marginal" cases where the decision isn't obvious.
Acceptance (RQ2): High intention to adopt. Experts perceived the approach as useful for MLOps and continuous monitoring. The only barrier identified was a potential "learning curve" regarding bootstrapping for practitioners unfamiliar with the technique.

5. Significance

Risk Quantification: The SCFC approach allows stakeholders to quantify the risk of deployment failure. It prevents the deployment of systems that have a high average score but high variability that could breach business limits in production.
Standardization: It offers a standardized, mathematical language for AI quality assurance, bridging the gap between business stakeholders (who define limits) and data scientists (who manage models).
Practical Feasibility: The study proves that the approach is computationally feasible (bootstrapping does not require re-running model inferences, only resampling results) and applicable to real-world industrial scenarios.
Future Direction: The paper advocates for a shift in AI engineering culture from "point-estimate validation" to "statistical confidence validation," providing a roadmap for more reliable and trustworthy AI systems.