📊 statistics

Blockwise Joint Conformal Prediction for Simultaneous Multi-Component Condition Monitoring of Hydraulic Systems

This paper proposes a blockwise joint conformal prediction framework that calibrates prediction sets over operational blocks rather than individual components, demonstrating on hydraulic system data that this approach achieves significantly higher simultaneous coverage for multi-component states compared to conventional marginal methods, albeit with backbone-dependent set sizes and strict reliance on exchangeability assumptions.

Original authors: Bùi Thanh Lâm

Published 2026-08-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Bùi Thanh Lâm

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the hidden world of industrial machinery, a hydraulic system is a complex network of pumps, valves, and fluid lines working together to power heavy equipment. Like a human body, it has multiple organs that can fail simultaneously, and a single breakdown can stop a factory or cause a safety hazard. To prevent this, engineers use automated systems that constantly listen to the machine's sensors, trying to spot trouble before it becomes a disaster. For years, these systems have relied on machine learning to predict the health of individual parts. However, a critical gap has remained in how these predictions are trusted. When a computer says it is 95 percent sure that a specific valve is healthy, and 95 percent sure that a pump is healthy, it does not mean the entire system is 95 percent safe. The risks of multiple parts failing at once, or a single part failing over a sequence of time, multiply in ways that simple percentages cannot capture. Engineers need to know not just if a single part is likely broken, but if the entire diagnosis for a specific time window is reliable enough to act upon.

This question of reliability is the focus of a new study by Bùi Thanh Lâm, a researcher at the Hanoi University of Science and Technology. The study addresses a fundamental flaw in how automated diagnostic systems report their confidence. In many industrial settings, a machine runs through repeated cycles of operation. A maintenance decision is rarely made on a single snapshot; instead, it is based on the state of several components over a block of time, perhaps ten consecutive cycles. The researcher set out to test whether standard methods of calculating uncertainty could handle this complexity. The standard approach treats each component separately, giving a confidence score for the valve, another for the pump, and so on. The study demonstrates that this method fails when applied to the whole system. If you have four components and you are 95 percent confident about each one individually, the chance that you are right about all four at the same time drops significantly, to roughly 82 percent. If you extend that requirement over ten cycles, the reliability of the entire diagnosis plummets to less than 50 percent.

To solve this, the researcher developed a new way of calibrating these diagnostic systems. Instead of checking the confidence of each part in isolation, the new method looks at the entire "block" of operation as a single unit. Imagine a block as a complete ten-second snapshot of the machine's life, containing data from all four components. The method works by taking the worst-case error from any component within that block and using it to set the safety threshold for the whole group. This approach was tested using data from a hydraulic test rig, a standard benchmark in the field that simulates various fault conditions. The data included 1,440 stable operating cycles, organized into 144 distinct blocks representing different combinations of component health. The researcher split this data into training, calibration, and testing groups, running the experiment thirty times to ensure the results were robust.

The results were striking. When the researchers used the old, separate-component method, the system's claim of 95 percent reliability was an illusion. In reality, the system only correctly identified the full state of the machine in about 82 percent of the cycles, and it failed to cover the entire ten-cycle block in nearly half of the test cases. By switching to the new block-based method, the researchers restored the promised reliability. The system now correctly identified the full state of the machine in 99 percent of the cycles and covered the entire ten-cycle block in 97 percent of the cases for one type of model, and 95 percent for another. The system finally delivered on its promise of 95 percent confidence, but only because it stopped looking at parts in isolation and started looking at the whole picture.

However, this increased reliability came with a cost, which the study carefully measured. To guarantee that the system is right about the entire block, the computer had to be less specific about exactly which state the machine was in. In the old method, the system might narrow the possibilities down to just a few likely scenarios. In the new, safer method, the system often had to list many more possibilities to ensure it didn't miss the truth. For one of the models tested, the list of possible system states grew from a manageable number to nearly the entire universe of possibilities, effectively saying "anything could be happening" to be safe. For the other model, the list grew much more moderately, staying within a useful range. This highlights a crucial trade-off: you can have a guarantee of safety, or you can have a very specific guess, but getting both is difficult. The study found that the choice of the underlying computer model mattered greatly; a more complex model produced tighter, more useful lists of possibilities while still maintaining the high safety guarantee, whereas a simpler model produced lists so broad they were almost useless.

The study also tested the limits of this new method by applying it to unstable data—cycles where the machine had not yet settled into a steady state. This is a common real-world scenario where a machine is starting up or changing speed. When the researchers applied the safety guarantees derived from stable data to these unstable cycles, the system's performance dropped. The block-based method, which worked perfectly on steady data, saw its reliability fall significantly when faced with these shifting conditions. This finding serves as a vital warning: a safety guarantee is only valid if the machine is operating in the same way it was when the system was trained. If the machine enters a new regime, such as a startup phase or a sudden load change, the old safety numbers no longer apply. The system must recognize when it is out of its comfort zone and stop making confident claims.

Ultimately, this research changes how we think about trust in automated maintenance. It proves that you cannot simply add up the confidence of individual parts to get the confidence of the whole system. To make safe decisions, engineers must define exactly what they are trying to protect—a single moment, a sequence of time, or a specific group of components—and calibrate their systems to match that definition. The study offers a practical tool, a "blockwise" method, that allows engineers to get the reliability they need, provided they are willing to accept a bit more uncertainty in the details. It is a reminder that in the complex world of industrial automation, being right about the big picture requires a different kind of math than being right about the small parts. The path to safer factories lies not in making better guesses about individual gears, but in understanding how those gears move together over time.

Technical Summary: Blockwise Joint Conformal Prediction for Simultaneous Multi-Component Condition Monitoring

Problem Statement
Automated condition-monitoring systems in industrial settings, such as hydraulic systems, often assess multiple components simultaneously. A critical gap exists in current reliability reporting: providing separate 95% prediction sets for individual components does not guarantee 95% reliability for the complete system state or for a sequence of operating cycles. Under independence assumptions, four components with 95% marginal coverage yield a joint coverage of only $0.95^4 \approx 0.81$ . This discrepancy is compounded when maintenance decisions require a state to persist over multiple cycles (e.g., ten cycles). Standard conformal prediction methods, which typically calibrate on individual observations, fail to provide valid coverage guarantees for these "whole-system, whole-window" operational units. Furthermore, existing joint conformal methods often address time-series regression or nested data structures that do not directly map to the multi-class, multi-component classification tasks found in industrial fault diagnosis.

Methodology
The study proposes a Blockwise Joint Conformal Prediction framework tailored for hydraulic condition monitoring. The core innovation is redefining the unit of calibration from individual sensor readings or cycles to an operating-condition block.

Data Structure: The method utilizes a dataset of 1,440 stable cycles from a UCI hydraulic test rig, organized into 144 blocks. Each block represents a unique combination of four component states (cooler efficiency, valve switching, pump leakage, accumulator pressure) and contains ten consecutive operating cycles.
Score Function: For each calibration block $j$ , a nonconformity score is defined as the maximum nonconformity across all components ( $K=4$ ) and all cycles ( $L=10$ ) within that block:
$R_{block}^j = \max_{1 \le \ell \le L} \max_{1 \le k \le K} s_k(X_{j\ell}, Y_{j\ell k})$
where $s_k$ is the standard probability-complement score ( $1 - \hat{p}_k$ ).
Calibration: A finite-sample quantile $q_\alpha$ is computed from the distribution of these block-level scores. The resulting prediction sets for each component $k$ are Cartesian products defined by $C_k(x) = \{c : s_k(x, c) \le q_\alpha\}$ .
Theoretical Guarantee: Under the assumption that calibration blocks and future test blocks are exchangeable, the method guarantees that the probability of the true label vector for every component in every cycle of a future block being contained within the prediction sets is at least $1-\alpha$ .
Experimental Design: The study employs a rigorous, leakage-resistant benchmark using 30 block-disjoint partitions (72 training, 43 calibration, 29 test). Two distinct backbone classifiers (logistic regression and random forest) are evaluated to test model-agnostic validity and efficiency differences.

Key Contributions

Distinction of Coverage Units: The paper formally distinguishes between component-wise, simultaneous per-cycle, and simultaneous per-block diagnostic coverage, demonstrating that interpreting component-wise coverage as a system-level claim leads to significant reliability overstatement.
Block-Level Construction: It introduces a specific split-conformal construction that converts arbitrary component classifiers into interpretable, simultaneous prediction sets while controlling the omission of any true label over an entire decision window.
Rigorous Benchmarking: The study provides a leakage-resistant evaluation on a complete factorial hydraulic experiment, ensuring that no component-state combination or its adjacent cycles appear in multiple partitions (training, calibration, or testing).
Comprehensive Assessment: It evaluates coverage, set efficiency (size), empty-set rates, and sensitivity to window size and calibration size, including a post-hoc stress test on unstable (non-steady-state) cycles.

Results

Coverage vs. Marginal Claims: At a nominal 95% target, conventional marginal sets achieved only ~82% simultaneous cycle coverage and ~50% whole-block coverage. The proposed blockwise joint method successfully restored coverage to 0.971 (logistic regression) and 0.946 (random forest) for whole blocks.
Efficiency Trade-off: Achieving system-level validity came at the cost of set size (ambiguity).
- Logistic Regression: Produced very large sets, with a mean of 103.9 out of 144 possible system states (median 144), rendering the output nearly vacuous for decision-making.
- Random Forest: Produced significantly sharper sets, with a mean of 13.7 states (median 6.0), demonstrating that the "price" of validity is highly backbone-dependent.
Stress Testing: When applied to unstable cycles (regime shifts) outside the calibration distribution, the random forest model's block coverage dropped to 0.830, while the logistic regression model remained high (0.942) but largely due to its overly broad sets. This confirms that validity does not extrapolate across regime shifts.
Sensitivity: Increasing the protected window length required wider sets to maintain coverage. Varying calibration size did not yield monotonic improvements in set sharpness, highlighting finite-sample variability.

Significance and Claims
The paper argues that reliability claims in multi-component diagnosis must explicitly match the operational decision unit. A guarantee that "each component is 95% reliable" is insufficient for maintenance decisions requiring a complete system state over a time window.

The author concludes that while blockwise joint conformal prediction offers a simple, reproducible wrapper to achieve valid system-level coverage, it is not a panacea. The method's utility is constrained by the underlying classifier's ability to order correct labels (efficiency) and the strict requirement that the deployment regime remains exchangeable with the calibration data. The study emphasizes that valid prediction sets must be audited for both sharpness and the validity of the exchangeability assumption, rather than relying solely on coverage metrics. The work serves as a cautionary demonstration that system-level reliability cannot be inferred from component-level metrics without explicit joint calibration.

Technical Summary: Blockwise Joint Conformal Prediction for Simultaneous Multi-Component Condition Monitoring

More like this