📊 statistics

Feasibility Determination for Subjective Probability Constraints

This paper proposes a statistically valid procedure that directly utilizes Bernoulli-distributed simulation data to efficiently determine the feasibility of systems under subjective probability constraints with multiple thresholds, outperforming existing methods adapted for normal distributions while offering heuristic approaches for dynamically adjusting thresholds.

Original authors: Taehoon Kim, Sigrun Andradottir, Seong-Hee Kim, Yuwei Zhou

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Taehoon Kim, Sigrun Andradottir, Seong-Hee Kim, Yuwei Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a general trying to pick the best strategy from a list of 100 different battle plans. However, you don't know exactly how well each plan will work because war is chaotic and full of luck. You can only run simulations to see what might happen.

Your goal isn't just to find the "best" plan; it's to find the plans that are safe enough to use. You have specific rules, like: "The chance of losing more than 10% of our troops must be less than 5%."

This paper is about how to test these battle plans efficiently when the data you get from simulations is simple: a Yes (1) or a No (0). In the paper's language, this is called "Bernoulli distributed data."

Here is the breakdown of the problem and the authors' solution, using everyday analogies.

The Problem: The "Batching" Bottleneck

Traditionally, statisticians have a rule for testing these kinds of problems: "Don't look at individual coin flips; look at the average of 100 flips at a time." They do this because averages behave nicely (they look like a bell curve), making the math easier.

The authors say this is like trying to taste a soup by only taking a spoonful every hour. It's safe, but it's wasteful.

If you need to know if a soup is too salty, you don't need to wait an hour to taste a whole bowl. You can taste a drop immediately.
In the simulation world, waiting to gather "batches" of data to make them look "normal" wastes a huge amount of computer time. You might run 1,000 extra simulations just to satisfy a math rule that doesn't actually fit your "Yes/No" data.

The Solution: The "Random Walk" Detective

The authors propose a new method called MPB (Multi-pass Bernoulli Feasibility). Instead of waiting for batches, they treat every single simulation result as a clue immediately.

They use a concept called a Random Walk, which is like a drunk person walking down a hallway with two walls:

Wall A (Feasible): If the person hits this wall, the plan is Safe.
Wall B (Infeasible): If the person hits this wall, the plan is Unsafe.
The Middle: As long as the person is in the middle, you keep walking (running more simulations).

The genius of their method is how they set the walls. Instead of just measuring distance (e.g., "Is the plan 5% better?"), they measure the odds (e.g., "Is the plan 1.5 times more likely to succeed?"). This is more practical because it handles extreme probabilities (like a 99% success rate) much better than simple distance measurements.

The "Subjective" Twist: The Flexible Commander

Real-world decision-makers are rarely rigid. A commander might say:

"I want a 90% chance of victory." (Too hard? No plan works.)
"Okay, let's try 85%." (Still too hard.)
"How about 80%?" (Ah, we found some!)

Old methods forced you to test all these percentages at once, which is slow. The authors' method allows for a Multi-Pass approach:

Pass 1: Test a wide range of strict rules (e.g., 90%, 80%, 70%).
Pass 2: If you found too many good plans, tighten the rules (try 85%, 82%). If you found none, loosen the rules (try 75%, 70%).

They also invented a way to recycle data. Imagine you tested a plan against a 90% rule. When you later decide to test it against an 85% rule, you don't throw away the first test results. You reuse them. It's like using the same ingredients to bake a cake, then using the leftover batter to make cookies, rather than starting from scratch.

The Results: Speed and Accuracy

The authors ran thousands of experiments to prove their method works.

Accuracy: Their method guarantees that they won't make a mistake more than 5% of the time (a standard safety net).
Speed: Compared to the old "batching" method (called RF), their new method was 3 to 9 times faster. In one specific example involving inventory management, the new method used only 33% of the computer time required by the old method.
The "Heuristic" Bonus: They also showed that their "Multi-Pass" approach (testing loose rules first, then tightening them) is even faster than testing everything at once, saving massive amounts of time when you have many systems to check.

Summary

In short, this paper says: "Stop waiting for big batches of data to make your math easy. Just look at the 'Yes/No' results as they come in, use a smart 'odds-based' walking model to decide if a plan is safe, and reuse your data when you change your mind about the rules."

This saves time, saves computer power, and helps decision-makers find good solutions much faster without losing accuracy.

Technical Summary: Feasibility Determination for Subjective Probability Constraints

Problem Statement
This paper addresses the problem of identifying feasible systems within a finite set of simulated alternatives where the performance measures are subject to probability constraints. Unlike traditional Ranking and Selection (R&S) problems that often assume normally distributed observations and focus on expected performance measures, this work considers scenarios where observations follow Bernoulli distributions (outcomes of 0 or 1). The core challenge is to determine feasibility when the decision maker imposes subjective constraints, meaning they are willing to consider multiple threshold values for each constraint simultaneously or sequentially.

A key difficulty arises because standard statistically valid procedures for feasibility determination are designed for normally distributed data. While these can be adapted to Bernoulli data by using batch means to achieve approximate normality, this approach often requires large batch sizes, leading to the unnecessary waste of simulation observations before a decision is reached. Furthermore, existing procedures typically handle a single fixed set of thresholds, whereas decision makers in fields like military operations or inventory management often need to explore a range of thresholds (e.g., relaxing a constraint from 90% to 85% if no solution is found).

Methodology
The authors propose a new procedure, MPB (Multi-pass Bernoulli), which includes a statistically valid first-pass procedure (BRF) and heuristic multi-pass extensions for adding thresholds sequentially.

Odds-Ratio Indifference Zone (IZ):
Instead of using a difference-based IZ (which can be impractical for probabilities near 0 or 1) or a relative risk IZ, the authors adopt an odds-ratio IZ ( $\theta > 1$ ). This parameter defines the "slippage" between a desirable system (one that clearly meets the constraint) and an unacceptable system. The odds-ratio approach is preferred because it accounts for the difficulty of changing probabilities near the extremes (0 or 1) and allows for the use of a random walk model without requiring variance estimation or normality assumptions.
Random Walk Model with Dummy Variables:
The procedure utilizes a simple random walk model to process Bernoulli data directly. It introduces dummy Bernoulli variables ( $I_{i\ell mn}$ ) with a success probability equal to the threshold $h$ . The decision to declare a system feasible or infeasible is based on the cumulative difference between the observed Bernoulli outcomes ( $Y_{i\ell n}$ ) and these dummy variables. The process stops when the cumulative sum exits a continuation region $(-H_\ell, H_\ell)$ , where $H_\ell$ is an integer derived from the desired confidence level and the odds-ratio parameter. This eliminates the need for initial sample sizes to estimate variance or for batching observations.
Subjective Constraints and Multi-Pass Heuristics:
- First Pass (BRF): The procedure is proven to be statistically valid when all thresholds are defined in a single pass. It recycles observations across different thresholds within the same pass by maintaining upper and lower bounds on the sample means.
- Subsequent Passes (Heuristic): To accommodate decision makers who wish to add thresholds sequentially (e.g., tightening constraints if many systems are feasible, or relaxing them if none are), the authors propose three heuristic approaches:
  - $BRF_B$ : Uses the same random walk logic as the first pass but compares new thresholds against the bounds established in previous passes.
  - $BRF_N$ : Assumes that after the first pass, the sample means are approximately normally distributed, allowing for feasibility checks similar to procedures for normal data.
  - $BRF_{BN}$ : A hybrid approach that combines $BRF_B$ and $BRF_N$ , making a decision as soon as either method concludes.

Key Contributions
The paper makes four primary contributions:

Direct Bernoulli Feasibility Procedure: It proposes a procedure for feasibility determination with probability constraints that utilizes Bernoulli-distributed observations directly, avoiding the inefficiencies of batch means and normal approximations.
Subjective Constraints: It extends the analysis to incorporate subjective probability constraints, allowing for multiple thresholds per constraint to be considered either simultaneously or sequentially.
Statistical Validity and Efficiency: The authors prove the statistical validity of the first-pass procedure (BRF) and demonstrate empirically that it yields significant savings in the number of observations compared to the existing RF procedure (Zhou et al., 2022), which was designed for normal data and adapted for Bernoulli data.
Heuristic Multi-Pass Procedures: It introduces heuristic procedures that allow decision makers to add thresholds sequentially, demonstrating through experiments that these approaches can efficiently identify feasible systems with respect to preferred threshold combinations.

Experimental Results
The authors conducted extensive experiments to evaluate the proposed procedures:

Comparison with RF: In single-pass scenarios, BRF consistently achieved the target probability of correct decision (PCD) across various probability values and odds-ratio settings. In contrast, the RF procedure (adapted for Bernoulli data) was highly sensitive to batch size and tolerance levels. To achieve comparable PCD, RF required batch sizes as large as 400, resulting in observation counts roughly 1.5 to 9.3 times larger than BRF.
Multi-Pass Efficiency: The heuristic multi-pass procedures (specifically MPB $_{BN}$ ) were shown to be substantially more efficient than running a single-pass BRF with all thresholds included. In an optimization setting with 100 systems, MPB $_{BN}$ required only about 6% of the observations needed by BRF while maintaining or improving the PCD.
Inventory Application: In a realistic $(s, S)$ inventory problem with 77 systems and two probability constraints, MPB $_{BN}$ required approximately 33% of the observations needed by BRF and roughly 1/13th of the observations needed by RF, while maintaining high statistical validity.

Significance and Claims
The paper claims that its proposed procedures offer a statistically valid and highly efficient alternative to existing methods for feasibility determination under probability constraints. By leveraging the odds-ratio IZ and a random walk model, the authors eliminate the need for variance estimation and the inefficiencies associated with batching Bernoulli data. The ability to handle subjective constraints—where decision makers iteratively adjust thresholds—allows for a more practical and resource-efficient approach to simulation-based decision making. The authors emphasize that while the multi-pass procedures are heuristic and not rigorously proven to be statistically valid in all sequential scenarios, experimental results indicate they do not violate validity requirements and offer significant computational savings.