When Is Enough Precedent Enough? A Decision-Responsive Stopping Rule for Regulatory Precedent Research in Target Product Profile and Label Development
This paper proposes a decision-responsive stopping rule for regulatory precedent research that synthesizes Value of Information, saturation, and capture–recapture frameworks to determine when sufficient evidence has been gathered for Target Product Profile and label development, demonstrating through GLP-1 case studies that fixed-volume approaches often lead to premature conclusions while context-aware methods align with the evolving regulatory landscape.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a detective trying to solve a mystery, but instead of a crime scene, your office is a library that never closes. Every day, new books, letters, and secret files arrive, and old ones sometimes get burned or rewritten. Your job is to find the "perfect" collection of evidence to prove a specific point, like whether a new medicine is safe for the heart or the kidneys. The problem? The library is infinite. If you try to read everything to be "comprehensive," you'll never finish, and the library will keep growing while you work. So, the big question for any detective is: When do you stop looking? Do you stop when you've read 50 books? When you've spent 10 hours? Or when you've read enough that reading one more book wouldn't change your conclusion?
This paper tackles that exact puzzle in the world of medicine regulation. It introduces a few key ideas to help detectives decide when they have "enough." First, there's the Target Product Profile, which is like a blueprint or a wish-list for what a new medicine should look like and what it should be allowed to do. Second, there's Regulatory Precedent, which means looking at how regulators (the government officials who approve medicines) have handled similar cases in the past. Finally, there's the concept of Value of Information, which is a fancy way of asking: "Is the next piece of information I'm about to find worth the time and money it will cost me to get it?" If the next book in the library just repeats what you already know, it's not worth reading. But if it might change your mind, you have to keep going.
The authors of this paper, Saman Kay from Basil Systems, Inc., propose a new "stopping rule" to help researchers know exactly when to close the library door. They argue that you can't just pick a random number of documents to read. Instead, you need a smart strategy that changes based on where the answer actually hides.
The "Concentric Rings" Game
To explain their method, the authors imagine the library isn't just a big pile of books, but a set of three concentric rings, like ripples in a pond.
- Ring 1 (The Center): These are the books about the exact same medicine or the exact same disease. This is the most obvious place to look.
- Ring 2 (The Middle): These are books about medicines that are slightly different but work in a similar way or treat a related condition.
- Ring 3 (The Outer Edge): These are books about completely different medicines that might share a specific rule, a type of test, or a way of measuring success.
The old way of doing things was to just read until you felt tired or until you hit a time limit. The authors suggest a smarter way: Follow the "Marginal Yield." This means you keep reading as long as every new book you pick up gives you something new and important that changes your decision. As soon as a whole ring of books stops giving you new clues, you stop.
The Two Stories: Heart vs. Kidney
To prove their rule works, the authors tested it on two real-life scenarios involving a popular class of diabetes drugs called GLP-1 agonists.
Story 1: The Heart Claim (The Easy Stop)
First, they looked at whether these drugs could reduce heart risks. They started in Ring 1 (other heart drugs in the same class). They found that the very first ring of books contained all the answers they needed. The outer rings (Ring 2 and 3) were just repeating the same facts.
- The Result: Their new rule told them to stop after reading just 60% of the total available documents. They saved a huge amount of time because the answer was right in the center.
- The Trap: If they had used a "naive" rule (like "stop when you feel like you've read enough" or "stop when a computer says you're 95% done"), they would have stopped way too early, at only 25% of the documents, and missed some crucial details.
Story 2: The Kidney Claim (The Long Hunt)
Next, they looked at whether these drugs could protect the kidneys. This time, the story was totally different. The drugs in Ring 1 (the same class) had very little information about kidneys. The real answers were hiding in Ring 3, written by completely different types of drugs (like SGLT2 inhibitors) that had already set the rules for kidney protection years ago.
- The Result: Their new rule told them to keep going all the way to 100% of the documents. They had to read every single ring because the answer was at the very edge.
- The Trap: If they had stopped early based on a "feeling" or a simple count, they would have missed the most important part of the story entirely.
The "PRO" Surprise
The authors also tested this on a third, real-world case involving a rare nerve disease and a "patient-reported outcome" (a way of measuring how a patient feels). They didn't know where the answer would be. The rule correctly sent them all the way to Ring 3, where they found a completely different drug that had the perfect "playbook" for how to measure patient feelings, even though it treated a different disease. This proved that the rule is flexible enough to find answers in unexpected places.
What the Paper Says (and Doesn't Say)
The authors are very clear about what they have and haven't done. They demonstrated that this new "decision-responsive" rule works better than just counting documents or guessing. They showed that in some cases, you can stop early, and in others, you must read everything.
However, they explicitly rule out the idea that there is a single "magic number" of documents that works for every situation. They also argue against using simple computer estimates (like the Chao2 estimator mentioned in the text) as the final stop signal, because those computers can get fooled into thinking you're done when you're actually missing the most important clues.
The paper presents this as a proof of concept. It's a "test drive" of a new method using reconstructed data and one real client project. The authors admit that while the logic is sound and the results look promising, this isn't a final, statistical proof that works for every single drug in the world yet. They suggest that the next step is to test this rule on many more decisions to see if it holds up.
The Takeaway
The main lesson is that "enough" isn't a fixed number. It's a moving target. If the answer is in the center, you stop early. If the answer is at the edge, you keep going. The best way to decide when to stop isn't to look at a clock or a page count, but to ask: "If I read one more document, will it change my mind?" If the answer is no, you're done. If the answer is yes, you keep digging. This approach saves time when possible but ensures you never miss the critical evidence hidden in the periphery.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.