Provenance Guided Incremental Learning Under Evolving Concept Definitions
This paper introduces a provenance-guided incremental learning framework that efficiently adapts machine learning models to explicit rule-induced concept shifts by automatically relabeling affected records and selectively retraining only on changed components, thereby significantly reducing computational latency and data reprocessing while maintaining high accuracy compared to full retraining.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of machine learning, computers are often taught to make predictions by studying patterns in data. Imagine a system trained to spot fraudulent bank transactions. It learns what a normal transaction looks like and what a suspicious one looks like based on the rules and definitions set by its creators. However, the real world is not static. The rules that define what counts as "fraud" can change. A bank might decide that a transaction is only suspicious if it exceeds a certain amount, or if it comes from a specific country, or if it involves a new type of device. When these definitions change, the computer's old training becomes outdated. The data it studied yesterday is still the same, but the meaning of the answer has shifted. This creates a difficult problem: how do you update a learning system when the very definition of the truth has been rewritten, without having to throw away everything it has learned and start over from scratch?
For a long time, researchers have dealt with this by watching for mistakes. They wait until the computer starts making errors, infer that something has changed, and then try to adjust the model. But this approach is slow and often misses the mark. It treats the change as a mystery to be solved by guessing, rather than a known fact to be acted upon. In many real-world systems, the change is not a mystery at all; it is a deliberate update to a policy or a rule. The new definition is known immediately. The challenge is figuring out which of the millions of past records actually need to be re-evaluated because of this new rule, and which ones can be left alone because they are unaffected.
A new study introduces a method that treats this problem like a careful audit rather than a blind guess. The researchers developed a system that looks at the old rule and the new rule side by side to find the exact difference between them. It then traces how that specific difference connects to the history of past data. By following these connections, the system can identify a small group of records that are truly affected by the change. It can also prove that the vast majority of records are safe and do not need to be touched. This allows the computer to update its knowledge by focusing only on the small, relevant slice of data, while keeping the rest of its understanding intact.
The researchers tested this approach on a wide variety of real-world scenarios, including financial transactions, demographic data, cybersecurity logs, and complex networks of relationships. They created a set of challenges where the rules for labeling data were changed in different ways: by raising or lowering a threshold, by adding a new condition, by removing an old one, or by changing how different conditions were combined. In every case, the new method was able to identify the records that needed attention with high precision. Instead of re-checking every single record in the database, the system only reprocessed about 15 percent of the data. Despite looking at so little data, it maintained an accuracy of 92.3 percent, which was nearly identical to the result of re-checking the entire database.
The speed of this process was the most striking result. When the researchers forced the system to re-label and re-train on the entire history of data, it took an average of 993 seconds to complete the update. With the new method, the same update took only 179 seconds. This represents a reduction in time of more than five times. The system achieved this by using a "provenance" map, which is essentially a record of how each past decision was made. If a past decision relied on a part of the rule that did not change, the system knows that the decision is still valid. If a decision relied on a part that did change, the system flags it for review. This allows the computer to certify that most of its past knowledge is still correct and only spend time on the parts that are broken.
The study also addressed what happens when the new rule is not perfectly clear. Sometimes, a revised policy might depend on information that is missing or requires human judgment. In these cases, the system does not guess. Instead, it identifies the specific records where the new rule is ambiguous and asks a human for help only on those cases. This makes the use of human experts much more efficient, as they are not wasting time on records that the computer can handle on its own. Furthermore, the system keeps a memory of past versions of the rules. If a rule changes and then changes back to a previous version, the system can instantly recall the old configuration and the data that went with it, rather than having to learn the pattern all over again.
The researchers found that this approach works well across different types of data, from simple lists of numbers to complex networks of connections. However, they also noted its limits. The method is most effective when the change in the rule is localized, affecting only a small portion of the data. If a new rule changes the meaning of almost every record, the system has to re-check almost everything, and the speed advantage disappears. Similarly, if the history of how decisions were made was not recorded, the system cannot easily tell which records are safe, and it must be more cautious, checking more data than it would otherwise need to.
Ultimately, this work shifts the perspective on how learning systems adapt. Instead of waiting for errors to appear and then reacting, the system can use the explicit knowledge of a rule change to surgically update its understanding. It proves that by understanding the structure of the change and how it connects to history, a computer can preserve what it already knows while efficiently fixing what has changed. This allows learning systems to remain accurate and responsive over long periods, adapting to new policies without the heavy cost of constant, total retraining.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.