💻 computer science

Finite Evidence, Infinite Horizons: The Artificial Age Score as a Case Study in Longitudinal AI Measurement

This paper argues that finite longitudinal AI evaluation records, exemplified by the Artificial Age Score (AAS), cannot determine infinite-horizon system properties or convergence without explicit stipulated continuation models, as demonstrated through counterexamples and stress tests showing that bounded observations alone yield only conditional, model-based inferences rather than definitive conclusions.

Original authors: Seyma Yaman Kayadibi

Published 2026-09-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Seyma Yaman Kayadibi

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, researchers are constantly trying to measure how well these systems learn, remember, and perform over time. Just as a teacher might track a student's grades across a semester to predict future success, scientists use longitudinal scores to monitor AI systems as they face new questions, updates, and changing environments. These scores are meant to be more than just a snapshot of a single moment; they are intended to tell a story about the system's journey, its stability, and its ability to keep working effectively as time passes. However, a fundamental challenge lurks in this approach: we only ever have a finite record of what has happened so far. We have a history of past tests, but we do not have the future. The critical question is whether a pattern observed in a limited number of past tests can truly tell us anything definitive about how the system will behave indefinitely into the future, or if we are simply reading too much into a short story.

A new study by Seyma Yaman Kayadibi at Victoria University tackles this problem directly, using a specific measurement tool called the Artificial Age Score as a case study to expose a hidden gap in how we interpret AI performance. The research does not argue that these scores are useless; rather, it demonstrates that a score calculated from a finite history of observations cannot, on its own, prove that a system will continue to exist, remain stable, or perform well forever. The study shows that a system could appear perfectly stable for twenty sessions and then fail immediately after, or it could continue perfectly, and the mathematical score calculated from those first twenty sessions would look identical in both scenarios. The paper argues that any claim about the infinite future of an AI system is not a consequence of the data we have collected, but rather a result of extra assumptions we make about how the system will continue.

To illustrate this, the researchers examined a real-world dataset involving an AI system tested over seventy-four weekly releases. They compared two versions of the system: one that operated in a closed environment and another that could search the internet for answers. The data showed that the version with internet access consistently performed better, resulting in a lower "burden" score, which indicates fewer errors. The researchers broke down exactly why this difference occurred, finding that the improvement came from better performance in both multiple-choice questions and open-ended text generation. However, when they tried to predict the future performance of these systems using simple rules—such as assuming the next score would be the same as the last one, or that it would be the average of the last four—they found that every single prediction rule made mistakes. The rule that worked best for one version of the system did not work best for the other, proving that there is no single, universal law hidden inside the scores that dictates the future.

The study also explored a deeper mathematical reality: for any finite sequence of scores, it is possible to imagine an infinite number of different futures that fit that sequence perfectly. One future might show the system improving forever, another might show it failing immediately, and a third might show it oscillating between good and bad performance. All of these different futures would produce the exact same score for the past twenty sessions. The researchers proved that without adding extra rules about how the system behaves over time, the score itself cannot distinguish between these possibilities. They also looked at how the passage of real time affects these measurements. If you count every test as one unit of time, you might get a different picture of the system's total workload than if you measure by the actual hours and days between tests. The paper shows that these different ways of measuring time can change the interpretation of how much "exposure" or stress the system has endured, yet neither method can predict what happens next.

Ultimately, the paper concludes that we must be much more careful about how we talk about the long-term reliability of artificial intelligence. A score that looks stable today is not a guarantee of stability tomorrow. To make claims about the future, researchers must explicitly state the assumptions they are making about the system's lineage, its environment, and its rules of operation. The study proposes a new way of reporting results that clearly separates what was actually observed, what was assumed to make a prediction, and what remains unknown. By treating the future as a conditional possibility rather than a mathematical certainty derived from past data, the research offers a more honest and rigorous framework for understanding the limits of what we can know about the artificial age. The findings serve as a reminder that while we can measure the past with precision, the future remains a separate domain that requires its own set of rules and assumptions to navigate.

Technical Summary: Finite Evidence, Infinite Horizons: The Artificial Age Score as a Case Study in Longitudinal AI Measurement

1. Problem Statement

The paper addresses a fundamental epistemological gap in the longitudinal evaluation of Artificial Intelligence (AI) systems: the conflation of finite observed evidence with infinite-horizon mathematical claims.

While longitudinal scoring (e.g., tracking performance over sessions, updates, or deployments) is scientifically useful, there is a prevalent tendency to treat a finite record of scores as the beginning of an infinite sequence. Researchers often infer properties of the unobserved future—such as system persistence, state convergence, or indefinite operation—based solely on the boundedness or trends of a finite score prefix.

The core difficulty identified is the "horizon gap," which separates three distinct objects:

The Evidential Horizon: The finite, registered record of observations ( $E_N$ ).
The Mathematical Horizon: The infinite sequence or stochastic process defined by a continuation model.
The Target Horizon: The physical or computational system itself, its lineage, and its actual future availability.

The paper argues that a finite score prefix does not deductively determine the convergence, cumulative exposure, or continued existence of the system. Without explicit bridge assumptions, infinite-horizon conclusions are conditional model-based inferences, not consequences of the finite data alone.

2. Methodology

The paper employs a hybrid methodology combining formal mathematical proofs with an empirical stress test using the "Artificial Age Score" (AAS) as a concrete case study.

2.1 Formal Framework

The author establishes a theoretical framework using the AAS, a redundancy-adjusted composite score. The methodology involves:

Observation Fibre Analysis: Defining the set of all admissible system models ( $M$ ) that produce a specific observed prefix. The paper proves that for any finite prefix, the "fibre" contains models with incompatible asymptotic behaviors (e.g., converging to zero, diverging, or oscillating).
Matched-Prefix Constructions: Constructing explicit counterexamples where different infinite tails (satisfying different limit properties) share the exact same finite prefix. This demonstrates that no statistic computed solely from an unrestricted finite prefix can determine the class of the unobserved tail.
Three-Horizon Ledger: Developing a reporting structure that explicitly separates what is observed, what is assumed (bridge conditions), what is derived, and what is not established.

2.2 Empirical Stress Test

To ground the theoretical claims, the paper conducts a secondary analysis of the RealTimeQA public archive (74 release batches from June 2022 to December 2023).

Data: Paired comparisons of two configurations: "closed GPT-3" and "GPT-3 with Google Custom Search (GCS) and retrieval fallback."
Metric Mapping: The AAS was applied to batch-level multiple-choice accuracy and generation F1 scores with equal weights and no estimated redundancy ( $R=0$ ).
Analyses Performed:
1. Paired Contrast: Decomposing the mean difference in AAS between configurations into component penalties.
2. Exploratory Redundancy Sensitivity: Applying a proxy for item-level co-deficit ( $O_{def}$ ) to test how varying redundancy assumptions ( $R = \lambda O_{def}$ ) affects the score contrast.
3. Aggregation Offset Audit: Quantifying how opposing component movements offset each other when summed into a scalar score (using a "release-index aggregation offset" metric, $K_N$ ).
4. Rolling-Origin Forecasting: Comparing the Mean Absolute Error (MAE) of four finite continuation rules (persistence, moving mean, scalar AR(1), and component-wise AR(1)) to demonstrate that no single rule is universally superior and all retain non-zero error.
5. Duration-Weighted Exposure: Calculating cumulative exposure using physical time intervals rather than just cycle counts.

3. Key Contributions

3.1 Theoretical Contributions

The Horizon Gap Framework: A formal distinction between evidential, mathematical, and target horizons, clarifying that moving from a finite record to an infinite claim requires explicit bridge assumptions (e.g., lineage rules, continuation models, non-Zeno duration schedules).
Matched-Prefix Underdetermination Theorems: Proofs showing that for any finite AAS prefix, there exist algebraically admissible infinite continuations that converge to zero, diverge, oscillate, or terminate, rendering the prefix insufficient to determine the system's asymptotic fate.
Scalar Convergence vs. State Identification: A proposition demonstrating that scalar score convergence does not necessarily imply state convergence if the measurement map is non-injective (i.e., different system states can yield the same score).

3.2 Empirical Contributions

AAS Decomposition: In the RealTimeQA audit, the GCS configuration showed a lower mean AAS (better performance) than the closed configuration (Mean difference: $-0.738$). This difference was decomposed into a $-0.261$ contribution from multiple-choice accuracy and a $-0.477$ contribution from generation F1.
Aggregation Offset: The study quantified that opposing component changes offset approximately 14.5% of the gross movement for the closed configuration and 11.8% for the GCS configuration. This highlights how scalar aggregation can mask significant internal volatility.
Forecasting Limitations: All rolling-origin forecasts retained non-zero error. The "best" forecasting rule depended on the configuration (Scalar AR(1) was best for closed; Mean-4 was best for GCS), indicating that finite prefixes do not select a universal continuation law.
Redundancy Sensitivity: While the direction of the performance contrast remained negative across five levels of imposed redundancy ( $\lambda \in [0, 1]$ ), the magnitude of the difference attenuated significantly (from $-0.738$ to $-0.259$), demonstrating that redundancy assumptions are critical measurement parameters, not observed facts.

4. Results

Finite Prefixes are Indeterminate: A finite record of 20 zero AAS scores (from a published bilingual study) is mathematically compatible with infinite futures where the system continues perfectly, fails immediately, oscillates, or terminates. The score prefix alone establishes none of these.
No Universal Forecasting Rule: In the RealTimeQA stress test, no single forecasting rule minimized error across all configurations and horizons. This supports the claim that finite evidence constrains but does not uniquely select a future model.
Measurement Map Dependence: The magnitude of performance differences is highly sensitive to the choice of redundancy coefficients ( $R$ ) and component weights. The paper shows that treating overlap as redundancy is an assumption, not a derived fact.
Physical Time vs. Cycle Count: The paper demonstrates that counting cycles (events) is distinct from measuring physical time. Without a non-Zeno duration schedule (where total time $\to \infty$ ), an infinite number of cycles does not imply operation over an infinite physical horizon.

5. Significance and Claims

The paper modestly claims to provide an audit framework rather than a new predictive theory or a validation of the AAS itself.

Clarification of Induction: The primary contribution is making the "horizon gap" explicit. It argues that infinite-horizon conclusions are conditional model-based inferences, not consequences of bounded scores or finite observations alone.
Reframing Longitudinal Claims: The paper proposes a "claim ledger" (Table 10) for longitudinal AI studies. This ledger requires researchers to explicitly record:
- Observed: The finite record and endpoint status.
- Assumed: Lineage rules, continuation models, and physical-time schedules.
- Derived: Mathematical consequences of the record plus assumptions.
- Not Established: Claims about causal mechanisms, indefinite persistence, or system survival that require additional evidence.
Role of AAS: The AAS is used as a "concrete measurement map" to illustrate general problems of projection. The paper does not claim to derive new single-cycle properties of AAS or validate it as a universal construct. Instead, it uses AAS to show how a static score can be misinterpreted as a dynamic survival model.
Scientific Role: The paper positions its elementary impossibility results (matched-prefix constructions) as tools to locate missing premises. It argues that the scientific value lies not in proving prediction is impossible, but in ensuring that the bridge assumptions required for any projection are visible and auditable.

In conclusion, the paper asserts that longitudinal AI scores are useful for monitoring and comparison within a declared protocol, but they cannot, by themselves, establish that a system will persist, converge, or remain comparable indefinitely. Any such claim requires an explicitly declared continuation model and bridge assumptions regarding lineage and physical time.