🤖 AI

ASIA: an Autonomous System Identification Agent

The paper introduces ASIA, an autonomous agent framework that leverages large language models to automate the end-to-end system identification process—from model selection to training and evaluation—based solely on plain-English problem descriptions, thereby reducing reliance on expert trial-and-error while highlighting challenges in transparency and reproducibility.

Original authors: Dario Piga, Marco Forgione

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dario Piga, Marco Forgione

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to drive a car, but you don't know the rules of the road, the engine's quirks, or the best way to steer. In the past, a human expert would have to spend years guessing which teaching method works best, tweaking the lessons, and trying again and again until the robot finally learns. This process is slow, expensive, and relies heavily on the expert's gut feeling.

This paper introduces ASIA (Autonomous System Identification Agent), a new tool that acts like a super-smart, tireless apprentice who can do this guessing and tweaking all by itself.

Here is how ASIA works, broken down into simple concepts:

1. The Problem: The "Trial-and-Error" Maze

Think of building a model of a real-world system (like a water tank or a drone) as trying to find the perfect recipe for a cake. You have to decide:

The Ingredients: Should you use flour, sugar, or maybe a secret spice? (In tech terms: choosing the model type, like a neural network or a physics equation).
The Oven Settings: What temperature and how long? (In tech terms: hyperparameters like learning speed or data size).
The Mixing Method: How do you combine them? (In tech terms: the training algorithm).

Usually, a human chef has to taste the cake, realize it's too dry, add more eggs, bake it longer, and taste again. This takes forever.

2. The Solution: ASIA as the "Robot Chef"

ASIA is a computer program powered by a Large Language Model (think of it as a very advanced AI that can write and understand code). Instead of a human chef, you give ASIA a simple instruction in plain English, like: "Here is data from a water tank system; please figure out how it works."

ASIA then enters a loop of self-improvement:

Hypothesis: It guesses a new "recipe" (a new model structure).
Baking: It writes the code to test this recipe.
Tasting: It runs the code and sees how well the model predicts the future.
Learning: If the cake is bad, it reads the results, thinks, "Maybe I should use a different type of flour," and tries again.

It does this hundreds of times without ever needing a human to touch the keyboard, until it finds the best possible recipe.

3. The Test Drive: Two Real-World Challenges

The authors tested ASIA on two very different "kitchens" to see if it could cook up a good meal:

The Water Tanks (The Simple Kitchen): Imagine two buckets of water connected by a pipe. The goal is to predict the water level in the bottom bucket based on how much water is poured in the top one.
- Result: ASIA started with a basic, simple model. After a few tries, it realized the simple model wasn't strong enough. It upgraded to a more complex "brain" (an LSTM network) and then found a specific type of network (LTC) that naturally mimics how water drains. It ended up matching the best results ever published for this problem.
The Nanodrone (The Complex Kitchen): Imagine a tiny, four-propeller drone flying around. It has 12 different moving parts (position, speed, angles) all interacting at once.
- Result: ASIA started with a standard approach. Then, it had a brilliant idea: instead of trying to predict the drone's position directly, it decided to predict how the speed changes and then calculate the position from that (mimicking how physics actually works). It also figured out how to stabilize the training process.
- Comparison: When the authors compared ASIA to a "random search" (where a computer just picks random settings like a lottery), ASIA found a much better solution much faster. It beat the random search and even outperformed the best human-designed methods from previous studies.

4. The Catch: The "Black Box" and the "Ghost in the Machine"

While ASIA is impressive, the authors are honest about its flaws:

The "Ghost" (Test Leakage): Sometimes, even if you tell the AI "don't look at the final exam," it might accidentally learn hints from the exam questions because it read about them in its training data or the research papers you gave it. It's like a student who memorized the answer key from a textbook rather than learning the math.
The "Black Box" (Lack of Transparency): ASIA can cook a perfect cake, but it might not be able to explain why it chose those specific ingredients. In science, knowing why a model works is often just as important as the model itself. Because ASIA changes its own code, the path it took to get there is a bit of a mystery.
Reproducibility: If you ask ASIA to do the same task twice, it might come up with two slightly different solutions because AI is a bit random. It's like asking a human chef to make the same dish twice; they might use slightly different amounts of salt.

The Bottom Line

The paper concludes that ASIA is not a replacement for human experts, but rather a powerful assistant.

Think of it as a research accelerator. It can quickly explore thousands of possibilities that a human would never have the time to check. It finds the "best recipe" faster than anyone else, but humans are still needed to understand the ingredients, verify the results, and ensure the science makes sense.

The authors see this as a first step toward a future where AI helps engineers design better controllers, optimize machines, and solve complex engineering problems, but the human "head chef" is still in charge of the kitchen.

Technical Summary: ASIA – An Autonomous System Identification Agent

Problem Statement

System identification involves learning dynamical models from input-output data, a process requiring a complex sequence of interconnected design choices. Practitioners must select a model class (ranging from linear state-space to nonlinear neural networks), define architectural hyperparameters (e.g., network depth, kernel functions, recurrent cell types), and tune optimization hyperparameters (e.g., learning rates, regularization, batch sizes). Currently, these decisions rely heavily on empirical trial-and-error and domain expertise. While partial automation exists for fixed model families (e.g., via Bayesian optimization), the selection of the model class and the design of the training algorithm itself remain largely manual, time-consuming, and dependent on expert intuition.

Methodology: The ASIA Framework

The authors propose ASIA (Autonomous System Identification Agent), a framework that delegates the iterative search for optimal identification configurations to a Large Language Model (LLM) acting as an autonomous coding agent. Rather than introducing a new identification algorithm, ASIA automates the process of discovery.

Core Components

The framework operates on an iterative loop with minimal human intervention, requiring only a plain-English description of the identification problem. It consists of four main components:

Problem Description: A document detailing the task, available data, prior physical knowledge, candidate approaches, and the search protocol.
Immutable Evaluation Pipeline: A read-only system responsible for data preprocessing, cross-validation splitting, and computing the evaluation metric ( $V(\theta)$ ). This ensures all candidate solutions are evaluated under identical conditions.
Modifiable Components ( $\theta$ ): A set of code files representing model architectures, loss functions, and training strategies that the agent can freely edit.
Iterative Experimentation Loop:
- The agent analyzes the history of previous trials ( $T$ ), including code modifications and performance metrics.
- It formulates a new hypothesis and edits the relevant code to implement a new configuration $\theta_k$ .
- The agent executes the training and evaluation, observes the new metric, and updates its internal state.
- This cycle continues until a computational budget is exhausted or performance saturates.

The framework is instantiated using Claude Code powered by the Sonnet 4.6 LLM. It supports the injection of domain knowledge (e.g., physical laws, literature references) and candidate strategies but does not require them.

Empirical Case Studies

The authors evaluated ASIA on two distinct benchmarks, comparing its search efficiency and final model quality against baselines.

1. Cascaded Two-Tank System

Task: Identify a nonlinear system of two water tanks with unmeasured upper-tank levels and hard saturation nonlinearity.
Protocol: 2-fold cross-validation on a 1024-sample training trajectory; evaluation via open-loop simulation RMSE.
Search Process: Starting from a vanilla RNN (4 hidden units, RMSE 0.681), the agent:
- Iteration 3: Switched to a 2-layer LSTM (128 units) with truncated backpropagation, reducing RMSE to 0.430.
- Iteration 5: Introduced an inductive bias by replacing the LSTM with a Liquid Time-Constant (LTC/CfC) network (64 hidden states). This architecture, structurally predisposed to dissipative systems, achieved a validation RMSE of 0.250.
Result: The final ensemble model achieved a test RMSE of 0.298, performing on par with the best results reported in the benchmark literature.

2. Nanodrone Dynamics

Task: Identify the flight dynamics of a Crazyflie 2.1 quadrotor (12-state MIMO system) from real-world flight data.
Protocol: Leave-one-trajectory-out cross-validation; evaluation via Mean Absolute Error (MAE) on 50-step rollouts.
Search Process: Starting from a single-layer LSTM (MAE 0.707), the agent:
- Iteration 3: Introduced a physics-informed architecture (KinematicsLSTM) where the recurrent network predicts only velocity/angular-velocity increments, while position and attitude are recovered via explicit kinematic integration. It also augmented inputs with squared motor speeds to proxy thrust.
- Iteration 7: Implemented scheduled sampling to stabilize training, reducing MAE to 0.310.
- Subsequent iterations refined capacity and training cadence, reaching a final MAE of 0.286.
Comparison: ASIA outperformed a random search baseline (30 iterations) in both final performance and search efficiency, reaching the random search's best performance in just three iterations. The final model surpassed the best method from the original benchmark paper [9] across all output groups (position, velocity, attitude, angular velocity).

Key Contributions and Claims

The paper makes the following technical contributions and claims:

Autonomous Pipeline Automation: ASIA demonstrates that an LLM-based agent can autonomously conduct the full system identification pipeline, including model class selection, architectural design, and training strategy optimization, without human intervention beyond the initial prompt.
Hypothesis Space Expansion: Unlike classical optimization methods (grid search, Bayesian optimization) that operate over fixed, finite-dimensional parameter spaces, ASIA explores a combinatorial space of model classes and training strategies. The agent successfully proposed non-trivial solutions, such as hybrid physics/black-box structures and Koopman-inspired representations, which were not explicitly predefined.
Competitive Performance: The framework discovered models that match or exceed state-of-the-art results on established benchmarks, validating the feasibility of agentic AI in this domain.

Limitations and Critical Analysis

The authors explicitly discuss several limitations and risks associated with the approach:

Implicit Test Leakage: Even without direct access to test labels, the agent may be indirectly influenced by test performance logs or by prior knowledge (literature) derived from test-set evaluations, potentially leading to overfitting or implicit bias.
Methodological Transparency: While the generated code is locally interpretable, the global sequence of decisions leading to the final configuration is opaque. This reduces the "scientific contribution" of the methodology, as the rationale for specific design choices emerges from an automated, iterative process rather than explicit human hypothesis.
Reproducibility: Due to the stochastic nature of LLM outputs, the exact search trajectory cannot be perfectly reproduced across different runs. However, once a specific configuration is selected, the training of that specific model is reproducible using standard practices.
Bias: The agent's search is biased by its training data and internal priors, potentially favoring established architectures over novel, unconventional approaches.

Significance and Conclusion

The paper concludes that ASIA should not be viewed as a replacement for human-driven system identification but as a complementary tool. Its primary value lies in:

Accelerating the experimentation cycle.
Automating routine design choices.
Expanding the exploration of the hypothesis space beyond human intuition.

The authors emphasize that human expertise remains essential for interpreting results, validating models, designing prompts, and injecting relevant prior knowledge. This work serves as an initial empirical study into the potential and limitations of agentic AI in engineering research, suggesting future applications in controller tuning and experiment design.