⚡ electrical engineering

Privileged observations enable rapid and reliable policy discovery directly in the physical world

This paper demonstrates that privileged observations of a chaotic physical system are decisive for enabling reinforcement learning agents to rapidly discover high-performing policies directly in the real world, as agents lacking such flow data failed to learn drag-increasing strategies despite successfully learning drag-reducing ones.

Original authors: Antonio Terpin, Raffaello D'Andrea

Published 2026-09-15

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Antonio Terpin, Raffaello D'Andrea

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a swimmer trying to move through water. If they could see the currents swirling around them, they might learn to twist their body in a way that cuts through the resistance, or perhaps in a way that catches the water to push themselves forward. This is the essence of active flow control: a field where engineers try to manipulate the invisible, chaotic movement of fluids to improve how objects move or stay still. For decades, scientists have known that spinning a cylinder in a stream can change how much the water pushes against it. Sometimes, a steady spin reduces the drag, letting the object slip through more easily. Other times, a specific rhythmic spinning can actually increase the drag, holding the object back. The challenge has always been figuring out the exact timing and speed of that spin to get the best result, especially because water flow is chaotic and hard to predict.

Usually, when engineers want to teach a computer how to control such a system, they build a virtual model of the water and let the computer practice there. But virtual models are never perfect; they miss the tiny, messy details of real water. So, a team of researchers at ETH Zürich decided to skip the simulation entirely. They built a physical water channel and taught a computer agent to learn directly from the real, churning water. The question they asked was simple but profound: to learn the best way to control the water, does the computer need to see the water swirling around the object while it is learning? Or can it figure it out just by feeling the push and pull on the object itself?

The researchers set up a tabletop water channel, a clear tank where water circulates in a loop. Inside, a cylinder sits in the flow, and a motor can spin it at any speed or direction. To measure how well the water is pushing against the cylinder, they used a sensitive torque sensor. To see the water, they used a technique called particle image velocimetry. This involves shining a laser sheet through the water, which contains tiny floating particles, and taking rapid photos. By tracking how those particles move between photos, a computer can build a detailed, real-time map of the water's speed and direction right behind the cylinder. This map is the "privileged observation"—extra information that the computer can see during training but might not need later.

They trained a general-purpose learning agent, a type of artificial intelligence, to control the cylinder. In one set of experiments, the agent was given both the feeling of the drag and the detailed map of the water flow. In another set, the agent was given only the feeling of the drag, with the water map hidden from it. The goal was twofold: first, to find a way to spin the cylinder that would reduce the drag as much as possible, and second, to find a way to spin it that would increase the drag as much as possible.

The results were striking and revealed a surprising asymmetry. When the agent had access to the detailed map of the water flow, it learned incredibly fast. Within minutes of interacting with the real water, it discovered a strategy to reduce the drag by about 32 percent. It also quickly found a strategy to increase the drag by about 25 percent. These numbers matched or even slightly exceeded the performance of the best known human-designed strategies, which had been developed over decades of study.

However, when the researchers hid the water map and forced the agent to learn using only the drag measurement, the story changed completely. The agent was still able to learn how to reduce the drag, eventually achieving a reduction of about 31 percent. It just took a little longer and was a bit more variable. But when the goal was to increase the drag, the agent failed. Without seeing the water flow, none of the training runs managed to learn a strategy that significantly increased the drag. The agent could not figure out how to make the water push harder, even though the best possible strategy existed and was physically possible.

To understand why this happened, the researchers looked closely at what the water was doing. They found that whenever the cylinder started to spin, the water almost always reacted by first pushing less against the cylinder, regardless of whether the goal was to push more or less. This initial drop in drag is a natural physical response. For the agent trying to reduce drag, this initial drop was a helpful sign that it was on the right track. But for the agent trying to increase drag, this initial drop was confusing; it felt like the opposite of what it wanted to achieve. Without the detailed map of the water to see the bigger picture and understand that the drag would eventually rise, the agent got stuck in this initial dip and never learned the correct strategy.

The researchers then tested whether the strategies the agent learned with the water map could be used without it. They recorded the exact spinning patterns the agent used when it had the map, and then replayed those exact patterns as fixed sequences, without any new observations. Surprisingly, these fixed sequences worked just as well as the live, thinking agent. The agent had learned a specific set of movements that worked so well that it didn't need to keep watching the water to execute them. This proves that the water map was not needed to perform the task, but it was absolutely essential to discover the task.

This finding highlights a crucial distinction in learning: what you need to find a solution is often different from what you need to use it. In this case, the rich, detailed view of the water flow acted as a teacher, guiding the agent through the confusing initial reactions of the fluid. Once the agent found the right path, it could walk it blindfolded. The study suggests that in complex physical systems, having access to extra information during the learning phase can be the deciding factor between success and failure, even if that information is not available when the system is actually running. It shows that for artificial intelligence to master the messy, chaotic real world, it may need to be allowed to see more than it will ever be allowed to use.

Technical Summary: Privileged Observations for Physical Policy Discovery

Problem Statement
This study investigates how "privileged information"—observations available during training but potentially unavailable during deployment—affects the discovery of high-performing policies when training a reinforcement learning (RL) agent directly on a physical system. The authors focus on active flow control, a domain characterized by infinite-dimensional, chaotic nonlinear dynamics (governed by the Navier–Stokes equations) that are difficult to model or simulate accurately. Specifically, the task involves controlling the rotation of a cylinder in a water channel to either minimize or maximize the hydrodynamic drag acting on it. While decades-old experimental studies have established that simple, periodic open-loop policies (e.g., sinusoidal rotation) can significantly alter drag, the challenge lies in discovering these effective sequences autonomously without prior knowledge of the optimal strategy.

Methodology
The authors designed and deployed a low-cost, tabletop circulating water channel (CWC) instrumented with a real-time, hardware-accelerated Particle Image Velocimetry (PIV) system. This setup allows for dense, full-field flow observations (velocity vectors and vorticity) to be computed and fed to the agent during training.

System: A cylinder (radius $R=0.021$ m) is rotated by a brushless motor within a confined water channel. The flow is driven by propellers and stabilized by honeycomb structures and guide vanes. The Reynolds number is approximately $5.0 \times 10^3$ .
Agent: The authors utilize DreamerV3, a generalist, model-based RL algorithm, with default hyperparameters and no task-specific inductive bias.
Observation Sets: The study compares two observation configurations:
1. Augmented Set: Includes the standard scalar drag measurement, previous commanded rotation rate, motor feedback, plus the dense flow field estimate from the PIV system.
2. Reduced Set: Includes only the scalar drag measurement, previous command, and motor feedback (withholding flow observations).
Reward: The reward is defined as the change in drag relative to the episode start ( $r_t = s(d_t - d_0)$ ), where $s=+1$ for maximization and $s=-1$ for minimization.
Protocol: Training occurs directly on the physical system over 60-minute budgets (60 episodes). The authors perform extensive ablations, varying observation sets, model sizes, and learning algorithms. They also record action trajectories from successful online runs and replay them as fixed open-loop sequences to test if online feedback is required for execution.

Key Results
The experiments reveal a striking asymmetry in how privileged observations influence policy discovery depending on the task objective:

Drag Maximization:
- With Flow Observations: The agent rapidly discovers high-performing policies, increasing drag by 25.5% ± 0.9% within tens of minutes.
- Without Flow Observations: The agent fails to discover a high-performing policy across all 10 training runs, achieving only a negligible 1.8% ± 4.3% increase.
- Replay: Action trajectories recorded during online execution (with flow observations) can be replayed open-loop (without observations) and still achieve a 23.2% ± 2.2% drag increase, demonstrating that the actions themselves do not require continuous flow feedback to be effective, but the discovery of these actions does.
Drag Minimization:
- With Flow Observations: The agent learns to reduce drag by 32.4% ± 1.6%.
- Without Flow Observations: The agent still succeeds, reducing drag by 31.4% ± 2.1%, though learning is slower and more variable.
- Replay: Open-loop replay of recorded trajectories achieves a 32.1% ± 3.0% reduction, comparable to online execution.
Mechanism of Asymmetry:
The authors identify a "trajectory-level inverse response" as the cause of the asymmetry. In 94.4% of online trajectories, the agent's initial actions cause an immediate decrease in drag, regardless of the ultimate goal.
- For minimization, this initial decrease provides a positive learning signal, allowing the agent to refine the policy even without flow data.
- For maximization, this initial decrease acts as a negative learning signal (adverse to the goal), misleading the agent. Privileged flow observations allow the agent to "see" the underlying flow dynamics and correct this initial misstep, enabling the discovery of the counter-intuitive, high-drag strategy.
Ablations:
- The failure to maximize drag without flow observations persists across different model sizes (1M vs. 25M parameters) and learning algorithms (PPO, SAC), confirming the result is not an artifact of a specific architecture.
- The asymmetry is robust across different observation set configurations.

Significance and Claims
The paper claims to provide the first physical demonstration that privileged observations can be decisive for policy discovery even in scenarios where the resulting optimal actions can be executed in open loop without those observations.

Decoupling Discovery from Execution: The study separates the difficulty of finding a high-performing policy from the necessity of using rich observations during execution. It shows that while dense flow fields are not strictly required to execute a high-drag strategy (as proven by replay), they are essential for the RL agent to discover that strategy in the presence of an initial adverse physical response (non-minimum phase behavior).
Physical RL Feasibility: The work demonstrates that generalist RL agents can rapidly and reliably discover complex control policies in chaotic, real-world fluid systems without relying on high-fidelity simulations, provided they have access to rich, privileged training data.
Practical Implication: The findings suggest that rich, expensive, or computationally intensive sensing (like full-field PIV) can be highly valuable during laboratory training phases to "bootstrap" policy discovery, even if such sensors are impractical to deploy continuously in the field. The trained policy can then operate effectively with a reduced observation set.

The authors remain modest regarding generalization, noting that the results are specific to the tested reward specifications, training budgets, and the strongly wall-confined flow regime of their apparatus. They frame the work as a starting point for mapping how observation availability interacts with physical dynamics to shape learning outcomes.

Technical Summary: Privileged Observations for Physical Policy Discovery

More like this