Reinforcement Learning Assisted Quantum Simulation of Many-Body Excited States and Real-Time Dynamics

This paper extends the reinforcement learning contracted quantum eigensolver (RL-CQE) to efficiently compute many-fermion electronic excited states and real-time dynamics by employing a deep Q-network to adaptively select compact two-body operators, achieving chemical accuracy with scalable state representations and constant-scaling time evolution.

The Gist

Imagine you are trying to solve a massive, complex puzzle. In the world of quantum chemistry, this puzzle is figuring out how electrons behave in molecules, especially when they are excited (like when a plant absorbs sunlight) or when they are moving rapidly over time.

Traditionally, solving this puzzle on a quantum computer is like trying to climb a mountain by taking tiny, fixed steps in every direction at once. It works, but it's slow, requires a huge amount of energy, and if you take a wrong step, you might get stuck.

This paper introduces a smarter way to climb that mountain using a "guide" called Reinforcement Learning (RL). Here is how the authors' new method works, broken down into simple concepts:

1. The Problem: The "All-at-Once" Climb

The old method (called CQE) tries to adjust the entire puzzle solution simultaneously. Imagine trying to fix a tangled ball of yarn by pulling on every single strand at the same time. It's messy, and you often end up with a knot that is hard to untangle. In quantum terms, this means the computer needs to run a very long, complex sequence of operations (a deep "circuit") to get the right answer.

2. The Solution: The "Smart Guide" (RL-CQE)

The authors replaced the "pull everything at once" strategy with a Reinforcement Learning agent. Think of this agent as a highly skilled hiker with a map.

• How it works: Instead of pulling all strands, the hiker looks at the current state of the puzzle and asks, "Which single move will get me closest to the solution right now?"
• The Result: The hiker picks the best move, takes it, and then re-evaluates. This creates a much shorter, more direct path to the solution. The paper shows that this "one-move-at-a-time" approach uses far fewer steps (operators) than the old method while still reaching the same high level of accuracy (chemical accuracy).

3. Tackling the "Excited" States

Usually, quantum computers are great at finding the "ground state" (the most relaxed, calm state of a molecule). But nature is often dynamic; molecules get excited, jump to higher energy levels, and do crazy things.

• The Challenge: Finding these excited states is like trying to find the peaks of several different mountains at the same time.
• The Innovation: The authors adapted their "Smart Guide" to handle multiple mountains at once. They proved that the guide can navigate these complex, excited landscapes just as well as the calm ground states. They also showed that the guide doesn't need to know the exact weight of every mountain beforehand; it can figure out the right balance on its own, making it much more robust and less likely to fail.

4. The Time Travel Problem: Simulating Motion

Simulating how a molecule changes over time (real-time dynamics) is usually a nightmare for quantum computers.

• The Old Way: To simulate 10 seconds of time, you might need to break it into 1,000 tiny steps. To simulate 100 seconds, you need 10,000 steps. The "circuit" (the list of instructions) grows longer and longer until the computer crashes.
• The New Way: The authors discovered a trick. Because they are looking at a group of states together (the "purified ensemble"), they can reuse the same set of "moves" for the entire duration of the simulation.
• The Analogy: Imagine you are recording a video. The old method is like filming every single frame individually and storing them all, requiring massive storage. The new method is like realizing that the camera movement follows a specific pattern. You only need to store the pattern (the fixed set of moves) and the starting point. No matter how long the video is, the "storage" (circuit size) stays the same. This allows them to simulate time evolution without the computer getting overwhelmed.

5. The Proof: Testing on Simple Molecules

The authors tested this new "Smart Guide" on two simple molecules: Hydrogen ($H_2$) and a chain of three Hydrogens ($H_3^+$).

• The Results: The guide found the correct energy levels for these molecules across different shapes and distances with incredible precision.
• Efficiency: It did this using a very small number of steps (sometimes as few as 2 or 5 moves), whereas the old method would have required many more.
• Time: When simulating these molecules moving over time, the "circuit" size remained constant, proving that the method scales well and doesn't get heavier as time goes on.

Summary

In short, this paper presents a new way to use quantum computers to study how molecules behave when they are excited or moving. By using an AI "guide" that picks the best single move at each step, they created a method that is:

1. Faster: It needs fewer steps to solve the puzzle.
2. Smarter: It handles complex, excited states without needing perfect prior knowledge.
3. Scalable: It can simulate time passing without the computer getting bogged down by an ever-growing list of instructions.

This brings us closer to using today's limited quantum computers to solve real-world problems in chemistry and physics that were previously impossible to simulate.

Technical Summary

Technical Summary: Reinforcement Learning Assisted Quantum Simulation of Many-Body Excited States and Real-Time Dynamics

Problem Statement
Accurately computing electronic excited states and real-time quantum dynamics for many-fermion systems remains a fundamental challenge, particularly for near-term quantum devices. Standard single-reference methods often fail due to the strong multi-configurational character and near-degeneracy found in these states. While the Contracted Quantum Eigensolver (CQE) offers a scalable approach based on the contracted Schrödinger equation (CSE), its extension to excited states via a purified ensemble approach suffers from slow convergence and sensitivity to critical hyperparameters, specifically the ensemble weight vector. Furthermore, conventional time-evolution simulations typically rely on Trotterization, resulting in circuit depths that grow unboundedly with simulation time, posing a significant barrier for long-duration dynamics.

Methodology
This work generalizes the Reinforcement Learning Contracted Quantum Eigensolver (RL-CQE), previously developed for ground states, to address excited states and real-time dynamics. The core methodology involves:

1. RL-CQE for Excited States: The algorithm formulates the CQE update procedure as a Markov Decision Process (MDP). A Deep Q-Network (DQN) agent acts as the policy, adaptively selecting a single two-body operator from a pool of sign-free qubit operators at each iteration. The state representation for the agent is the vector of Anti-Hermitian Contraction Schrödinger Equation (ACSE) residuals. Crucially, the dimension of this state vector depends only on the one-particle basis size and is independent of the number of targeted excited states ($K$). The agent maximizes a reward function combining energy minimization and residual suppression.
2. Sign-Free Operator Equivalence: The authors extend the theoretical validation of sign-free qubit operators (which differ from standard fermionic operators by non-local sign factors) to the excited-state regime, demonstrating their equivalence to the original fermionic operators in this context.
3. Time Evolution via Shared Unitary Structure: To simulate real-time dynamics $|\Psi(t)\rangle = e^{-i\hat{H}t}|\Psi(0)\rangle$, the authors leverage the purified ensemble framework where all targeted eigenstates share the same set of unitary transformations. By expanding the time-dependent wavefunction in the basis of these shared eigenstates, the time evolution is expressed as a fixed set of unitary transformations acting on a time-dependent reference state. This reference state is prepared using a secondary set of unitaries, also optimized via RL. This approach ensures the total number of operators remains constant regardless of the simulation time $t$.

Key Contributions

• Generalization of RL-CQE: The first application of RL-CQE to the direct optimization of excited-state wavefunctions within a quantum eigensolver framework.
• Scalable State Representation: Introduction of a state representation based on ACSE residuals that scales with the basis size but remains independent of the number of excited states, overcoming a major bottleneck in ensemble methods.
• Robustness to Hyperparameters: Demonstration that the RL-based adaptive selection of operators yields solutions significantly more robust to the choice of ensemble weight vectors compared to conventional CQE, which requires precise, system-specific tuning.
• Constant-Scaling Time Evolution: Development of a time-evolution algorithm that maintains a fixed ansatz size (constant operator count) independent of simulation time, contrasting with the linearly or polynomially growing depth of Trotter-based methods.
• Theoretical Extension: Verification of the equivalence of sign-free qubit operators in the excited-state setting, extending a result previously established only for ground states.

Results
The algorithm was benchmarked on the $H_2$ molecule and the linear equidistant $H_3^+$ ion across various bond lengths:

• Excited State Energies: For $H_2$, RL-CQE achieved chemical accuracy (within $10^{-3}$ Hartree of Full Configuration Interaction) using at most 5 unitary transformations. The method proved largely insensitive to the ensemble weight vector; simple strictly descending weight vectors (e.g., $[4, 3, 2, 1]$) performed comparably to optimized vectors, whereas conventional CQE is highly sensitive to this choice.
• Operator Efficiency: In $H_2$, the algorithm converged with only 2 operators for various bond lengths, significantly fewer than the simultaneous updates required by conventional CQE. For $H_3^+$, the method successfully reproduced potential energy curves and selected distinct, geometry-adapted operator sequences.
• Time Evolution: Applied to $H_2$ and $H_3^+$, the RL-CQE time-evolution algorithm achieved high fidelity (near unity) with a fixed number of steps (5 for $H_2$, 20 for $H_3^+$) regardless of the simulation time $t \in [0, 20]$ a.u. This confirms the theoretical prediction of constant scaling in time.

Significance
The paper claims that RL provides an effective and flexible framework for minimizing quantum resource requirements in hybrid quantum-classical algorithms. By enabling compact ansätze and robust convergence without prior knowledge of system symmetry or complex hyperparameter tuning, RL-CQE addresses critical limitations of current variational quantum eigensolvers. The ability to simulate real-time dynamics with a constant operator count offers a pathway to scalable many-body dynamics on near-term hardware, potentially extending to open quantum systems, non-equilibrium dynamics, and larger molecular systems where conventional methods become intractable. The work establishes a foundation for using model-free reinforcement learning to navigate complex optimization landscapes in quantum chemistry without requiring gradient information or explicit modeling of the quantum device's internal details.