Orbital Optimization and Neural-Network-Assisted Configuration Interaction Calculations of Rydberg States

This paper presents a method combining variational orbital optimization with plane-wave basis sets and neural-network-assisted configuration interaction to accurately calculate Rydberg states of molecules, overcoming the limitations of traditional diffuse atomic basis sets and achieving results in close agreement with experimental data.

The Gist

Imagine you are trying to take a photograph of a very shy, invisible ghost that lives far away from a house. This ghost is a Rydberg electron. In chemistry, when an atom or molecule gets excited, an electron jumps up to a very high energy level. Unlike normal electrons that huddle close to the nucleus (the "house"), this Rydberg electron spreads out like a giant, faint cloud that stretches for miles.

The problem scientists face is that their standard tools (called "atomic basis sets") are like taking a photo with a camera that has a very short zoom lens. No matter how much they try, the camera cuts off the edges of the photo. The ghost looks squashed and trapped inside the house, rather than the giant, diffuse cloud it actually is. This leads to wrong calculations about how much energy it takes to create this ghost.

This paper presents a clever two-step solution to fix this "squashed ghost" problem.

Step 1: Redrawing the Map (Orbital Optimization)

Think of the electrons as dancers on a stage. Usually, scientists calculate the dance moves based on how the dancers look when the music is slow and calm (the ground state). But when the music speeds up (the excited state), the dancers spread out and move differently.

The authors say: "Why not teach the dancers to practice specifically for the fast music before we try to choreograph the complex routine?"

They used a special technique called Plane Waves. Imagine instead of using a grid of small, fixed tiles to describe the stage (which leaves gaps at the edges), they used a flexible, infinite sheet of fabric that can stretch as far as the dancers need to go. By optimizing the "dance moves" (orbitals) specifically for the excited state using this infinite fabric, they captured the true, giant shape of the Rydberg electron.

Step 2: The Smart Filter (Neural Network CI)

Once they had the right map of the stage, they still faced a massive problem: calculating the exact energy of the system requires considering every possible way the electrons could dance together. For a molecule like water or ammonia, the number of possible dance combinations is larger than the number of atoms in the universe. It's impossible to check them all.

This is where the Neural Network (AI) comes in.
Imagine you are looking for a specific needle in a haystack the size of a mountain.

• The Old Way: You pull out every single piece of hay, one by one, to see if it's the needle. This takes forever.
• The New Way (NNCI): You train a smart robot (the Neural Network) to look at a few pieces of hay and guess which ones might be the needle. The robot learns to ignore the 99.999% of hay that is definitely not the needle and only focuses on the tiny fraction that matters.

In this paper, the AI acts as a filter. It learns to identify which electron configurations are important and which are irrelevant. It turns a calculation that would take a supercomputer a million years into one that takes a few hours, while still getting the answer right.

The Results: A Perfect Match

The authors tested this method on three molecules: Hydrogen ($H_2$), Ammonia ($NH_3$), and Water ($H_2O$).

1. Hydrogen: They showed that if you use the old "short lens" method, the energy is way off (like guessing a ghost is 40 feet tall when it's actually 10). But with their new "infinite fabric" map, the calculation matched the theoretical gold standard perfectly.
2. Ammonia and Water: These are trickier. The AI filter, combined with the new map, produced results that matched real-world experiments almost exactly.
   • The "Trap": When other scientists used standard tools (even expensive ones), they got results that were too high because they couldn't see the "tails" of the electron cloud.
   • The Fix: By stretching the map (Plane Waves) and using the AI to pick the right dancers, the authors got the energy levels right, even for the most difficult, spread-out electrons.

The Big Picture

This paper is like inventing a new camera lens that can zoom out infinitely to catch a ghost, and then hiring a smart AI assistant to quickly find the ghost in the photo without checking every single pixel.

Why does this matter?
Rydberg states are crucial for understanding how molecules react in the atmosphere, how they interact with light, and even in quantum computing. By making these calculations faster and more accurate, this method helps scientists design better materials, understand chemical reactions, and solve problems that were previously too difficult to crack.

In short: They fixed the "blurry edges" of the electron cloud and used AI to ignore the noise, allowing them to see the true nature of these elusive, high-energy electrons.

Technical Summary

1. Problem Statement

Accurately calculating the electronic structure of Rydberg excited states in molecules is a significant challenge in quantum chemistry. These states are characterized by highly diffuse electron distributions that extend far from the molecular core.

• Basis Set Limitations: Conventional atomic orbital basis sets (even large, diffuse ones like aug-cc-pVTZ) often fail to adequately represent the long-range tails of Rydberg orbitals. This leads to an artificial confinement of the electron, resulting in significant overestimation of excitation energies.
• Methodological Limitations: Standard methods like Time-Dependent Density Functional Theory (TDDFT) and Equation-of-Motion Coupled Cluster (EOM-CC) struggle with Rydberg states due to basis set limitations and approximations in electron correlation treatment.
• Computational Cost: Full Configuration Interaction (FCI) calculations, which provide exact solutions within a given basis, become computationally intractable for molecules with more than a few electrons due to the exponential scaling of the Hilbert space.

2. Methodology

The authors propose a hybrid approach combining state-specific orbital optimization using plane waves with a Neural-Network-assisted Selective Configuration Interaction (NNCI) method.

A. State-Specific Orbital Optimization (Hartree-Fock)

• Plane Wave Representation: Instead of relying solely on localized atomic orbitals for the initial guess, the authors use a plane wave (PW) basis set in Hartree-Fock (HF) calculations.
• Variational Optimization: Molecular orbitals (MOs) are variationally optimized specifically for the target excited state (e.g., a specific Rydberg state) rather than the ground state.
  • For excited states, the calculation targets a saddle point on the electronic energy surface, promoting an electron from the HOMO to the target Rydberg orbital.
  • This ensures the occupied orbitals (including the Rydberg orbital) are fully flexible and can capture the diffuse nature of the electron density without the confinement imposed by atomic basis sets.
• Initialization: The PW optimization is initialized using localized atomic orbitals (e.g., cc-pVTZ or aug-cc-pVTZ) plus numerical $s$-type orbitals, but the final MOs are represented by plane waves.

B. Neural-Network-Assisted Selective CI (NNCI)

• Hamiltonian Formulation: The many-body problem is solved using the Hamiltonian $H = H_0 + H_{int} - \langle H_{int} \rangle_{MF}$, where integrals are evaluated using the excited-state optimized HF orbitals.
• Iterative Selection: The NNCI method iteratively selects the most relevant Slater determinants from the full Hilbert space.
  • Initialization: A small FCI calculation is performed on a subset of low-energy orbitals to generate an initial set of determinants.
  • Neural Network Classifier: A neural network is trained to classify candidate determinants as "important" or "irrelevant" for the convergence of the target observable (excitation energy).
  • State Tracking: The algorithm specifically tracks the target excited state by identifying the state with the maximal occupation of the diffuse Rydberg orbital.
• Efficiency: This approach avoids the combinatorial explosion of FCI by including only a tiny fraction of the total determinants required for convergence.

3. Key Contributions

1. Demonstration of Orbital Optimization: The paper proves that optimizing orbitals specifically for the excited state using a plane wave basis significantly improves the description of Rydberg states compared to using ground-state orbitals or standard atomic basis sets.
2. Extension of NNCI to Excited States: The authors successfully extend their previously developed NNCI method (originally for ground states) to excited electronic states, handling both single-configurational and multiconfigurational cases.
3. Overcoming Basis Set Confinement: The study demonstrates that even when initialized with standard atomic basis sets (like aug-cc-pVTZ), the subsequent PW-based orbital optimization corrects the artificial confinement of Rydberg orbitals, yielding results comparable to much larger basis sets.
4. Computational Efficiency: The method achieves FCI-level accuracy using orders of magnitude fewer determinants (approx. $10^5$ vs. $10^{10}$ for full CI) by leveraging the neural network selection.

4. Results

The methodology was tested on three molecules: H₂, NH₃, and H₂O.

• H₂ (2s Rydberg State):

  • Full CI was performed to benchmark the method.
  • Using ground-state orbitals with cc-pVTZ resulted in a massive overestimation of excitation energy (~4 eV error) due to orbital confinement.
  • Using excited-state optimized PW orbitals reduced the error to <0.1 eV, matching the Theoretical Best Estimate (TBE) almost perfectly.
  • The position of the avoided crossing with the $(1\sigma_u)^2$ state was also accurately reproduced only with excited-state optimized orbitals.

• NH₃ (3s and 3p Rydberg States):

  • NNCI calculations were performed using 52 MOs.
  • Calculations using ground-state orbitals with aug-cc-pVTZ overestimated the 3p excitation energy by ~0.8 eV compared to experiment.
  • Excited-state optimized orbitals brought the NNCI results into close agreement with experimental values and high-level extrapolated FCI (exFCI) calculations.
  • Convergence was achieved with only $\sim 10^5$ determinants (5 orders of magnitude fewer than full CI).

• H₂O (3s, 3p, and Multiconfigurational States):

  • The method successfully handled the multiconfigurational character of the lowest $1A_1$ excited states (mixing of 3s from HOMO-1 and 3p from HOMO).
  • Standard high-level methods (like GMS SU CCSD) using aug-cc-pVTZ overestimated the 3p excitation energy by >1 eV due to basis set confinement.
  • The NNCI with PW-optimized orbitals matched experimental data and high-level EOM-CCSDTQ/exFCI results, despite using a smaller initial atomic basis set for initialization.
  • The method correctly captured the mixing weights (~55% / ~40%) of the configurations.

5. Significance

• Accuracy vs. Cost: The paper presents a highly efficient workflow that delivers high-accuracy results for challenging Rydberg states without requiring prohibitively large atomic basis sets or full CI calculations.
• General Applicability: The approach is not limited to Rydberg states; the authors suggest it is equally beneficial for other types of excited states where ground and excited state orbitals differ significantly, such as long-range charge-transfer excitations.
• Future Directions: The use of plane waves naturally suggests extensions to continuum problems and shape resonances, where the target state is metastable and embedded in a continuum.
• Benchmarking: The results provide a new benchmark for calibrating faster, less accurate methods (like TDDFT) for Rydberg states, highlighting the critical need for proper orbital treatment in excited-state calculations.