Refining ensemble $N$-representability of one-body density matrices from partial information

This paper introduces a systematic relaxation of the ensemble $N$-representability problem for one-body density matrices with partial information, linking it to a generalized Horn's problem to derive explicit constraints and a convex polytope for lattice site occupations in excited-state density functional theory.

The Gist

Imagine you are trying to solve a giant, complex puzzle. In the world of quantum physics, this puzzle is figuring out how a group of electrons (tiny particles) are behaving together. Scientists use a tool called a "density matrix" to describe this behavior, but there's a catch: not every mathematical description of these electrons actually corresponds to a real, physical state of nature. This is known as the N-representability problem. It's like having a drawing of a house that looks perfect on paper but is physically impossible to build because the walls are too thin or the roof is upside down.

For a long time, scientists have had a set of basic rules (like the "Pauli Exclusion Principle") to check if a drawing is buildable. However, these rules are often too loose, allowing many "impossible" drawings to slip through.

This paper introduces a smarter way to filter these drawings, especially when we are looking at excited states (electrons that have been energized and are jumping to higher levels). Here is the breakdown of their new method:

1. The "Partial Knowledge" Advantage

Usually, when scientists try to predict how a group of electrons will behave, they start with almost no information about the specific states involved. They just know the general rules.

This paper says: "What if we already know some of the pieces?"
Imagine you are trying to guess the final shape of a sculpture. If you are told, "We know for a fact that the base of the sculpture is a perfect cube," that changes everything. You don't have to guess the base; you just have to figure out what can sit on top of that cube.

In the paper's terms, they assume we already know the "density matrix" (the blueprint) for the ground state (the lowest energy state) or some low-lying excited states. They ask: Given that we know these specific pieces, what are the new, stricter rules for the rest of the ensemble?

2. The "Relaxation" Strategy

The problem with knowing a specific blueprint is that it's incredibly complex. It involves not just the numbers (how many electrons are where) but also the specific "directions" or "orbits" they are taking. Calculating this perfectly is like trying to solve a Rubik's cube while blindfolded and wearing heavy gloves—it's too hard to do for large systems.

So, the authors propose a systematic relaxation.

• The Metaphor: Instead of keeping the full, detailed blueprint of the known pieces (which includes their exact orientation and shape), they throw away the orientation details and keep only the numbers (how many electrons are in each spot).
• The Result: They trade a tiny bit of precision for a massive gain in solvability. They replace the complex, rigid shape with a simpler "shadow" of that shape. This makes the problem solvable with standard math tools while still keeping the most important physical constraints.

3. The "Horn's Problem" Connection

To solve this simplified version, the authors connect their problem to a famous mathematical puzzle called Horn's Problem.

• The Metaphor: Imagine you have two buckets of water with specific amounts in them. You know the total amount of water you have, and you know the amount in the first bucket. The question is: What are the possible amounts you could have in the second bucket?
• Horn's Problem is the mathematical rulebook for figuring out the possible sums of these "buckets" (or eigenvalues). By combining this rulebook with their new "relaxed" rules, the authors create a new, tighter set of boundaries.

4. The "Tighter Net"

The main result of the paper is that by using this partial knowledge and the Horn's Problem connection, they can draw a much smaller, tighter net around the possible solutions.

• Old Way: The net was huge, letting many impossible electron configurations pass through.
• New Way: Because we know the "base" (the ground state), the net shrinks. It now excludes configurations that were previously allowed but are actually impossible given what we know about the ground state.

5. Why This Matters for "Lattice" Systems

The paper also shows how this applies to "lattice" systems (electrons sitting on specific grid points, like atoms in a crystal). They prove that this new method creates a "convex polytope" (a multi-sided geometric shape) that defines exactly which electron counts are allowed on these grid points.

• The Analogy: If you are trying to pack suitcases into a car, the old rules said, "As long as the total weight is under 500kg, you're good." The new rules say, "Since we know the trunk is already filled with a specific heavy box, you can only put suitcases in the back seat that weigh less than X." This prevents you from trying to pack a suitcase that would tip the car over.

Summary

In simple terms, this paper says: "If you know the blueprint for the ground state of a quantum system, you can use that knowledge to create much stricter, more accurate rules for the excited states."

They achieved this by:

1. Ignoring the overly complex "direction" details of the known states to make the math manageable.
2. Using a classic mathematical theorem (Horn's Problem) to figure out the limits of the remaining unknowns.
3. Creating a new set of "guardrails" that are much tighter than the old ones, ensuring that only physically possible electron configurations are considered.

This helps scientists avoid wasting time calculating impossible scenarios and leads to more accurate predictions of how molecules and materials behave when they are excited.

Technical Summary

1. Problem Statement

The $N$-representability problem is a fundamental challenge in quantum chemistry and many-body physics. It asks whether a given reduced density matrix (RDM) corresponds to a valid physical $N$-particle quantum state.

• Context: In Reduced Density Matrix Functional Theory (RDMFT) and Ensemble Density Functional Theory (EDFT), one seeks to describe excited states using ensembles of quantum states $\Gamma = \sum_i w_i |\Psi_i\rangle\langle\Psi_i|$ with fixed weights $w_i$.
• The Gap: While the domain of the universal functional for pure states is constrained by generalized Pauli constraints, the domain for ensembles (the $w$-ensemble) is traditionally defined only by the Pauli exclusion principle and normalization.
• The Specific Challenge: In practical calculations (e.g., sequential computation of ground and excited states), one often possesses partial information about the ensemble. Specifically, the full one-body reduced density matrix (1RDM), $\gamma^{(i)}$, of certain states (like the ground state) may be known a priori.
• The Core Question: How does fixing the 1RDMs of specific ensemble elements refine the set of admissible 1RDMs for the entire ensemble? The authors identify that the exact solution to this "refined" problem is non-convex and depends on the specific natural orbitals of the known states, making it computationally intractable for general systems.

2. Methodology

The authors propose a systematic hierarchy of relaxations to transform the intractable refined problem into a solvable spectral problem.

A. The Refined Problem Definition

They define a refined ensemble set $E^1_N(w, K_K)$ where the 1RDMs of a subset of states (indexed by $K$) are fixed to known values $\gamma^{(i)}$.

• Issue: This set is non-convex and not invariant under unitary transformations on the one-particle Hilbert space because it depends on the specific natural orbitals of the fixed $\gamma^{(i)}$.

B. Systematic Relaxation Strategy

To make the problem tractable, the authors introduce two levels of relaxation:

1. Convex Hull Relaxation: They first take the convex hull of the non-convex set, allowing for probabilistic mixtures of states that share the fixed 1RDMs. This yields a set $\tilde{E}^1_N(w, K_K)$ but still retains dependence on natural orbitals.
2. Spectral Relaxation (Key Step): They discard the information regarding the natural orbitals (eigenvectors) of the fixed 1RDMs, retaining only their natural occupation number vectors (eigenvalues), denoted as $\lambda^{(i)}$.
   • This transforms the problem into finding the set of admissible spectra $\Lambda(w, L_K)$ for the total ensemble, given fixed spectra for a subset of states.
   • This step converts the problem into a purely spectral one, invariant under unitary transformations.

C. Connection to Horn's Problem

The spectral relaxation is mathematically mapped to the Generalized Horn's Problem.

• Horn's Problem: Determines the possible eigenvalues of a sum of Hermitian matrices ($Y = \sum X^{(i)}$) given the eigenvalues of the summands.
• Application: The total 1RDM $\gamma$ is a weighted sum of fixed 1RDMs and a residual term. The authors show that the set of admissible total natural occupation numbers $\Lambda^\downarrow(w, L_K)$ is the intersection of:
  1. The solution to the Generalized Horn's problem (constraints on the sum of eigenvalues).
  2. The standard $w$-ensemble $N$-representability constraints (the spectral polytope $\Sigma(w)$).

D. Derivation of System-Size Independent Bounds

Recognizing that the full Horn constraints grow rapidly with system size (dimension $d$), the authors derive outer approximations.

• They derive explicit upper and lower bounds on the sums of the $k$-largest natural occupation numbers.
• These bounds depend on the known ground state spectrum $\lambda^{(1)}$ and the weights $w$, but crucially, they provide constraints that are independent of the system size (dimension $d$) and the number of particles $N$ in their functional form, making them scalable for large basis sets.

3. Key Contributions

1. Formulation of the Refined Problem: The paper formally defines the $w$-ensemble $N$-representability problem when partial 1RDM information is available, generalizing Coleman's classical solution.
2. Spectral Relaxation via Horn's Problem: The authors establish a rigorous link between refined ensemble representability and the Generalized Horn's problem, providing a complete (though complex) characterization of the admissible spectral set.
3. Explicit System-Size Independent Constraints: They derive a set of linear inequalities (hyperplane representation) that tighten the standard $w$-ensemble constraints. These constraints are:
   • Tighter: They strictly reduce the domain of admissible 1RDMs compared to standard ensemble theory.
   • Scalable: They do not require solving the full Horn problem for large $d$, making them applicable to realistic quantum chemical systems.
4. Convexification for Lattice DFT: The authors construct the convex hull of the refined spectral set, denoted $\Sigma(w, \lambda^{(1)})$. They prove this set is a permutahedron generated by a specific vertex, allowing for efficient implementation in lattice ensemble DFT.

4. Results

• Tightening of Residuals: The authors demonstrate that the "residuals" (the slack in the inequality constraints) of the standard $w$-ensemble conditions are significantly reduced when partial information is included.
  • Correlation Dependence: The tightening is most pronounced when the known state (e.g., the ground state) exhibits strong static correlation (i.e., its natural occupation numbers deviate significantly from the Hartree-Fock limit of 1s and 0s).
  • Numerical Validation: Using exact diagonalization on Hydrogen molecules ($H_2$ and $H_4$) in the cc-pVDZ basis, they show that the new lower bounds on residuals are strictly tighter than the trivial bounds ($R \ge 0$) in strongly correlated regimes (e.g., bond dissociation).
• Geometric Structure: The refined spectral set $\Lambda(w, \lambda^{(1)})$ is shown to be non-convex in general, but its convex hull $\Sigma(w, \lambda^{(1)})$ is a well-defined polytope.
• Lattice Site Occupation: The derived constraints are applied to lattice site occupation numbers in Ensemble DFT. The results show that knowing the ground state spectrum restricts the possible lattice site occupations for excited states, improving the predictive power of ensemble functionals.

5. Significance and Outlook

• Improved Accuracy in Functional Theories: The derived constraints offer a pathway to improve the accuracy of $w$-RDMFT and Ensemble DFT for excited states. By incorporating known ground-state information (natural occupation numbers) as additional constraints, the optimization landscape is restricted to a more physically relevant subset, reducing errors in approximate functionals.
• Sequential Calculation Strategy: The framework supports a sequential approach to excited-state calculations: as each state is computed, its spectral data can be fed back as a constraint for subsequent calculations, progressively refining the solution.
• Broader Applications: The methodology extends beyond functional theories to:
  • Variational 2-RDM methods: Providing constraints for excited-state solutions.
  • Quantum Tomography: Improving the reconstruction of quantum states from partial data.
  • Machine Learning: Offering physically grounded constraints to train neural networks for quantum systems, enhancing reliability and interpretability.

In summary, the paper bridges the gap between abstract mathematical constraints (Horn's problem) and practical quantum chemistry, providing a rigorous, scalable method to refine the description of excited states using partial information from ground-state calculations.