Exact density-functional theory as parallel ensemble variational hierarchies: from Lieb's formulation to Kohn-Sham theory

This paper reorganizes exact density-functional theory into two parallel variational hierarchies rooted in Lieb's interacting and exact ensemble noninteracting formulations, clarifying their relationship to the Kohn-Sham construction and unifying concepts like fractional particle numbers and derivative discontinuities within a single framework.

The Gist

Imagine you are trying to understand the weather in a massive, chaotic city. You want to predict the temperature at every single street corner without having to track the movement of every single air molecule, car, and person. This is the challenge of Density-Functional Theory (DFT) in physics: trying to understand complex systems of electrons by looking only at their "density" (how crowded they are in different spots), rather than tracking every single electron.

For decades, scientists have told the story of this theory as a single, straight line: "We started with a theorem, then we made a clever shortcut (Kohn-Sham), and that's how we do it."

Nan Sheng's paper argues that this story is too simple. It's like describing a complex machine as just "a box that works." The author suggests we need to look at the machine's internal gears.

Here is the paper's big idea, broken down with simple analogies:

1. The Two Parallel Worlds (The "Parallel Hierarchies")

The paper says exact DFT isn't one story; it's two parallel stories running side-by-side, connected by a bridge.

• World A: The Real, Messy City (Interacting Hierarchy)
  Imagine a crowded city where everyone bumps into everyone else. Electrons repel each other, dance around, and create a chaotic mess. This is the Interacting World. It's the "real" physics.

  • The Paper's Insight: To understand this world perfectly, we shouldn't just look at one specific arrangement of people (a "pure state"). We should look at ensembles—statistical averages of many possible arrangements. This allows us to handle "fractional" numbers (like having 2.5 electrons on average) and smooth out the math. This is Lieb's formulation.

• World B: The Ghost City (Non-Interacting Hierarchy)
  Now, imagine a "Ghost City" where people don't bump into each other at all. They glide past one another perfectly. This is the Non-Interacting World. It's a mathematical fantasy, but it's much easier to calculate.

  • The Paper's Insight: Even this Ghost City has its own complex rules and "ensemble" versions. It's not just a simple list of empty streets; it has its own deep structure involving fractional occupations (ghosts that are half-present).

The Bridge: The famous Kohn-Sham theory is the bridge connecting these two worlds. It says, "If we can make the Ghost City look exactly like the Real City (same density), we can use the easy Ghost City math to solve the hard Real City problem."

2. The "Remainder" is Actually a "Bridge"

In the old story, the Exchange-Correlation part of the theory was treated as the "unknown garbage."

• Old View: "We know the easy part (Kinetic) and the crowd part (Hartree). Everything else we don't understand, so we just call it 'Exchange-Correlation' and hope for the best."
• New View (The Paper): Exchange-Correlation is not just garbage. It is the Interface. It is the specific quantity that measures the difference between the Real City and the Ghost City. It's the "translation cost" required to make the Ghost City mimic the Real City. By seeing it as a bridge between two parallel worlds, we understand its structure much better.

3. The "Staircase" vs. The "Ramp" (Fractional Numbers)

One of the most confusing parts of quantum physics is what happens when you add a tiny bit of an electron (fractional particle number).

• The Analogy: Imagine a staircase. You can stand on step 1 or step 2, but not in between.
• The Paper's Insight: If you look at the "Real City" with the right mathematical tools (ensembles), the energy doesn't jump up and down like a staircase. It forms a straight line connecting step 1 and step 2.
  • This "straight line" (Piecewise Linearity) is a fundamental law of nature.
  • If your approximation (your model) creates a curve or a bump between the steps, you have broken the laws of physics. The paper explains why this happens: it's because the "Real City" allows for a mix of states (ensembles), creating a smooth path between integer steps.

4. The "Gap" Problem (Why the Bridge isn't Perfect)

Scientists often look at the "energy gap" (the difference between the highest occupied seat and the lowest empty seat) in the Ghost City to predict how the Real City behaves.

• The Problem: The Ghost City's gap is usually too small. It misses a crucial piece of the Real City's energy.
• The Paper's Explanation: This isn't a mistake in the math; it's a feature of the two parallel worlds.
  • The Real City gap is about the cost of adding/removing a person (a "slope" in the energy).
  • The Ghost City gap is just the difference between two seats in a quiet room.
  • The missing piece is called the Derivative Discontinuity. It's a "jump" in the math that happens exactly when you cross an integer number of electrons. The paper clarifies that this jump is the "price" you pay to translate the Ghost City's simple spectrum into the Real City's complex reality.

Summary: Why Does This Matter?

Think of the old way of teaching DFT as a "magic trick." "Here is the formula, it works, don't ask how."

This paper pulls back the curtain. It says:

1. Stop compressing the story. There are two distinct worlds (Real and Ghost) running in parallel.
2. Respect the Ensembles. The math works best when we allow for "fuzzy" averages (ensembles) rather than demanding perfect, single-state certainty.
3. Re-evaluate the "Unknowns." The things we call "Exchange-Correlation" or "Derivative Discontinuity" aren't just errors or leftovers. They are the structural glue holding the two parallel worlds together.

By understanding the architecture of these two parallel worlds, scientists can build better, more accurate models for chemistry and materials science, rather than just patching up broken formulas. It turns a "black box" into a clear, logical map.

Technical Summary

1. Problem Statement

Standard introductions to Ground-State Density-Functional Theory (DFT) typically follow a compressed historical narrative: starting with the Hohenberg–Kohn (HK) theorem, moving to the Levy constrained search, and culminating in the Kohn–Sham (KS) construction. While elegant, this narrative often conflates formally distinct layers of the theory, blurring critical logical distinctions between:

• Interacting vs. Noninteracting theories: Treating the noninteracting system merely as a tool for KS rather than an exact variational theory in its own right.
• Ensemble vs. Pure-state formulations: Overlooking the necessity of ensemble (mixed-state) formulations to handle convexity, fractional particle numbers, and derivative discontinuities rigorously.
• Functional domains vs. Representability classes: Confusing the domain where a functional is well-defined with the smaller class of densities that admit a supporting potential (v-representability).
• Density reproduction vs. Spectral interpretation: Assuming that KS orbital energies directly map to many-body ionization potentials without acknowledging the structural gap.

The paper argues that these distinctions are not mere technical caveats but are native features of the exact theory that are obscured by compressed expositions.

2. Methodology

The author employs a formal reorganization of existing mathematical results rather than deriving new theorems. The methodology involves:

• Convex Analysis and Duality: Utilizing the Banach space setting established by Lieb, where the density $\rho$ and external potential $v$ are treated as dual variables. This allows the use of Legendre–Fenchel transforms, subgradients, and convex conjugates to define exact functionals.
• Parallel Hierarchies: Constructing two distinct but parallel variational frameworks:
  1. An Interacting Hierarchy rooted in Lieb's ensemble formulation.
  2. A Noninteracting Hierarchy rooted in exact ensemble noninteracting theory (based on the kinetic energy functional $T_s$).
• Auxiliary Coupling: Defining the Kohn–Sham theory not as the starting point, but as the specific auxiliary construction that links these two hierarchies on a common admissible density class via the exchange-correlation decomposition.
• Specialization Analysis: Systematically demonstrating how familiar formulations (HK, Levy, pure-state KS) arise as narrower specializations of these broader ensemble hierarchies under additional restrictions (e.g., non-degeneracy, pure-state constraints).

3. Key Contributions

A. The Parallel Ensemble Hierarchy Framework

The paper posits that exact DFT is best understood as two parallel structures:

1. Interacting Hierarchy: Starts with Lieb's ensemble formulation. The universal functional $F[\rho]$ is the convex dual of the ground-state energy $E[v]$. This framework naturally accommodates fractional particle numbers and defines representability via subgradients ($\partial F$).
2. Noninteracting Hierarchy: Starts with the exact noninteracting kinetic energy functional $T_s[\rho]$, defined as the infimum of $\text{tr}(\Gamma_s T)$ over noninteracting ensembles. It possesses its own dual energy functional $E_s[v_s]$ and supporting potential structure.

Key Insight: The Kohn–Sham construction is the bridge between these two, not the definition of the noninteracting side itself.

B. Clarification of Fractional Quantities and Derivative Discontinuity

• Fractional Particle Number: In the ensemble framework, fractional $N$ is not an external correction but a natural consequence of convexifying the state space. The energy $E(N)$ is piecewise linear between integers (PPLB relation).
• Derivative Discontinuity: The "kink" in the energy vs. particle number curve at integer $N$ corresponds to a jump in the supporting potential. The paper clarifies that the derivative discontinuity is the potential-side manifestation of the non-differentiability of the energy functional, arising naturally from the ensemble geometry.
• One-Sided Chemical Potentials: Ionization energy ($I$) and electron affinity ($A$) are identified as the left and right slopes of the energy curve, respectively.

C. Reinterpretation of Kohn–Sham Theory

• Auxiliary Nature: KS theory is an auxiliary construction that couples the interacting and noninteracting hierarchies. It is exact in the sense of density reproduction and total energy minimization, but the KS orbitals and eigenvalues are properties of the noninteracting hierarchy, not the interacting one.
• Janak's Relation: The relation $\partial E / \partial n_i = \epsilon_i$ is reinterpreted as an occupation-space stationarity condition within the exact ensemble KS framework, rather than a universal spectral identification of orbital energies with many-body addition/removal energies.
• The Gap Decomposition: The fundamental gap is decomposed as $E_g^{true} = E_g^{KS} + \Delta_{xc}$. The paper emphasizes that $E_g^{KS}$ is an auxiliary spectral gap, while the true gap is a many-body slope quantity. The derivative discontinuity $\Delta_{xc}$ is the interfacial term required to reconcile the two.

D. The Exchange-Correlation Functional as an Interface

The paper redefines the exchange-correlation functional ($E_{xc}$). Instead of being a "residual" unknown, $E_{xc}$ is the structured interface quantity that records the difference between the interacting and noninteracting hierarchies after the Hartree term is removed.

• It encodes the kinetic energy difference and the non-classical interaction effects.
• Exact constraints (scaling, Lieb-Oxford bound, adiabatic connection) are shown to be structural properties of this interface, reflecting the relationship between the two hierarchies.

4. Results and Findings

• Hierarchy of Formulations: The paper establishes a clear inclusion hierarchy:
  • Broadest: Lieb's Ensemble DFT (Interacting) and Exact Ensemble Noninteracting DFT.
  • Intermediate: Levy Constrained Search (Pure-state restriction).
  • Narrowest: Hohenberg-Kohn (Non-degenerate pure-state) and standard KS (Pure-state determinant).
  • Result: Pure-state formulations are exact only when the ensemble minimizer happens to be a pure state; otherwise, they are approximations of the true ensemble minimum.
• Representability: The distinction between the domain of the functional ($\text{dom } F$) and the set of densities admitting a supporting potential ($\text{dom } \partial F$) is rigorously maintained. The latter is a subset of the former, and the closure of the latter equals the former.
• Spectral Interpretation: The paper concludes that interpreting KS eigenvalues as exact many-body ionization potentials requires additional assumptions (e.g., asymptotic behavior, specific system types) and is not a direct consequence of the KS construction itself.

5. Significance

• Conceptual Clarity: The paper resolves long-standing conceptual confusions in DFT by separating the "interacting" and "noninteracting" variational problems before linking them. This prevents the conflation of auxiliary spectral data with physical many-body observables.
• Foundation for Approximations: By framing $E_{xc}$ as a structured interface rather than a random remainder, the paper provides a more rigorous basis for deriving exact constraints for approximate functionals. It highlights that errors in approximate functionals (e.g., curvature between integers) are structural failures to preserve the exact ensemble geometry.
• Pedagogical Restructuring: It suggests that DFT should be taught and understood as a pair of linked ensemble variational theories, with KS as the coupling mechanism, rather than a linear historical progression from HK to KS.
• Handling of Fractional Charges: It provides the most robust theoretical framework for understanding fractional charges, piecewise linearity, and derivative discontinuities, which are critical for describing charge transfer, band gaps, and strong correlation.

In summary, Nan Sheng's work offers a "formal reorganization" that elevates the ensemble perspective to the central organizing principle of exact DFT, clarifying the logical status of every major component of the theory and distinguishing between what is exact by construction (density reproduction) and what requires specific physical conditions (spectral interpretation).