← Latest papers
🔬 materials science

sdos: spectral densities of large sparse Hermitian and generalized eigenproblems

The paper introduces **sdos**, an open-source Python package that efficiently computes the density of states and related spectral quantities for very large sparse Hermitian and generalized eigenproblems using Chebyshev moments and the kernel polynomial method, demonstrating high performance and scalability on massive atomic systems for both electronic and vibrational applications.

Original author: David A. Drabold

Published 2026-11-10
📖 1 min read☕ Coffee break read

Original author: David A. Drabold

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the author. For technical accuracy, refer to the original paper. Read full disclaimer

Technical Summary: sdos – Spectral Densities of Large Sparse Hermitian and Generalized Eigenproblems

Problem Statement
The paper addresses the computational challenge of determining the density of states (DOS) and related spectral quantities for very large sparse Hermitian matrices (N∼105N \sim 10^5 to 10710^7) and generalized eigenproblems (Aψ=λBψA\psi = \lambda B\psi). Traditional diagonalization, which scales as O(N3)O(N^3) in time and O(N2)O(N^2) in memory, is infeasible for systems such as amorphous networks, nanostructures, and large supercells. The goal is to compute the spectral density ρ(E)=N−1Tr δ(E−H)\rho(E) = N^{-1}\text{Tr}\,\delta(E-H) and derived quantities (band energies, Fermi levels, thermodynamics) with linear scaling in NN.

Methodology
The sdos package implements a stochastic moment-based approach. The core algorithm relies on estimating Chebyshev moments μn=N−1Tr Tn(H~)\mu_n = N^{-1}\text{Tr}\,T_n(\tilde{H}) of a scaled operator H~\tilde{H} using random vectors.

  • Stochastic Trace Estimation: Moments are estimated via μn≈N−1ξ†Tn(H~)ξ\mu_n \approx N^{-1}\xi^\dagger T_n(\tilde{H})\xi, where ξ\xi are random vectors (Rademacher or random phases). The trace is replaced by a stochastic estimator, reducing the cost per moment to O(N)O(N) operations via matrix-vector products.
  • Reconstruction: The moments are reconstructed into a density using two primary methods:
    1. Kernel Polynomial Method (KPM): A damped Chebyshev series (default using the Jackson kernel), which is linear, positive, and guaranteed to converge.
    2. Maximum Entropy (Maxent): Finds the positive density of maximal entropy reproducing the moments. This method resolves sharp features better than KPM but requires solving a convex dual problem.
  • Generalized Problems: For Aψ=λBψA\psi = \lambda B\psi, the method applies B−1AB^{-1}A via preconditioned conjugate gradients (PCG) or replaces B−1B^{-1} with a fixed Chebyshev polynomial p(B)≈B−1p(B) \approx B^{-1} to avoid inner solves.
  • Exact Low-Order Moments: The package can compute exact low-order moments (n≤Kn \le K) locally (scaling O(N)O(N)) and substitute them into the stochastic estimator to reduce variance significantly.
  • Additional Features: The code supports spectral windows (via shift-invert or Chebyshev filtering), exact HOMO–LUMO gap determination via inertia counting, local/projected DOS (Mulliken weights), and matrix-free operators for vibrational problems (Hessians from force fields).

Key Contributions
The paper presents sdos as an open-source Python implementation that integrates established methods with specific technical innovations and validations:

  1. Hybrid Moment Estimator: A novel approach combining exact low-order moments (up to order KK) with sampled higher-order moments in a single estimator. The paper provides arguments for when this is unbiased and quantifies the variance reduction (factors of 35–50 compared to plain random vectors).
  2. Robust Maxent Solver: Implementation of a Maxent solver featuring a "moment-uncertainty floor" (σn≥s0\sigma_n \ge s_0) and a bootstrapped climbing strategy. This addresses the ill-conditioning of Maxent when applied to exact moments of singular densities (e.g., van Hove singularities), ensuring convergence where plain solvers fail.
  3. Generalized Eigenproblem Handling: Implementation of multiple strategies for applying the inverse overlap matrix B−1B^{-1} (PCG with block-Jacobi, incomplete Cholesky, algebraic multigrid, and Chebyshev polynomial approximation), verified to yield identical moments.
  4. Comprehensive Validation: The code is validated against dense diagonalization and independent codes (DFTB+, SIESTA, LAMMPS) across diverse systems:
    • Electronic: Orthogonal tight-binding (Kwon Hamiltonian), non-orthogonal DFTB, and SIESTA.
    • Vibrational: Matrix-free Hessians from Stillinger-Weber and ACE potentials.
    • Scale: Tests range from small clusters to an 884,736-atom amorphous silicon model and a 2,097,152-atom crystal.

Results

  • Performance: On a workstation (4 threads, 19 GB RAM), the DOS of an 884,736-atom amorphous silicon model (3.5 million orbitals) is computed in 71 seconds (stochastic moments, M=100,R=2M=100, R=2) or 383 seconds (with default exact moments). A matrix-free 2,097,152-atom crystal calculation takes 10 minutes in 1.8 GB.
  • Accuracy:
    • KPM: Converges steadily; at M=400M=400 moments, the L1L_1 error against the exact spectrum is 3.7×10−33.7 \times 10^{-3}.
    • Maxent: With exact moments and the uncertainty floor, Maxent achieves band-energy errors 5–60 times smaller than KPM for M≤400M \le 400. However, for pointwise resolution of fine structure at very high MM, KPM remains superior as Maxent can oscillate.
    • Generalized Problems: The three methods for B−1B^{-1} (PCG, Chebyshev, IC(0)) yield identical moments. The Chebyshev polynomial approach avoids iterative solver stacks, making it suitable for matrix-free/GPU settings, though it may consume more memory.
  • Applications:
    • Fullerenes: Reproduced the approach of fullerene DOS to graphene-like spectra for C3840C_{3840}, resolving van Hove peaks.
    • Amorphous Silicon: Validated against exact Bloch spectra for a 17317^3 replica of a 216-atom cell (1 million atoms). Maxent successfully resolved the gap edges that KPM smeared.
    • Thermodynamics: Calculated zero-point energy and heat capacity for large systems, agreeing with exact results within 0.07% (ZPE) and 0.1% (heat capacity at 300 K).

Significance and Claims
The paper positions sdos not as a discovery of new fundamental physics, but as a rigorous, open, and tested implementation of linear-scaling spectral methods.

  • Modesty: The author explicitly states that the underlying methods (Chebyshev moments, KPM, Maxent, non-orthogonal KPM) are due to Silver, Röder, Kress, Voter, Drabold, Sankey, and collaborators. The contribution lies in the combination of these methods, the specific implementation details (e.g., the uncertainty floor for Maxent, the hybrid moment estimator), and the quantitative validation across a wide range of system sizes and problem types.
  • Limitations: The paper notes that shift-invert and exact gap bisection do not scale linearly due to fill-in. Maxent with high-order exact moments can be ill-conditioned without the uncertainty floor. The spectral bounds estimation can fail for matrix-free operators if the finite-difference step is too large.
  • Availability: The package is released under BSD-3-Clause, available via PyPI and GitHub, with extensive test suites and benchmark scripts included.

In summary, sdos provides a scalable, validated tool for computing spectral densities of large sparse systems, offering a choice between the robustness of KPM and the sharpness of Maxent, with specific optimizations for generalized eigenproblems and variance reduction.

Drowning in papers in your field?

Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.

Try Digest →