🔬 condensed matter

Prediction of the Hubbard U parameter from scanning tunneling microscopy images of moire systems using image recognition

This paper presents a machine learning methodology that successfully predicts the Hubbard U parameter from scanning tunneling microscopy images of twisted bilayer graphene, demonstrating robustness against experimental noise and revealing interaction-dependent spectral weight redistribution.

Original authors: Nachiket Tanksale, Tobias Stauber

Published 2026-07-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nachiket Tanksale, Tobias Stauber

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery inside a tiny, invisible city made of atoms. In the world of quantum physics, this city is often a special kind of material called "twisted bilayer graphene," where two sheets of carbon are stacked and twisted slightly, creating a giant, repeating pattern known as a "moiré" superlattice. Inside this city, electrons don't just float around; they bump into each other with a force called the "Hubbard U." Think of this force like a strict bouncer at a club: if the bouncer is too strict (a high U), the electrons refuse to hang out in the same spot, creating a "correlated" state that can lead to amazing things like superconductivity (electricity flowing with zero resistance). The problem is, we can't easily see how strict this bouncer is just by looking at the crowd. We need a way to measure this invisible "U" parameter to understand how the material behaves.

Enter the detective's tool: a Scanning Tunneling Microscope (STM). This machine acts like a super-powered eye that takes pictures of the electrons' "local density of states" (LDOS)—essentially a map of where the electrons are hanging out. But here's the twist: when you look at these maps for different levels of electron strictness, they look almost identical to the human eye, like two twins wearing the same outfit. For years, scientists have struggled to tell them apart. This paper asks a bold question: If a human can't see the difference, can a computer learn to spot the tiny, hidden clues that reveal exactly how strict the electron bouncer is?

The authors of this study say yes, but with a catch: they had to teach a computer to be a super-detective using a technique called machine learning. They didn't just look at the raw pictures; they transformed them into "Fourier-transformed" images, which is like taking a photo of a city and then looking at its shadow to see the underlying structure of the streets. They created thousands of simulated STM images for twisted bilayer graphene under different conditions, ranging from a very relaxed bouncer (U = 0 eV) to a very strict one (U = 5 eV).

The results are impressive. The researchers trained two different types of artificial intelligence (a custom-built neural network and a modified version of a famous one called ResNet-18) to predict the value of U directly from these images. Even though the images looked nearly identical to us—with a similarity score of over 99.98%—the AI could distinguish them with remarkable accuracy. In their simulations, the models predicted the interaction strength with an average error of just 0.12 to 0.14 eV, which is incredibly precise.

However, the paper is careful not to claim this is a magic wand for real-world experiments just yet. The team tested their AI on "noisy" and imperfect images to mimic real-world scanning errors, like a shaky hand or a dirty lens. They found that while the AI struggled a bit more with these imperfections, training it on messy, simulated data actually helped it get better at handling real-world noise. This suggests that the method is robust, but it also highlights that the AI is currently learning from perfect, computer-generated simulations.

One of the most fascinating discoveries was how the AI solved the puzzle. By using "interpretability tools," the authors peeked inside the AI's brain and saw that it wasn't just guessing. At low interaction levels, the AI looked at the whole picture, but as the interaction got stronger, it focused its attention on very specific, tiny spots in the center of the image. This suggests the AI learned to detect a subtle shift in how the electrons' energy is distributed—a "spectral weight redistribution"—that humans can't see but the machine can.

In short, this paper suggests that we can use machine learning to read the "personality" of electrons in these complex materials by analyzing their microscopic shadows. While the results are currently based on simulations, the study provides a promising roadmap for turning blurry, confusing microscope images into precise measurements of the forces that drive the next generation of quantum materials. It's a step toward teaching computers to see the invisible rules that govern the quantum world.

Technical Summary: Prediction of the Hubbard U Parameter from Scanning Tunneling Microscopy Images of Moiré Systems Using Image Recognition

Problem Statement
The on-site Hubbard interaction parameter ( $U$ ), representing local Coulomb repulsion, is a critical input for theoretical models of correlated electron systems, particularly in twisted bilayer graphene (TBG) and other moiré materials. However, determining $U$ experimentally remains a significant challenge. While Scanning Tunneling Microscopy (STM) provides high-resolution real-space images of the local density of states (LDOS), quantitatively extracting microscopic interaction parameters like $U$ directly from these images is difficult. This is because the Fourier-transformed LDOS (FT-LDOS) images corresponding to different interaction strengths exhibit extremely high visual similarity, making simple distance metrics insufficient for differentiation.

Methodology
The authors propose a machine learning (ML) framework to infer the Hubbard $U$ parameter directly from simulated FT-LDOS images of TBG in the flat-band regime.

Theoretical Model: The system is modeled as twisted bilayer graphene under hydrostatic pressure ( $\theta \approx 3.5^\circ$ ), which mimics the magic-angle condition. The Hamiltonian includes a tight-binding description, long-range Coulomb interactions (screened by a double-gated potential), and an on-site Hubbard term ( $U$ ). The system is solved self-consistently using the restricted Hartree-Fock approximation within the FORGE package, assuming spin-symmetric solutions to preserve spatial and valley symmetries.
Data Generation: FT-LDOS images are generated for integer filling $\nu=3$ and a fixed dielectric constant ( $\epsilon=10$ ). The dataset covers $U$ values from 0 to 6 eV. The training set consists of 600 images based on discrete anchor points ( $U = 0, 1, \dots, 5$ eV), while a separate holdout set of 66 samples with fractional $U$ values is used to test interpolation capabilities.
Preprocessing: Raw FT-LDOS maps are Fourier transformed, rescaled to $256 \times 256$ pixels, converted to grayscale, and standardized (zero mean, unit variance) to ensure numerical stability.
Neural Network Architectures: Two Convolutional Neural Networks (CNNs) are employed:
- A custom CNN designed to preserve lattice symmetries and facilitate physical interpretation, consisting of three convolutional blocks followed by fully connected layers.
- A modified ResNet-18 (pre-trained on ImageNet) with a regression head, serving as a robust feature extractor for subtle momentum-space variations.
Robustness and Augmentation: To assess performance under realistic experimental conditions, the authors introduce physically motivated perturbations to the data, including Gaussian noise, intensity rescaling, anisotropic blurring (simulating tip effects), scan-line artifacts, and affine distortions (simulating weak heterostrain and symmetry breaking).
Interpretability: Gradient-weighted Class Activation Mapping (Grad-CAM) and Guided Backpropagation are used to identify which momentum-space regions and pixel-level features drive the regression predictions.

Key Results

High Regression Accuracy: Both CNN architectures successfully regress $U$ from FT-LDOS images with high precision. On the standard test set, the custom CNN achieved a Mean Absolute Error (MAE) of $0.14 \pm 0.03$ eV and an $R^2$ of $0.984 \pm 0.003$ . The ResNet-18 model performed comparably with an MAE of $0.12 \pm 0.02$ eV and $R^2$ of $0.985 \pm 0.006$ .
Interpolation Capability: The models demonstrated the ability to interpolate to unseen fractional $U$ values in the holdout set, though with reduced accuracy (MAE $\approx 0.5$ eV, $R^2 \approx 0.85$ ), indicating that generalization beyond the training manifold is possible but more challenging.
Feature Extraction: Despite the visual similarity of FT-LDOS images (cosine similarity $> 0.9998$ ), the CNNs successfully extracted weak, systematic features. Principal Component Analysis (PCA) revealed that the dataset lies on a low-dimensional manifold, with the first principal component (PC1) capturing 56.8% of the variance and showing a nearly monotonic dependence on $U$ .
Physical Interpretation: Interpretability tools revealed that the networks focus on specific momentum-space features. At weak coupling ( $U \lesssim 1$ eV), attention is distributed across central and satellite moiré peaks. As $U$ increases, attention shifts to the relative intensity contrast between these features, eventually localizing near the $\Gamma$ -point in the strong-coupling regime ( $U \gtrsim 4$ eV). The networks respond primarily to sharp gradients at the edges of reciprocal lattice peaks rather than background noise.
Robustness: Augmentation-based training significantly improved the models' generalization to noisy and symmetry-broken images, demonstrating that the learned representations are not merely artifacts of idealized simulations but are robust to experimental imperfections.

Significance and Claims
The paper claims to demonstrate a systematic methodology for extracting the Hubbard $U$ parameter directly from LDOS images using machine learning. The primary significance lies in proving that, despite the extreme visual similarity of FT-LDOS images across different interaction strengths, deep learning models can identify subtle, interaction-dependent redistributions of spectral weight.

The authors suggest that this approach opens new opportunities for quantitatively determining interaction parameters in moiré systems, potentially aiding in the engineering of magnetic phases. Furthermore, the analysis of the learned features suggests a gradual interaction-dependent redistribution of spectral weight, with a possible crossover scale of order $U/t \sim 1$ . The work establishes that image-based ML techniques can bridge the gap between experimental STM observables and microscopic Hamiltonian parameters, provided that robust training strategies (including data augmentation) are employed to handle experimental noise and symmetry breaking.

Technical Summary: Prediction of the Hubbard U Parameter from Scanning Tunneling Microscopy Images of Moiré Systems Using Image Recognition

More like this