🔬 condensed matter

Large scale theoretical investigation of the phase diagram of twisted bilayer MoTe $_2$ at fractional fillings: agreements and contradictions with current experiments

This study employs comprehensive exact-diagonalization calculations on twisted bilayer MoTe $_2$ to demonstrate that including interaction-driven band mixing in a two-band-per-valley model is essential for accurately reproducing the experimentally observed hierarchy of fractional Chern insulators and charge density wave states, while also ruling out the proposed non-abelian Pfaffian state at specific fillings.

Original authors: Heqiu Li, Jiabin Yu, Xiaodong Xu, B. Andrei Bernevig, N. Regnault

Published 2026-08-26

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Heqiu Li, Jiabin Yu, Xiaodong Xu, B. Andrei Bernevig, N. Regnault

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the hidden world of solid materials, electrons usually behave like a chaotic crowd, bumping into one another and moving in random directions. But under the right conditions, they can organize into a highly ordered, almost liquid state where their collective behavior creates new rules of physics. For decades, scientists have studied a famous example of this: the fractional quantum Hall effect. This phenomenon occurs when electrons are trapped in a flat layer and subjected to a powerful magnetic field, forcing them to form a rigid, crystalline-like structure that conducts electricity without resistance in a very specific, exotic way. The challenge has always been that this state requires an enormous, unwieldy magnetic field to create, making it difficult to study or use in practical devices.

Recently, a breakthrough occurred in the laboratory using a material called twisted bilayer MoTe2. By stacking two sheets of this material and twisting them slightly relative to each other, researchers created a new, repeating pattern of hills and valleys known as a moiré superlattice. This pattern acts like a built-in magnetic field, allowing the electrons to organize themselves into similar exotic states without needing any external magnets. These new states are called fractional Chern insulators. They are the solid-state cousins of the fractional quantum Hall effect, but they exist naturally within the twisted material. The question that has kept physicists busy is whether these materials can reliably produce these exotic states across different conditions, or if the complex interactions between electrons and the material's structure will disrupt them.

A team of researchers has now performed a massive, detailed computer simulation to map out exactly where these exotic states appear and where they disappear. They focused on twisted bilayer MoTe2, testing a wide range of twist angles, from about 2.13 degrees to 4 degrees, and various levels of electron density. Using powerful supercomputers, they solved the equations governing how thousands of electrons interact with one another within these twisted layers. Their goal was to create a complete map, or phase diagram, that predicts what state the material will be in under any given condition, and to see if their predictions matched the growing list of real-world experiments.

The researchers found that the answer depends heavily on how many layers of the material's internal structure they include in their calculations. Early theories often looked at just the topmost layer of electron energy levels, a simplification that worked well for some cases but failed to explain others. By expanding their view to include the second layer of energy levels, the team discovered that the electrons mix between these layers in ways that significantly change the outcome. This "band mixing," as it is called, acts like a filter, stabilizing some exotic states while destroying others.

Their most significant finding is a clear agreement with recent experiments at a twist angle of roughly 3.7 degrees. In this region, the simulations confirmed that the material forms robust, stable exotic states at specific electron densities, such as when the material is filled to two-thirds, three-fifths, or four-sevenths of its capacity. These states are the fractional Chern insulators that experimentalists have been hunting for. However, the simulations also explained why these states do not appear at other nearby densities. For instance, at a filling of one-third, the material does not form the exotic insulator; instead, it settles into a different, more common state called a charge density wave, where the electrons arrange themselves in a static pattern. This distinction, which was previously a point of confusion between different experimental groups, was resolved by the team's more complete model.

The study also clarified what happens at other twist angles. At smaller angles, around 2.13 degrees, the material behaves differently, and the researchers looked for a particularly rare and complex state known as a non-Abelian state. This state is special because it could potentially be used for a new type of quantum computing that is resistant to errors. Despite the theoretical promise, the team's simulations found no evidence that this state forms in the material under the conditions they tested. Similarly, at a half-filled density, the material was expected to behave like a strange, fluid metal known as a composite Fermi liquid. While the simplified models predicted this fluid state clearly, the more accurate, multi-layered simulations showed that the fluid becomes unstable and breaks down when the mixing between energy layers is taken into account.

By comparing their results with the "no-fitting" models derived directly from the fundamental properties of the atoms, the team ensured their predictions were not just artifacts of their mathematical choices. They found that the complex interplay between the different energy layers is essential for getting the physics right. Without including these extra layers, the simulations would have predicted exotic states in places where experiments show nothing, and missed them in places where they are clearly observed.

Ultimately, this work provides a unified and reliable guide for understanding twisted bilayer MoTe2. It tells scientists exactly where to look for these exotic states and where to expect them to fail. The research confirms that the material is a fertile ground for discovering new forms of matter, but it also sets strict boundaries on what is possible. The exotic states are real and robust, but they are fragile, existing only in a narrow window of conditions where the electrons can find the perfect balance between order and interaction. This detailed map not only validates the recent experimental successes but also offers a clear path forward for future research, showing that to truly understand these materials, one must look beyond the surface and account for the complex, layered nature of the electrons inside.

Technical Summary: Large Scale Theoretical Investigation of the Phase Diagram of Twisted Bilayer MoTe2 at Fractional Fillings

Problem Statement
Twisted bilayer MoTe2 (tMoTe2) has emerged as a primary platform for realizing Fractional Chern Insulators (FCIs) without external magnetic fields. While recent experiments have observed robust FCIs at various fractional hole fillings (e.g., $\nu = -2/3, -3/5, -4/7$ ) and specific twist angles (notably $\approx 3.7^\circ$ ), discrepancies exist between experimental observations and theoretical predictions. Specifically, single-band theoretical models often predict Fractional Chern Insulator (FCI) states at fillings where experiments observe Charge Density Waves (CDW) or no incompressible state (e.g., $\nu = -1/3, -4/3$ ). Furthermore, the role of multi-band physics, specifically the mixing between the top two moiré valence bands, remains a critical but computationally challenging variable to quantify across the full phase diagram. The central problem addressed is to establish a unified, quantitative numerical benchmark that reconciles theoretical models with experimental data by rigorously testing the necessity of multi-band effects.

Methodology
The authors employ large-scale Exact Diagonalization (ED) to investigate the interaction-driven phases of tMoTe2 across a wide range of twist angles ( $2.13^\circ$ to $4^\circ$ ) and hole fillings. The study utilizes two distinct single-particle models:

First-Harmonic (FH) Continuum Model: A widely used model where parameters are fitted to Density Functional Theory (DFT) band structures at a specific angle ( $3.89^\circ$ ).
DFT "No-Fitting" Model: A parameter-free approach constructed directly from DFT calculations for discrete twist angles ( $2.13^\circ$ to $3.89^\circ$ ), avoiding continuous parameter fitting and Wannierization.

The many-body calculations are performed using two projection schemes:

One-Band-Per-Valley (1BPV): Projects interactions onto the top moiré valence band only.
Two-Bands-Per-Valley (2BPV): Includes the top two moiré valence bands per valley, explicitly accounting for interaction-driven band mixing.

The interaction is modeled using a normal-ordered screened Coulomb potential relative to charge neutrality. The authors systematically compare ground state properties, including energy gaps, spin polarization, momentum degeneracy, and particle entanglement spectra (PES), against established criteria for identifying FCIs, CDWs, Composite Fermi Liquids (CFL), and non-Abelian states.

Key Contributions and Results

1. Resolution of Discrepancies at Odd-Denominator Fillings
The study demonstrates that the 2BPV calculation is essential for quantitative agreement with experiments, particularly at fillings where 1BPV fails:

$\nu = -2/3, -3/5, -4/7$ : The 2BPV results reproduce the experimentally observed hierarchy of robust FCIs around $\theta \approx 3.7^\circ$ . The 2BPV model correctly identifies incompressible states at these fillings while predicting the absence of an FCI at $\nu = -3/7$ for angles near $3.7^\circ$ , matching experimental reports.
$\nu = -1/3$ : While 1BPV calculations often predict an FCI, the 2BPV results (and the no-fitting model) consistently identify a K-CDW (charge density wave with ordering vector at $K_M$ ) as the ground state near $3.7^\circ$ . This aligns with recent experimental reports of CDW order at this filling, resolving a long-standing theoretical contradiction.
$\nu = -4/3$ : The 1BPV model predicts a fully spin-polarized FCI, contradicting experiments. The 2BPV calculation suppresses this spurious spin polarization and FCI, yielding a ground state with near-zero spin polarization, consistent with experimental observations.

2. Impact of Band Mixing on Spin Polarization and Stability
The inclusion of the second band (2BPV) significantly alters the phase diagram by suppressing spurious spin-polarized FCIs. The study finds that band mixing reduces the stability of ferromagnetic FCI phases at $\nu = -1/3$ and $-4/3$ , while robust FCIs at $\nu = -2/3, -3/5, -4/7$ persist over a wide range of twist angles ( $2.45^\circ$ to $4^\circ$ ). The authors also note that the "quick basis" in the no-fitting model, which yields a smaller bandwidth, tends to stabilize FCIs at lower interaction strengths compared to the "standard basis," highlighting the sensitivity of the $\nu = -1/3$ regime to band dispersion details.

3. Even-Denominator Fillings and Composite Fermi Liquids

$\nu = -1/2$ : In the 1BPV limit, the system exhibits clear signatures of a Composite Fermi Liquid (CFL), including the expected ground state degeneracy and momentum distribution. However, in the 2BPV calculation, band mixing introduces additional low-energy many-body states that destabilize the CFL ground state, suggesting that the half-filled state is more delicate and sensitive to multi-band effects than previously thought.
$\nu = -3/2$ : Motivated by the analogy to Landau levels at small twist angles ( $\theta \approx 2.13^\circ$ ), the authors searched for a non-Abelian Pfaffian state in the fully spin-polarized sector. The ED results find no evidence for this state; the expected six-fold quasi-degenerate ground state structure is not observed, and low-energy states with zero total momentum disrupt the Pfaffian signature.

4. Gap Scaling and Finite-Size Effects
The authors analyze the scaling of FCI gaps across the Jain sequence ( $\nu = -2/3, -3/5, -4/7$ ). They find that the gaps approximately follow a linear scaling with $1/(2p+1)$ , similar to Fractional Quantum Hall Effect (FQHE) gaps in the lowest Landau level. The study also emphasizes the importance of momentum mesh selection, noting that meshes containing the $K_M$ points are crucial for accurately capturing the bandwidth and stable FCI gaps.

Significance
The paper establishes a unified numerical benchmark for correlated and topological phases in twisted bilayer MoTe2. Its primary significance lies in demonstrating that multi-band physics is not merely a correction but is essential for a quantitative comparison with experiments. The 2BPV framework successfully reconciles theoretical predictions with the experimental hierarchy of FCIs, the suppression of FCIs at specific fillings (like $\nu = -3/7$ and $-4/3$ ), and the emergence of CDW orders at $\nu = -1/3$ . By clarifying where single-band approximations fail and where multi-band effects are critical, the work provides a refined theoretical foundation for interpreting current experimental data and guiding future investigations into non-Abelian states and competing orders in moiré systems.

Technical Summary: Large Scale Theoretical Investigation of the Phase Diagram of Twisted Bilayer MoTe2 at Fractional Fillings

More like this