Closed Orbits and Descents for Enhanced Standard Representations of Classical Groups

1. Problem Statement and Motivation

The paper addresses fundamental geometric problems in the representation theory of classical groups over an algebraically closed field $F$ of characteristic 0. The primary motivation stems from two major problems in the study of representations of $G(F)$ (where $F$ is a local field):

1. The Multiplicity One Problem: Determining whether the dimension of $\text{Hom}_{G(F)}(\pi, \mathbb{C})$ is at most 1 for an irreducible smooth admissible representation $\pi$. This often reduces to proving that certain equivariant generalized functions vanish, a task requiring detailed knowledge of the geometry of group actions.
2. Smooth Transfer: Establishing the existence of transfer factors and matching orbital integrals between different groups (e.g., in the Jacquet-Rallis relative trace formula).

To tackle these, one must understand the geometry of the action of a reductive group $G$ on a rational representation $V$. Specifically, the paper focuses on the Enhanced Standard Representation, denoted as $g \times E$, where:

• $G$ is a classical group: $GL_n(F)$, $O_n(F)$, or $Sp_{2n}(F)$.
• $g$ is the Lie algebra of $G$.
• $E$ is the natural representation (or the sum of the natural representation and its dual for $GL_n$).
• The action is extended to the MVW-extension $\check{G} = G \rtimes \{\pm 1\}$, which includes an involution (transpose or conjugation by a specific matrix).

Core Objectives:

1. Classify all closed orbits of $G$ (and $\check{G}$) in the space $g \times E$.
2. Prove that every closed $G$-orbit is stable under the larger group $\check{G}$.
3. Determine the descendants (stabilizer groups and their actions on the normal space) for these closed orbits.

2. Methodology

The author employs a combination of Geometric Invariant Theory (GIT), Lie algebra decomposition, and explicit matrix construction.

• Invariant Theory: The classification relies heavily on the algebra of invariants $F[g \times E]^G$. Theorem 2.2 identifies the generators of this algebra (traces and specific bilinear forms), allowing the authors to characterize closed orbits via the fibers of the quotient morphism $\pi: g \times E \to G \backslash (g \times E)$.
• Luna's Criterion: The paper utilizes Luna's criterion (Theorem 2.1), which states that an orbit $Hx$ is closed if and only if the orbit of the centralizer $Z_H(K)x$ is closed, where $K$ is the stabilizer of $x$. This reduces the problem to analyzing stabilizers of specific "canonical" elements.
• Jordan Decomposition: For general elements in $g \times E$, the authors use the Jordan decomposition $X = X_s + X_n$ (semisimple + nilpotent). They reduce the classification of closed orbits in the full space to the classification of closed orbits in the nilpotent cone $N_g \times E$ (Theorem 1.1) and then lift these results using the centralizer of the semisimple part.
• Canonical Representatives: The authors construct specific subsets $N_G \subset N_g \times E$ and $X_G \subset g \times E$ containing "canonical" representatives for each closed orbit. These representatives are defined using block diagonal matrices involving Jordan blocks ($J_k$) and specific vector configurations.
• Descendant Analysis: For a closed orbit $\mathcal{O}$ and a point $x \in \mathcal{O}$, the "descendant" is defined as the action of the stabilizer $\check{G}_x$ on the normal space $N^{\mathcal{O}}_{g \times E, x}$. The paper computes these explicitly by decomposing the tangent space and analyzing the action of the involution in $\check{G}$.

3. Key Contributions and Results

A. Classification of Closed Orbits (Theorems 1.1 & 1.2)

The paper provides a complete classification of closed orbits in $g \times E$:

• Nilpotent Case ($N_g \times E$): Theorem 1.1 establishes that every closed $G$-orbit in the nilpotent cone $\times E$ has a representative in a specific set $N_G$. These representatives are parameterized by an integer $k$ (related to the size of the Jordan block) and specific non-zero scalar parameters.
• General Case ($g \times E$): Theorem 1.2 extends this to the full space. Every closed orbit is represented by an element in $X_G$, which consists of block-diagonal matrices where the blocks are of the form $c_i I + N_i$ (semisimple part plus nilpotent part). The parameters must satisfy distinctness conditions ($c_i \neq c_j$ and $c_i \neq -c_j$ for orthogonal/symplectic cases) to ensure the orbit is closed.

B. Stability under MVW-Extension (Theorem 1.3)

A significant result is Theorem 1.3, which proves that every closed $G$-orbit in $g \times E$ is $\check{G}$-stable.

• This is non-trivial because the involution in $\check{G}$ (e.g., $g \mapsto g^{-t}$) does not obviously preserve all orbits.
• The proof relies on showing that the algebra of invariants for $G$ and $\check{G}$ coincide ($F[g \times E]^G = F[g \times E]^{\check{G}}$), implying that the categorical quotients are identical, and thus the closed orbits coincide.

C. Determination of Descendants (Theorem 1.6)

The paper explicitly determines the descendants for every closed orbit. For a closed orbit $\mathcal{O}$ and $x \in \mathcal{O}$:

• Stabilizer: The stabilizer $\check{G}_x$ is isomorphic to the MVW-extension of a product of smaller classical groups (e.g., $GL_k \times GL_{n-k}$ or $Sp_{2l} \times GL_k$).
• Normal Space Action: The normal space $N^{\mathcal{O}}_{g \times E, x}$ decomposes as a direct sum:
  $$N^{\mathcal{O}}_{g \times E, x} \cong h_{en} \oplus \text{triv}^{\oplus m}$$
  where $h_{en}$ is the enhanced standard representation of the stabilizer group, and $\text{triv}$ denotes the trivial representation. The multiplicity $m$ depends on the specific group type ($GL, O, Sp$) and the parameters of the orbit (specifically the number of zero eigenvalues or the "rank" $k$ of the nilpotent part).

D. Explicit Formulas

The paper provides explicit matrix forms for the representatives (Equations 1.2–1.12) and the stabilizer isomorphisms. For example, for $GL_n$, if the orbit corresponds to a partition with $k$ zero eigenvalues, the stabilizer is $\check{GL}_{n-k}$, and the normal space contains $2k$ copies of the trivial representation.

4. Significance and Applications

The results of this paper are foundational for several advanced topics in representation theory and automorphic forms:

1. Proof of Multiplicity One Theorems: The classification of closed orbits and the determination of descendants are crucial steps in the Gelfand-Kazhdan method. Theorem 1.6 provides a direct proof for specific propositions in previous works (e.g., [AG], [AGRS], [SZ2]), simplifying the verification of multiplicity one properties for classical groups.
2. Jacquet-Rallis Relative Trace Formula: The "smooth transfer" of orbital integrals relies on matching orbits between different groups. The reduction of the general linear side to the enhanced standard representation (as mentioned in the introduction) and the subsequent use of descendants for further reduction are now rigorously justified by these geometric classifications.
3. Geometric Understanding of Enhanced Cones: The paper generalizes previous work on "enhanced nilpotent cones" (by Achar-Henderson) and "exotic nilpotent cones" (by Kato) to the full Lie algebra setting and includes the action of the MVW-extension, providing a unified geometric framework for classical groups.
4. Computational Utility: By providing explicit representatives and stabilizer structures, the paper offers a concrete toolkit for researchers performing calculations involving orbital integrals, character sheaves, or local Langlands correspondence for these groups.

In summary, Chen Liang's paper provides a rigorous and complete geometric classification of the enhanced standard representations of classical groups, resolving the structure of their closed orbits and their behavior under the MVW-extension, thereby enabling significant progress in the proof of multiplicity one conjectures and the theory of smooth transfers.