Bessel operators on Jordan pairs and small representations of semisimple Lie groups (Q504640)

Let \( G \) be a connected, noncompact real simple Lie group with maximal parabolic subgroup \( P = MAN \subset G \) whose unipotent radical \( N \) is abelian. For \( \nu \in \mathfrak{1}{a}_{\mathbb C}^{\ast} \), consider the degenerate principal series \( \pi_{\nu} = \mathrm{Ind}_G^P (1_M \otimes e^{\nu} \otimes 1_N) \). These representations are relatively small and generically irreducible. However, at reducible points, they enjoy rich structure and attract interests in representation theory. As a consequence, there are many interesting results and extensive studies by many contributors which the readers can find in the list of references at the end of the paper. On the other hand there are still lots of things left to be investigated. The authors study small (unitary) irreducible quotients in those degenerate principal series representations by a mixture of algebraic (theory of Jordan pairs), geometric (nilpotent orbits), and analytic (Fourier transform and Bessel operators) methods. These apparently different methods are actually intimately related and produce rich ingredients to the study. There are also plenty of studies on this topic, especially for minimal representations. Let us explain the results obtained in this paper more explicitly. (However, the results are too complicated to state with the full accuracy, so we only describe them ``roughly'' and mainly explain the ideas and strategy. Please refer to the paper for correct and precise statements.) The degenerate principal series \( \pi_{\nu} \) is realized on the functions on \( \overline{N} \), which is the unipotent radical of the opposite parabolic to \( P \) (the non-compact picture). Since \( \overline{N} \) is abelian, it is isomorphic as a group to its Lie algebra \( \overline{\mathfrak{1}{n}} \), denoted by \( V^- \) in the paper. The pair \( (\mathfrak{1}{n}, \overline{\mathfrak{1}{n}}) = (V^+, V^-) \) has a natural structure of a Jordan pair, and there exists a fine structure theory for them (introduced in \S~1.2). For example, the authors describe the nilpotent \( L \) orbits in \( V^{\pm} \) in terms of the theory of Jordan pairs, which seems quite useful. In particular, they give explicit local coordinates of each \( L \)-orbits and identify their geometric structure as fiber bundles over compact flag manifolds with the fiber of reductive symmetric spaces (see \S~2.3). Moreover, they determine when the orbits admit an \( L \)-equivariant measure (\S~2.4). In \S~3, they introduce a so-called Bessel operator \( \mathcal{B}_{\lambda} \), which is a one-parameter family of second order differential operators on \( V^+ \) with the values in \( V^- \). It is interesting to know that \( \mathcal{B}_{\lambda} \) is tangential to a nilpotent \( L \)-orbit in \( V^+ \) for a finite number of specific parameters \( \nu_k \; (k = 1, 2, \dots r- 1) \), and it is symmetric on \( L^2(\mathcal{O}_k, d\mu_k) \) if the orbit \( \mathcal{O}_k \) has an \( L \)-equivariant measure \( d\mu_k \). After taking the Fourier transform (\S~4), the infinitesimal operators of the degenerate principal series can be expressed via multiplication of linear functions (for \( \overline{\mathfrak{1}{n}} \)), a directional derivative (for \( \mathfrak{1}{l} \)), and the Bessel operator (for \( \mathfrak{1}{n} \)). Thus, for the specific parameter \( \nu_k \) mentioned above, \( L^2(\mathcal{O}_k, d \mu_k) \) is stable under the action, and it is actually proved to be an irreducible unitary representation \( J(\nu_k) \) (Theorem 4.9). To prove the irreducibility, they use a special function called K-Bessel function as a spherical generating vector (Theorem 4.6). These unitary representations are small, and were already obtained (in various explicit cases) by Rossi and Vergne for Hermitian symmetric cases, and by Sahi and Dvorsky for non-Hermitian cases with \( P \) and \( \overline{P} \) conjugate. The smallest one is often being a minimal representation. For the rest of the cases (i.e., non-Hermitian cases with \( P \) and \( \overline{P} \) non-conjugate), these representations are newly obtained by the authors, but the emphasis is on the general and uniform construction in terms of the theory of Jordan pairs. In the picture of Fourier transform, the restriction of \( K \)-finite vectors of the degenerate principal series \( \pi_{\nu_k} \) to \( \mathcal{O}_k \) is an intertwiner (see [\textit{J. Möllers} and \textit{B. Schwarz}, J. Funct. Anal. 266, No. 6, 3508--3542 (2014; Zbl 1291.22017)]). Hence the above constructed small unitary representation \( J(\nu_k) \) yields an irreducible quotient of \( \pi_{\nu_k} \) (Theorem 4.13).