Degenerate principal series representations for \(U(n,n)\) (Q1359999)

Degenerate principal series representations of split classical groups over local fields which are induced from a character of the maximal parabolic subgroup with abelian unipotent radical play a central role in Howe correspondences. To understand this role it is essential to know points of reducibility, complete composition series, constituents arising from local theta-lifts, the behavior of the constituents under normalized intertwining operators and certain other information. In this paper the authors provide all this information for \(\text{U}(N,N)\) and \(\text{GL}_{2N}\) over a non-Archimedean local field. The methods used are similar to those used by the first author and \textit{S. Rallis} to study the case of \(\text{Sp}_{2N}\) [Isr. J. Math. 78, 209-256 (1992; Zbl 0787.22019)]. The results obtained have been applied to the study of local theta correspondences for unitary groups by the authors and \textit{M. Harris} [J. Am. Math. Soc. 9, 941-1004 (1996; Zbl 0870.11026)]. They are also of relevance for the extension of the Siegel-Weil formula to unitary groups. Questions of the reducibility of induced representations for classical groups have been studied by numerous authors; see, for instance, \textit{F. Shahidi} [Duke Math. J. 66, 1-41 (1992; Zbl 0785.22022), Invent. Math. 119, 1-36 (1995; Zbl 0852.22016)], \textit{D. Goldberg} [J. Reine Angew. Math. 448, 65-95 (1994; Zbl 0815.11029), Am. J. Math. 116, 1101-1151 (1994; Zbl 0851.22021)], \textit{Ch. Jantzen} [Compos. Math. 104, No. 1, 55-63 (1996; Zbl 0866.22017), Mem. Am. Math. Soc. 590, 100 p. (1996; Zbl 0866.22016)], \textit{M. Tadić} [Pitman Res. Notes Math. Ser. 311, 129-204 (1994; Zbl 0859.22011), J. Algebra 177, 1-33 (1995; Zbl 0874.22014)]. We will now summarize the main results of the paper under review. Let \(F\) be a non-Archimedean local field, and let \(E/F\) be a quadratic extension. Let \(G\) be the isometry group of the skew Hermitian form defined on the space of row vectors \(W=E^{2N}\) via \(\langle w,w'\rangle=wJ {}^t\overline{w}'\), with \(J=\left(\begin{smallmatrix} 0&1_N\\ -1_N&0\end{smallmatrix}\right)\). Thus \(g\in\text{GL}_{2N}(E)\) belongs to \(G\) if and only if \(gJ {}^t\overline{g}=J\). The group \(G\) has a maximal parabolic subgroup \(P=LU\) given by \[ L=\left\{l(a)=\left(\begin{matrix} a&0\\ 0&{}^t\overline{a}^{-1}\end{matrix}\right):a\in\text{GL}_{N}(E)\right\}, \quad U:=\left\{u(b)=\left(\begin{matrix} 1_N&b\\ 0&1_N\end{matrix}\right):b={}^t\overline{b}\in\text{Mat}_{N}(E)\right\}. \] For a given unitary character \(\chi\) of \(E^\times\) and for \(s\in{\mathbb C}\), let \(I(s,\chi)\) be the induced representation of \(G\) realized by the action of \(G\) by right translation on the space of smooth functions \(\Phi\colon G\rightarrow{\mathbb C}\) such that \(\Phi(u(b)l(a)g)=\chi(\text{det}(a)) |\text{det}(a)|_E^{s+{N\over 2}}\Phi(g)\) for all \(g\in G\), \(l(a)\in L\), and \(u(b)\in N\). (So that \(\Re(s)=0\) is the unitary axis.) In order to describe the quasi-character \(\chi |\;|^s_E\) by the pair \((s,\chi)\), it is convenient to normalize \(s\) and \(\chi\) to make this correspondence unique. The authors choose \(s\) and \(\chi\) so that they satisfy one or both of the properties (a), (b) below: (a) \(\Im(s)\in[0, {\pi\over\log(q_E)})\); (b) \(\chi(\varpi_E\cdot\overline\varpi_E)=1\), where \(x\mapsto\overline x\) denotes the action of the non-trivial Galois automorphism of \(E/F\), and \(\varpi_E\) denotes a generator (chosen once and for all) for the maximal ideal \({\mathfrak P}_E\) of the ring of integers \({\mathfrak O}_E\) of \(E\). (Here \(|\;|_E\) is normalized via \(|\varpi_E|_E=q^{-1}_E\), with \(q_E\) equal to the order of the finite residual field \({\mathfrak O}_E/{\mathfrak P}_E\).) Let \(\check\chi\) be the character defined by \(\check\chi(x):=\chi(\overline x)^{-1}\). We have \(\chi=\check\chi\) if and only if either \(\chi_{| F^\times}=1\) or \(\chi_{| F^\times}=\varepsilon_{E/F}\), where \(\varepsilon_{E/F}\) denotes the unique non-trivial quadratic character of \(F^\times\) with kernel equal to \(\text{N}^E_F(E^\times)\). The authors obtain the following results: \(\bullet\) If \(\chi\neq\check\chi\), then \(I(s,\chi)\) is irreducible for all \(s\); \(\bullet\) if \(\chi_{| F^\times}=1\), then \(I(s,\chi)\) is irreducible except when \(s\in\{-{N\over 2}, 1-{N\over 2}, 2-{N\over 2},\ldots,{N\over 2}\}\); \(\bullet\) if \(\chi_{| F^\times}=\varepsilon_{E/F}\), then \(I(s,\chi)\) is irreducible except when \(s\in\{-{N-1\over 2}, 1-{N-1\over 2}, 2-{N-1\over 2},\ldots, {N-1\over 2}\}\). (Note that, in each case, the points of reducibility consist either of integral or half-integral points, and that \(I(0,\chi)\) is reducible if and only if \(\chi_{| F^\times}=1\) and \(N\) is even, or \(\chi_{| F^\times}=\varepsilon_{E/F}\) and \(N\) is odd.) At each of the points of reducibility described above, the constituents of \(I(s,\chi)\) arise from the images of certain representations of \(G\) associated to an Hermitian form via the representation defined by \textit{André Weil} in [Acta Math. 111, 143-211 (1964; Zbl 0203.03305)]. If \(V\) is a non-degenerate Hermitian vector space with \(\dim_E(V)=N'\geq 1\), then we can view \(U(V)\times G\) as a dual reductive pair in the larger symplectic group \(\text{Sp}_{4NN'}(F)\). If \(\chi_{| F^\times}=\varepsilon_{E/F}^{N'}\), then \(\chi\) can be used to construct a splitting of the inverse image of \(G\) sitting in the metaplectic cover of \(\text{Sp}_{4NN'}(F)\); see \textit{St. Kudla} [Isr. J. Math. 87, 361-401 (1994; Zbl 0879.11026)]. The Weil representation then leads to a representation \(\omega\) of \(G\) in the Schwartz-Bruhat space \({\mathcal S}(V^N)\), and in fact, for \(f\in{\mathcal S}(V^N)\), the function \(\omega(g)f(0)\) lies in the space \(I(s_0,\chi)\) for \(s_0={N'-N\over 2}\). Let \(R(V,\chi)\) denote the image of \({\mathcal S}(V^N)\) in \(I(s_0,\chi)\). If \(\chi_{| F^\times}=1\), then \(\chi\) (which acts as a character of the parabolic subgroup \(P=LU\) of \(G\)) extends uniquely to a character \(\chi_G\) of \(G\). In this case, let \(R(0,\chi)\subset I(-{N\over 2},\chi)\) be the \(\mathbb C\)-linear span of \(\chi_G\). The authors show that these spaces form the constituent of \(I(s_0,\chi)\) as follows: Suppose that \(\chi_{| F^\times}=\varepsilon_{E/F}^{N'}\) and \(s_0={N'-N\over 2}\) for some \(N'\geq 0\). This accounts for all points of reducibility of \(I(s_0,\chi)\). Whenever \(N'\geq 1\), let \(V_1\) and \(V_2\) be the two inequivalent non-degenerate Hermitian vector spaces over \(E\) of dimension \(N'\). These are distinguished by \(\text{det}(Q_i)\in F^\times/\text{N}^E_F(E^\times)\), where, for \(i\in\{1,2\}\), \(Q_i\) is some choice of Hermitian matrix realizing the form on \(V_i\). (1) If \(N'=0\), then \(R(0,\chi)\) is the unique irreducible \(G\)-submodule contained in \(I(-{N\over 2},\chi)\), and the quotient \(I(-{N\over 2},\chi)/R(0,\chi)\) is also irreducible. (2) If \(1\leq N'\leq N\), so that \(-{N\over 2}<s_0\leq 0\), then \(R(V_1,\chi)\) and \(R(V_2,\chi)\) are irreducible and inequivalent, \(R(V_1,\chi)\oplus R(V_2,\chi)\) is a submodule of \(I(s_0,\chi)\), and \(I(s_0,\chi)/(R(V_1,\chi)\oplus R(V_2,\chi))\) is irreducible; in particular, if \(N'=N\), then \(R(V_1,\chi)\oplus R(V_2,\chi)=I(0,\chi)\). (3) If \(N<N'<2N\), so that \(0<s_0<{N\over 2}\), then \(R(V_1,\chi)\) and \(R(V_2,\chi)\) are distinct maximal submodules of \(I(s_0,\chi)\), so that \(I(s_0,\chi)=R(V_1,\chi)+R(V_2,\chi)\); also \(R(V_1,\chi)\cap R(V_2,\chi)\) is the unique irreducible submodule of \(I(s_0,\chi)\). (4) If \(N'=2N\), then let \(V_1\) be the split Hermitian space of dimension \(N'\), and \(V_2\) be the other space of the same dimension, in this case \(R(V_2,\chi)\) is the unique maximal submodule of \(I({N\over 2},\chi)\), and \(R(V_2,\chi)\) is irreducible with codimension \(1\); in addition \(R(V_1,\chi)=I({N\over 2},\chi)\). (5) Finally, if \(N'>2N\), so that \(I(s_0,\chi)\) is irreducible, then the submodules \(R(V_i,\chi)\), \(i\in\{1,2\}\), are both equal to all of \(I(s_0,\chi)\). In view of future global applications, the case \(E=F\oplus F\), with Galois automorphism \(x=(x_1,x_2)\mapsto\overline x:=(x_2,x_1)\), is also considered in the paper under review. In this case \(G=\text{GL}_{2N}(F)\), with maximal parabolic subgroup \(P=LU\), where \[ L:=\left\{l(a,d)=\left(\begin{matrix} a&0\\ 0&d\end{matrix}\right):a,d\in\text{GL}_N(F)\right\},\quad U:=\left\{u(b)=\left(\begin{matrix} 1_N&b\\ 0&1_N\end{matrix}\right):b\in\text{Mat}_N(F)\right\}. \] Given two unitary characters \(\sigma_1\), \(\sigma_2\) of \(F^\times\), let \(\sigma:=(\sigma_1,\sigma_2)\), and write \((s,\sigma)\) for the quasi-character of \(L\) defined by \[ (s,\sigma)(l(a,d)):=\sigma_1(\text{det}(a))|\text{det}(a)|_F^s\;\sigma_2(\text{det}(d))|\text{det}(d)|_F^{-s}. \] (1) If \(\sigma_1=\sigma_2\) and \(s\in\left\{\pm{1\over 2},\pm 1,\pm {3\over 2},\ldots,\pm {N\over 2}\right\}\), then \(I(s,\chi)\) is reducible as a \(\text{GL}_{2N}(F)\)-module; (2) for all other pairs \((s,\sigma)\), \(I(s,\sigma)\) is irreducible; (3) suppose that \(\sigma=\sigma_1=\sigma_2\) and let \(s_0:={N'-N\over 2}\) for \(0\leq N'\leq 2N\), \(N'\neq N\), so that \((s_0,\sigma)\) is a point of reducibility for \(I(s,\sigma)\). Then \(I(s_0,\sigma)\) has a unique submodule \(A\), and the quotient \(I(s,\sigma)/A\) is also irreducible and is not isomorphic to \(A\); for \(1\leq k\leq N\), let \(P_{N+k,N-k}\) be the standard parabolic subgroup of \(G\) with Levi subgroup isomorphic to \(\text{GL}_{N+k}(F)\times\text{GL}_{N-k}(F)\): (a) If \(N<N'\leq 2N\), so that \(0<s_0\leq{N\over 2}\), then the irreducible quotient of \(I(s_0,\chi)\) is isomorphic to \(\text{Ind}_{P_{N',2N-N'}}(\sigma)\); (b) on the other side of the unitary axis, if \(0\leq N'<N\) (so that \(-{N\over 2}\leq s_0<0\)), then the irreducible subrepresentation of \(I(s_0,\sigma)\) is isomorphic to \(\text{Ind}_{P_{2N-N',N'}}(\sigma)\); the irreducible quotient at \(s_0>0\) is isomorphic to the irreducible submodule at \(-s_0\) (see (5) below), and both are unitarizable; (4) these constituents are related to the Weil representation as follows: for \(\sigma=\sigma_1=\sigma_2\) and \(s_0={N'-N\over 2}\) (with \(1\leq N'\)), there is a construction of a dual pair \(G':=\text{GL}_{N'}(F)\) and \(G=\text{GL}_{2N}(F)\) in \(\text{Sp}_{4NN'}(F)\); the appropriate Weil representation restricts to a representation of \(G'\times G\), and there is an injection of the \(G'\)-coinvariants into the space \(\text{Ind}_P^G(|\;|_F^{s_0}\otimes|\;|_F^{-s_0})\); let \(R_N(N',\sigma)\) denote the submodule of \(I(s_0,\sigma)\) obtained by twisting this subspace by \(\sigma\): (a) If \(N'\geq N\), then \(R_N(N',\sigma)=I(s_0,\sigma)\); (b) if \(1\leq N'<N\), then \(R_N(N',\sigma)\) is the unique irreducible (unitarizable) submodule of \(I(s_0,\chi)\); (c) if \(N'=0\), then let \(R_N(0,\sigma):={\mathbb C}\cdot\sigma\subset I(-{N\over 2},\sigma)\); this is the unique irreducible submodule in this case; (5) there is a standard \(G\)-intertwining operator \(M(s,\sigma)\colon I(s,\sigma)\rightarrow I(-s,\check\sigma)\), where \(\sigma:=(\sigma_1,\sigma_2)\) is again arbitrary, and \(\check\sigma:=(\sigma_2,\sigma_1)\), defined by \[ M(s,\sigma) f(g):=\int_{\text{Mat}_N(F)}f(w_Nu(b)g) db, \] where \(w_N:=\left(\begin{smallmatrix} 0&1_N\\ -1_N&0\end{smallmatrix}\right)\); this converges for \(\Re(s)>{N\over 2}\); this operator can be normalized by setting \(\tilde M(s,\sigma):=a(s,\chi_\sigma)^{-1} M(s,\sigma)\), where \(\chi_\sigma:=\sigma_1/\sigma_2\) is a character of \(F^\times\), and \[ a(s,\chi_\sigma):=\zeta_F(2s,\chi_\sigma) \zeta_F(2s-1,\chi_\sigma)\cdots\zeta_F(2s-(N-1),\chi_\sigma) \] is a product of Tate zeta functions \(\zeta_F(s,\chi_\sigma)\), with: \[ \zeta_F(s,\chi_\sigma):=\begin{cases}{\displaystyle 1\over 1-q_F^{-s}\chi_\sigma(\varpi_F)}&\text{if \(\chi_\sigma\) is unramified}\\ 1&\text{if \(\chi_\sigma\) is ramified;}\end{cases} \] then for any \(\sigma\), \(\tilde M(s,\sigma)\) has an analytic continuation to the \(s\)-plane, and it is never identically the zero operator; when \((s,\sigma)\) is not a point of reducibility, \(\tilde M(s,\sigma)\) is an isomorphism; when \(s\) is a point of reducibility, \(\tilde M(s,\sigma)\) has the unique irreducible submodule of \(I(s,\sigma)\) as kernel, and maps the unique irreducible quotient of \(I(s,\sigma)\) to the irreducible submodule of \(I(-s,\sigma)\).