On the representations of Sp(p, q) and SO*(2n) unitarily induced from derived functor modules (Q4831313)

The decomposition formula of the representations of a real reductive Lie group unitarily induced from derived functor modules has been studied by various people. In this article H. Matumoto obtains a formula in the case of \(\text{Sp}(p,q)\) and \(\text{SO}^*(2n)\) as he did in his previous paper for \(\text{U}(m,n)\) [Compos. Math. 100, No. 1, 1-39 (1996; Zbl 0858.22013)]. Let \(G=\text{Sp}(p,q)\), \(p\geq q\) (resp. \(\text{SO}^*(2n)\)) and \(P_\kappa\), \(\kappa=(k_1,\dots,k_s)\), \(k_1+\cdots k_s\leq q\) (resp. \(k_s\leq n/2\)), be a parabolic subgroup of \(G\) whose Levi subgroup \(M_\kappa\) is of the form \[ M_\kappa=\text{GL}(k_1,{\mathbb H})\times\cdots\text{GL}(k_s,{\mathbb H})\times\text{Sp}(p',q')\;(\text{resp.}\;\text{SO}^*(2r)), \] where \(p'=p-k_1-\cdots k_s\) and \(q'=q-k_1-\cdots k_s\) (resp. \(r=n-2(k_1+\cdots+k_s)\)). Since any derived functor module of \(\text{GL}(k,{\mathbb H})\) is a unitary parabolic induction from one-dimensional representations \(A_k(-\infty,t)\) or quaternionic Speh representations \(A_k(\ell,t)\), \(t\in\sqrt{-1}{\mathbb R}\), \(\ell\geq-k\), our problem is reduced to study the following induced representation: \[ \text{Ind}_{P_\kappa}^G(A_{k_1}(\ell_1,t_1)\otimes\cdots\otimes A_{k_s}(\ell_s,t_s)\otimes Z), \] where \(Z\) is a derived functor module of \(\text{Sp}(p',q')\). We assume that \(\ell_i+1\in2{\mathbb Z}\) and \(t_i=0\) for some \(1\leq i\leq s\), and let \(i=s\) and \(\kappa'=(k_1,\dots,k_{s-1})\). Then induction by stages yields that this above representation is equivalent to \[ \text{Ind}_{P_{\kappa'}}^G(A_{k_1}(\ell_1,t_1)\otimes\cdots\otimes A_{k_{s-1}}(\ell_{s-1},t_{s-1})\otimes \text{Ind}_{P_{(k_s)}}^{M_{\kappa'}^\circ}(A_{k_s}(\ell_s,0)\otimes Z)). \] This means that the decomposition formula is reduced to \(\text{Ind}_{P_{(k_s)}}^{M_{\kappa'}^\circ}(A_{k_s}(\ell_s,0)\otimes Z)\). Applying change of polarization and reducibility results of the degenerate principal series (cf. the author's paper quoted above), we have an explicit decomposition formula of \(\text{Ind}_{P_{(k_s)}}^{M_{\kappa'}^\circ}(A_{k_s}(\ell_s,0)\otimes Z)\) into a direct sum of derived functor modules of \(M_{\kappa'}^\circ\). Therefore, whenever there is \(\ell_i+1\in2{\mathbb Z}\) and \(t_i=0\) for some \(1\leq i\leq s\), the decomposition problem is reduced to \[ \text{Ind}_{P_{\kappa}}^G(A_{k_1}(\ell_1,0)\otimes\cdots\otimes A_{k_h}(\ell_h,0)\otimes A_{k_{h+1}}(\ell_{h+1},t_{h+1})\otimes\cdots\otimes A_{k_s}(\ell_s,t_s)\otimes Z), \] where \(Z\) is an irreducible representation of \(M_{\kappa}^\circ\). Finally, considering the Hecke algebra module structure, we can reduce the irreducibility of the one above to the one of \[ \text{Ind}_{P_\tau}^{SO^*(4a)}(A_{k_1}(\ell_1,0)\otimes\cdots\otimes A_{k_h}(\ell_h,0)), \] where \(\tau=(k_1,\dots,k_h)\) and \(a=k_1+\cdots+k_h\). In particular, if \(\ell_1,\dots,\ell_h\) are all \(-\infty\), then \(\text{Ind}_{P_\tau}^{SO^*(4a)}(A_{k_1}(\ell_1,0)\otimes\cdots\otimes A_{k_h}(\ell_h,0))\) is irreducible, that is, representations of \(\text{Sp}(p,q)\) and \(\text{SO}^*(2n)\) induced from one-dimensional unitary representations of their parabolic subgroups are irreducible.