A method of proving non-unitarity of representations of \(p\)-adic groups. I (Q611982)

For a connected reductive group \(G\) over a \(p\)-adic field \(F\), the Steinberg representation \(St\) and its Aubert dual, the trivial representation, are both unitary. Casselman proved that all other irreducible representations having the same infinitesimal character as \(St\) are non-unitary. In this paper, the authors give a partial generalization of Casselman's result. They consider generalized Steinberg representations (defined below) of \(G_\ell=Sp(2\ell,F)\) or \(SO(2\ell+1,F)\). Let \(\nu = |\det|_F\). Let \(\rho\) be an irreducible unitary cuspidal representation of \(GL(k,F)\) and \(\sigma\) an irreducible cuspidal representation of \(G_\ell\). Suppose that \(\text{Ind}^{G_{k+\ell}}(\nu^\alpha \rho \otimes \sigma)\) reduces for some \(\alpha \in (1/2)\mathbb{Z}_{>0}\). Then the representation \[ \Pi= \text{Ind}^{G_{(n+1)k+\ell}}(\nu^{\alpha + n} \rho \otimes \nu^{\alpha + n-1} \rho \otimes \cdots \otimes \nu^{\alpha } \rho \otimes \sigma) \] has the unique irreducible subrepresentation \(\pi\) and the unique irreducible quotient \(\pi'\). The representation \(\pi\) is square-integrable (and hence unitary); it is called a generalized Steinberg representation. The representation \(\pi'\) is the Aubert involution of \(\pi\). This paper and its sequel prove that all irreducible subquotients of \(\Pi\) different from \(\pi\) and \(\pi'\) are not unitarizable. The method of proof is based on the fact that the representation induced from a unitary representation is unitary, and therefore semi-simple.