Relative Property (T) for nilpotent subgroups (Q1753328)

A pair \((G,H)\) consisting of a locally compact topological group \(G\) and a closed subgroup \(H\) is said to have relative property (T) if every unitary representation of \(G\) having almost-\(G\)-invariant vectors has actual \(H\)-invariant vectors. The case where \(H=G\) is the much studied property (T) of Kazhdan. The ``relative'' version of the property has found prominent applications in analysis and operator algebra theory. The case where the subgroup \(H\) is abelian and normal in \(G\) has been a focal point of past research. In this paper the authors study a natural generalisation of this case, in which the subgroup \(H\) is assumed to be nilpotent and normal in \(G\). The nilpotent case is related to the abelian case by one of the main results of the paper (Theorem 1.2), which asserts that if \(N\) is nilpotent and normal in \(G\) then the pair \((G,N)\) has relative property (T) if and only if the pair \((G/\overline{[N,N]}, N/\overline{[N,N]})\) has relative property (T). (Here \(\overline{[N,N]}\) denotes the closure of the commutator subgroup of \(N\).) New results about relative property (T) for nilpotent normal subgroups can, by virtue of the above theorem, be deduced from existing work on the abelian case. As an example, the authors obtain a characterisation of relative property (T) for pairs \((H\ltimes N, N)\), where \(N\) is a simply connected nilpotent Lie group. The paper contains, in addition to the theorems cited above, several results pertaining relative property (T) and variants thereof (relative property (T) with approximation; relative property (T) for triples; and relative property (T) for subsets). While the focus is on nilpotent subgroups, a number of interesting results are obtained for more general classes of groups (see particularly Theorem 1.5 and Section 9).