On the cortex of connected simply connected nilpotent Lie groups (Q1267424)

Let \(G\) be a locally compact group and \(\widehat G\) its dual space. In general, \(\widehat G\) is far from being a Hausdorff space. In fact, when \(G\) is a connected group, then \(\widehat G\) being Hausdorff forces \(G\) to have a relatively compact commutator subgroup. It is therefore of interest to study the set of (equivalence classes of) irreducible representations of \(G\) that cannot be separated by open sets from the trivial 1-dimensional representation of \(G\). This closed subset of \(\widehat G\) is called the cortex, \(\text{cor} (G)\), of \(G\) and has been investigated by several authors. The paper under review contributes to the understanding of \(\text{cor} (G)\) for \(G\) a connected and simply connected nilpotent Lie group. Let \({\mathfrak g}\) denote the Lie algebra of \(G, {\mathfrak g}^*\) the vector space dual of \({\mathfrak g}\) and \(\text{ad}^*\) the coadjoint representation of \({\mathfrak g}\) on \({\mathfrak g}^*\). Let \(f\to \pi_f\) denote the Kirillov map from \({\mathfrak g}^*\) onto \(\widehat G\), and put \(\text{cor} ({\mathfrak g}^*) =\{f\in {\mathfrak g}^*: \pi_f\in \text{cor} (G)\}\). Moreover with \(\gamma \) the conjugation representation of \(G\) on \(L^2(G)\), let \(\text{supp} (\gamma)\) denote the set of \(f\in {\mathfrak g}^*\) such that \(\pi_f\) is weakly contained in \(\gamma\). Now suppose that all generic Kirillov orbits of \({\mathfrak g}^*\) are linear varieties. Extending results of Bekka and Kaniuth for 2-step nilpotent groups, the author shows that \(\text{cor} ({\mathfrak g}^*)= \overline {\{\text{ad}^* (X)l: l\in{\mathfrak g}^*, X\in {\mathfrak g}\}} =\text{supp} (\gamma)\). In addition, if \(\text{cor} ({\mathfrak g}^*)\) is a linear subspace of \({\mathfrak g}^*\), then \(\text{cor} ({\mathfrak g}^*)\) coincides with the annihilator of the centre of \({\mathfrak g}\) in \({\mathfrak g}^*\). The second emphasis of the paper is on 3-step nilpotent groups with 1-dimensional centre. For such \(G\), a complete description of \(\text{cor} (G)\) seems to be difficult to achieve. However, the author provides several partial results and presents various illustrating examples.