Generalized matrix coefficients for infinite dimensional unitary representations (Q2927744)

Let \(G\) be a Lie group and let \((\pi, \mathcal H)\) be a unitary representation of \(G\). Let \(\mathcal H^\infty\) be the space of smooth vectors with the natural topology, and let \((\mathcal H^*)^{-\infty}\) be the topologically dual of \(\mathcal H^\infty\) equipped with the contragredient actions \(\pi^c\) of \(G\) and \({\mathrm {Lie}}\, G\) and the weak star topology. Let \(\mathcal H^*\) be the dual of \(\mathcal H\) and let \(\mathcal H^{-\infty}\) be the topological dual of \((\mathcal H^*)^\infty\). The author generalizes the definition of matrix coefficients to the pairs \((\alpha, \beta)\), where \(\alpha\in \mathcal H^{-\infty}\), \(\beta\in (\mathcal H^*)^{-\infty}\). Generalized matrix coefficients are in the space \({\mathbf D}'(G)\) of distributions on \(G\). It is proved that, fixing \(\beta\), the map \(\mathcal H^{-\infty}\to {\mathbf D}'(G)\) is continuous. If \(G\) is the Heisenberg group, generalized matrix coefficients can be considered as a generalization of the Fourier--Wigner transform.