Fourier-Stieltjes transforms over homogeneous spaces of compact groups (Q2319841)

Let \(G\) be a compact group, and let \(H\) be a closed subgroup of \(G\). Denote by \(\mu\) the normalized \(G\)-invariant measure on the homogeneous space \(G/H\) of left cosets of \(H\). Denote by \(\widehat{G}\) the set of all equivalence classes of irreducible unitary representations of \(G\); when \(G\) is abelian, \(\widehat{G}\) with the discrete topology is the dual group of \(G\). Set \[ H^\perp:=\{[\pi]\in \widehat{G}\colon \pi(t)=I\quad(t\in H)\} \] which, when \(H\) is normal, is isomorphic to the dual space \(\widehat{(G/H)}\) and, when \(G\) is abelian, is isomorphic to the dual group \(\widehat{(G/H)}\) of \(G/H\). However, when \(H\) is not normal, the dual space of \(G/H\) is defined as \[ \widehat{(G/H)}:= \Bigl\{[\pi]\in \widehat{G}\colon T^\pi_H:=\int_H\pi(t) \mathrm{d } t\neq 0\Bigr\}\,. \] It is always the case that \(H^\perp\subseteq \widehat{(G/H)}\), but equality occurs if and only if \(H\) is normal. In this paper, the author introduces the notion of a Fourier-Stieltjes transform of a measure on the homogeneous space \(G/H\) as follows (this occurs in the last section of the paper, after two introductory Sections 1 and 2, and three literature review Sections 3--5, where the author recalls his previous results). For each \(\lambda\in M(G/H)\) and \([\pi]\in\widehat{(G/H)}\), the author defines the Fourier-Stieltjes transform of \(\lambda\) at \([\pi]\) to be the operator \[ \mathcal{F}_{G/H}(\varphi)([\pi])=\hat{\varphi}([\pi])=\int_{G/H}\Gamma_\pi(tH)^* \,\mathrm{d }\lambda(tH)\in \mathcal{B}(\mathcal{H}_\pi) \] where \(\Gamma_\pi(tH)\) denotes the operator \(\pi(t)T^\pi_H\). This generalises the classical Fourier-Stieltjes transform on \(G/H\) when \(H\) is normal in \(G\) as well as a notion of Fourier transform on \(G/H\) introduced by the author in an earlier paper. Various basic properties of the new Fourier-Stieltjes transform are then proved, mirroring the properties of the classical notion.