Homogeneous spaces and square-integrable representations (Q747760)

This paper is devoted to study a square-integrability criterion for unitary representations of homogeneous spaces associated to compact subgroups of locally compact groups. Let \(G\) be a locally compact group and \(H\) be a compact subgroup of \(G\). The authors introduced the notion of ``continuous unitary representations'' over the left coset space \(G/H=\{xH:x\in G\}\) in [the first author et al., Int. J. Wavelets Multiresolut. Inf. Process. 10, No. 4, 1250038, 18 p. (2012; Zbl 1254.43003)]. The definition which they have presented is as follows: A continuous unitary representation of the homogeneous space \(G/H\) is a map \(\sigma\) from the left coset space \(G/H\) into the group \(\mathcal U(\mathcal H_\sigma)\), of all unitary operators on some nonzero Hilbert space \(\mathcal H_\sigma\) which satisfies the following conditions: (i) The map \(xH\mapsto\langle\sigma(xH)\zeta,\xi\rangle\) is continuous from \(G/H\) to \(\mathbb C\); (ii) \(\sigma(xyH)=\sigma(xH)\sigma(yH)\) and \(\sigma(x^{-1}H)=\sigma(xH)^\ast\), for all \(x,y\in G\) and \(\zeta,\xi\in\mathcal H_\sigma\). The reader should be warned that the authors' attitude concerning differences of ``homogeneous spaces'' and ``quotient groups'' toward rigor and precision is exceedingly carefree. As will be discussed in the following, this notion and hence the extended theory based on this approach is useless for homogeneous spaces and the results of the papers are interesting for quotient groups. In the following, we discuss the two most important aspects of the above notion of unitary representations over homogeneous spaces. I. From the categorical aspect, the main aim of the theory of representations in a category is to study and verify how much a given object of the category is similar to the concrete and building block objects of the category or simply important examples for objects of the category. For instance, in the category of groups, important examples of this category is the group that consists of unitary operators on some vector space. This leads to the classical definition of representations for groups, which associates any given group to the groups of unitary operators. In the category of \(C^\ast\)-algebras, important examples of this category are the \(C^\ast\)-algebra as that consist of bounded operators on some Hilbert space. Similarly, this leads to the classical definition of representations for \(C^\ast\)-algebras, which associates to any given \(C^\ast\)-algebra the \(C^\ast\)-algebra of bounded operators on some Hilbert space. Thus, associating any given homogeneous space, which is not a locally compact group in general, with a \textbf{group} of \textbf{unitary} operators on some Hilbert space is really strange. II. From the separation property (Gelfand-Raikov theorem) aspect, it can readily be proved that continuous unitary irreducible representations of the homogeneous space \(G/H\) (due to the above definition) separate points of \(G/H\) if and only if \(H\) is normal in \(G\). Hence, as is the case in the general theory of harmonic analysis over homogeneous spaces, this theory will not discover or add any significant content to the harmonic analysis over homogeneous spaces. In detail, for most of the homogeneous spaces of the form \(G/H\) with \(H\) compact and non-normal in \(G\), the set \(H^\perp\) given by \(\{[\pi]\in\widehat{G}:\pi(h)=I,\;\forall h\in H\}\), is really small and maybe even a singleton set. In a nutshell, the authors define new objects (which are called representations) on \(G/H\) such that the existence of sufficiently many important ones (i.e., irreducible representations) is guaranteed only when the homogeneous space is a quotient group, that is the trivial case in the framework of homogeneous spaces. Hence, we deduce that when \(H\) is not normal in \(G\), as is the case in general examples of homogeneous spaces, these defined objects are not interesting since they will not reproduce any information concerning the structure of \(G/H\).