Irreducibility of unitary group representations and reproducing kernels Hilbert spaces. Appendix on two point homogeneous compact ultrametric spaces in collaboration with Rostislav Grigorchuk. (Q1424727)

In this expository article, the authors give an introduction to Hilbert spaces with reproducing kernels and describe how they can be used in representation theory to prove irreducibility of representations. Recall that a reproducing kernel Hilbert space on a set \(X\) is a Hilbert space \({\mathcal H}\) of functions \(X\to {\mathbb C}\), such that the point evaluations \(\text{ev}_x : {\mathcal H} \to {\mathbb C}\) at elements \(x\in X\) are continuous linear functionals and hence of the form \(\langle\cdot , \phi_x\rangle\) for suitable elements \(\phi_x\in {\mathcal H}\). The map \(\Phi : X\times X\to {\mathbb C}\), \(\Phi(x,y):=\langle \phi_y,\phi_x\rangle\) is called the reproducing kernel of~\(\Phi\). Given a group action \(G\times X\to X\) on \(X\), it is well known that \((\pi(g).f)(x):=f(g^{-1}.x)\) for \(g\in G\), \(f\in {\mathcal H}\), \(x\in X\) defines a unitary representation~\(\pi\) of~\(G\) on~\({\mathcal H}\) if and only if the kernel \(\Phi\) is invariant in the sense that \(\Phi(g.x, g.y)=\Phi(x,y)\). The paper is centered around the following irreducibility criterion for representations associated with invariant kernels: If \(G\) acts transitively on~\(X\) and the subspace \({\mathcal H}^K\) of \(K\)-invariant vectors is one-dimensional for the isotropy subgroup \(K:=G_x\) of some point \(x\in X\), then \(\pi\) is irreducible (Proposition~2). Many examples are given to illustrate the usefulness of this criterion and its variants for representations defined using a cocycle (Proposition~6), resp., for representations on Hilbert spaces of vector-valued functions (Proposition~9). Further, more specialized examples are discussed in an appendix. Reviewer's remarks: An exposition of reproducing kernel spaces with a view towards representation theory has also been given in [\textit{K.-H. Neeb}, Holomorphy and convexity in Lie theory, de Gruyter Expositions in Math. 28 (Berlin, 1999; Zbl 0936.22001)], Part~A. In particular, one finds a version of Kobayashi's Theorem for Cocycle Representations there (Theorem~IV.1.10), which strengthens the authors' method in the special case of group actions on complex manifolds [cf. \textit{S. Kobayashi}, J. Math. Soc. Japan 20, 638--642 (1968; Zbl 0165.40504)].