Canonical representations related to hyperbolic spaces (Q1363024)

Canonical representations were introduced for the Lobachevsky plane by \textit{A. M. Vershik}, \textit{I. M. Gel'fand} and \textit{M. I. Graev} [Lond. Math. Soc. Lecture Note Ser. 69, 15-110 (1982); translation from Usp. Mat. Nauk 28, 82-128 (1973; Zbl 0288.22005)]. There this notion has been used for the first time. In a general setting -- for Hermitian symmetric spaces -- canonical representations were studied by \textit{F. A. Berezin} [Math. USSR, Izv. 9, 341-379 (1975); translation from Izv. Akad. Nauk SSSR, Ser. Mat. 39, 363-402 (1975; Zbl 0312.53050); Soviet Math., Dokl. 19, 786-789 (1978); translation from Dokl. Akad. Nauk SSSR 241, 15-17 (1978; Zbl 0439.47038)] for purposes of quantization. Berezin established the correspondence principle, found the spectral decomposition of a transform, called now the Berezin transform, and indicated the relation of canonical representations with tensor products. In the paper under review, the authors consider hyperbolic spaces \(G/K\), where \(G=U(1,n;{\mathbf F})\), \({\mathbf F}={\mathbf R},{\mathbf C},{\mathbf H}\), and \(K\) a maximal compact subgroup. These spaces are Riemannian symmetric spaces of rank one. For \({\mathbf F}={\mathbf C}\) they are Hermitian, the case where canonical representations were defined, but for \({\mathbf F}={\mathbf R},{\mathbf H}\) they do not carry some additional structure (complex or symplectic). All hyperbolic spaces can be realized as unit balls in \({\mathbf F}^n\) with the fractional linear action of \(G\). Using this realization, the authors define canonical representations for any \({\mathbf F}\) by the same formulae as for \({\mathbf F}={\mathbf C}\). These representations are unitary representations and are generated by a positive definite function \(\psi_\lambda(x)= (1-|x|)^\lambda\), \(\lambda\) being a parameter. -- The main results of the paper are: (1) The decomposition of the canonical representations into irreducible unitary representations. It turns out that for \(\lambda\) ``large'' the decomposition contains the principal series representations and for \(\lambda\) ``small'' a finite number of the complementary series representations is added. (2) Explicit expressions for the spherical transform of \(\psi_\lambda\). Moreover, it gives an explicit expression of the Berezin kernel in terms of the Laplacian. (3) The asymptotic behaviour of the Berezin kernel when \(\psi_\lambda\to\infty\) (three terms). Usually, one is content with two terms, which are needed for the correspondence principle. Reviewer's note: The mapping \(\Phi_k\) indicated in Part (ii) of Theorem 2 in Section 11 (concerning the embedding of complementary series representations) is not \(G\)-equivariant for \(k>0\).