On the Faraut-Koranyi hypergeometric functions in rank two. (Q1774084)

Let \(\Omega\) be an irreducible bounded symmetric domain of rank \(r\) and characteristic multiplicity \(a\) in \(\mathbb C^d\). The Faraut-Koranyi hypergeometric function on \(\Omega\) \textit{J. Faraut} and \textit{A. Koranyi} [J. Funct. Anal. 88, 64--89 (1990; Zbl 0718.32026)] is the series \[ _2F_1(\alpha,\beta;\gamma| x) = \sum_{\mathbf m} \frac{(\alpha)_{\mathbf m} (\beta)_{\mathbf m}}{(\gamma)_{\mathbf m}} K_{\mathbf m}(x,\overline x), \] where the sum is taken over the partitions of length \(r\) and \(K_{\mathbf m}(x, \overline x)\) is a polynomial in the singular values \(t_1, \ldots t_r\) of \(x\) which coincides, up to a multiplicative constant, with the Jack polynomial \(J_{\mathbf m}^{(2/a)}(t_1^2, \ldots, t_r^2).\) In the present paper, the authors study the boundary behaviour of this hypergeomeric function in the case where the rank of \(\Omega\) is two. This generalizes results of \textit{Z. Yan} [Can. J. Math. 44, 1317--1338 (1992; Zbl 0769.33014)]. The proof is based on an explicit integral representation for the Jack polynomials in rank \(2\) (where they are actually given in terms of Gegenbauer polynomials).