Fractional calculus and analytic continuation of the complex Fourier-Jacobi transform (Q1890470)

The paper deals with the one-dimensional integral transform involving a product of the Jacobi function of the second kind and powers of the hyperbolic sine and cosine in the kernel. Such a transform, with the exactness to a constant multiplier, coincides with the integral transform considered by \textit{M. Flensted-Jensen} [Ark. Mat. 10, 143-162 (1972; Zbl 0233.42012)] and \textit{T. H. Koornwiner} [Ark. Mat. 13, 145-159 (1975; Zbl 0303.42022) and Special functions: Group theoretical aspects and applications, Math. Appl., D. Reider Publ. Co. 18, 1-85 (1984; Zbl 0584.43010)]. The authors prove the representation of the considered transform in terms of a fractional integration of a function with respect to the hyperbolic cosine function in a real half-axis; see Section 18.2 in the book by \textit{S.G. Samko, A. A. Kilbas} and \textit{O. I. Marichev} [``Fractional integrals and derivatives: theory and applications'' (Russian) (1987; Zbl 0617.26004; Engl. translation 1993; Zbl 0818.26003)]. On the basis of this representation the Parseval formula and the inversion formula for the initial transform are proved and the property of analytic continuation is considered.