Some continuous analogs of the expansion in Jacobi polynomials and vector-valued orthogonal bases (Q883611)

Denote \[ R(s)=\left(\begin{matrix} \Gamma(\frac{1}{2}-\alpha-is)\Gamma(\frac{1}{2}-\alpha+is)&\Gamma(\frac{1}{2}-i\beta-is)\Gamma(\frac{1}{2}-i\beta+is)\\ \Gamma(\frac{1}{2}+i\beta-is)\Gamma(\frac{1}{2}+i\beta+is)&\Gamma(\frac{1}{2}+\alpha-is)\Gamma(\frac{1}{2}+\alpha+is)\end{matrix}\right). \] For the Hilbert space \(H_{\alpha,\beta},0\leq\alpha\leq 1/2,\beta\in\mathbb R,\alpha+i\beta\neq 0,\) with the inner product \[ \langle(\varphi_1,\varphi_2),(\psi_1,\psi_2)\rangle=\frac{1}{2\pi}\int_0^{\infty}(\varphi_1(s),\varphi_2(s))R(s)\left(\frac{\overline{\psi_1(s)}}{\psi_2(s)}\right)\frac{ds}{| \Gamma(2is)| ^2} \] the author finds an analogue of the index hypergeometric transform (it is called the double hypergeometric transform, \(f\to (\varphi_1,\varphi_2),\varphi_j(s)= \int_{-\infty}^{\infty}f(x)\overline{Q_j(\alpha,\beta;x,s)}dx)\) and proves that it is a unitary bijective operator from \(L^2(\mathbb R)\) to \(H_{\alpha,\beta}.\) The inversion formula is given by \[ f(x)=\langle(Q_1,Q_2),(\overline{\varphi_1},\overline{\varphi_2})\rangle. \] A generalization to \(\alpha>1/2\) is given (with the space \(H_{\alpha,\beta}\oplus W_{\alpha,\beta},\) where \(W_{\alpha,\beta}\) is the finite-dimensional linear space with the inner product \[ \langle c,c^{\prime}\rangle=\frac{1}{2\pi}\sum_{k=0}^n\frac{2\alpha-2k-1}{\Gamma(2\alpha-k)k!}c_k\overline{c_k}'). \] Also, an \({}_3F_2\)-orthogonal basis in a space of functions ranging in \(\mathbb C^2\) is constructed.The basis lies in the analytic continuation of continuous Hahn polynomials with respect to the index \(n\) of a polynomial. Proofs use the spectral decomposition of the hypergeometric differential operator on the contour \(\text{Re}\, z=1/2.\) Applications to tensor products of unitary representations of \(SL_2(\mathbb R)\) are discussed.