Harmonic analysis on a class of generalized Gel'fand pairs associated with hyperbolic spaces (Q1267415)

A pair \((G,H)\) of a unimodular Lie group \(G\) and a closed unimodular subgroup \(H\) is called generalized Gel'fand pair if for each irreducible unitary representation of \(G\) the space of \(H\)-fixed distribution vectors is at most one-dimensional. This paper is concerned with generalized Gel'fand pairs related to hyperbolic spaces and Heisenberg groups of the form \((MN,M)\) with \(M= U(1)\times U(k,l)\) and \(N\) the Heisenberg group. By direct computation and the use of differential equations satisfied by extremal positive definite distributions, the authors obtain explicit analytic expressions of spherical functions, the Plancherel formula and the extremal positive definite distributions. For regular parameters these distributions turn out to be locally integrable functions expressed by Laguerre polynomials; for non-regular parameters the distributions are supported by a cone.