Boundary values of holomorphic functions, singular unitary representations of \(O(p,q)\), and their limits as \(q\to\infty\) (Q1375167)

The voluminous paper which is rich in ideas and examples considers problems in noncommutative harmonic analysis and representation theory connected with the restriction of functions to submanifolds. Problems of this kind arise in a quite natural way in the standard constructions of representation theory: the restriction of a representation to a subgroup, the decomposition of the tensor product of representations. Slightly more detailed: Let a Lie group \(\widetilde G\) act on a manifold \(\widetilde M\), let \(\widetilde U\) be a representation of \(\widetilde G\) on a Hilbert space \(\widetilde H\) of functions on \(\widetilde M\) which is equipped with a nonlocal inner product (this means that the kernel \(K(x, y)\) of this inner product is not the delta function). Let \(G\) be a subgroup of \(\widetilde G\), and \(M\) a \(G\)-invariant submanifold of \(\widetilde M\). One has to investigate the restriction of \(\widetilde U\) to \(G\) and, in particular, to study the representation of \(G\) arising when the elements of \(\widetilde H\) are restricted to \(M\). This construction gives a possibility to show some `complementary series'' parts of the discrete spectrum of some natural representations. Moreover, making a generalization, we obtain a very intriguing problem concerning the action of \(G\) on distributions concentrated at \(M\) and, in duality, on transversal jets. Let, for example, \(\widetilde M\) be a bounded circular domain \(\Omega\) in \(\mathbb{C}^N\), let \(M\) be a submanifold of the boundary \(\partial\Omega\) of \(\Omega\), let \(\mu\) be a measure on \(\partial\Omega\) supported by \(M\), let \(H_K(\Omega)\) be the Hilbert space of holomorphic functions on \(\Omega\) generated by the reproducing kernel \(K(z,\overline u)\). Then, under some assumptions, one can determine: (a) the operator \(J: H_K(\Omega)\to L^1(M, \mu)\) which restricts the functions to \(M\) and the operator \(I: L^\infty(M, \mu)\to H_K(\Omega)\) (the Poisson transform with the kernel \(K(z,\overline u)\)); (b) corresponding Hilbert spaces \(E(\mu)\) and \(E^0(\mu)\) on \(M\). The operators \(I\) and \(J\) are joint to each other and the spaces \(E(\mu)\) and \(E^0(\mu)\) are dual to each other and isomorphic (a unitary equivalence is given by \(JI\)). This construction can be extended in a natural way to vector-valued functions. Now, let \(\rho\) be a multiplicator representation of the group \(\widetilde G\) on \(H_K(\Omega)\), let \(M\) be compact and invariant with respect to \(G\subset\widetilde G\), let \(\mu\) be a Lebesgue measure on \(M\). Then one obtains unitary representations \(\nu\) and \(\nu^0\) of \(G\) on \(E(\mu)\) and \(E^0(\mu)\), respectively. They are dual to each other. Several variants and versions of these constructions are also discussed. In the paper, mainly, the case is considered when \(\Omega\) is a matrix ball \(B_{pq}\subset \mathbb{C}^{pq}\) (the Cartan domain of type I), \(\widetilde G\) is the group \(\text{U}(p,q)\) (or its universal covering), \(\rho\) is a highest weight representation, \(M\) is the real Stiefel manifold \(M_{pq}= (\text{SO}(p+q)/\text{SO}(q))\) (it is the `real part'' of the Shilov boundary of \(\Omega\), and \(G= \text{SO}_0(p,q)\)). The authors call arising representations of \(\text{SO}_0(p,q)\) on \(E(\mu)\) and \(E^0(\mu)\) singular (special) representations. These representations enter discretely in the restriction of \(\rho\) to \(\text{SO}_0(p,q)\). A characterization of them is given as induced representations. In particular, for \(p= 1\) a description of Hilbert spaces where spherical complementary series representations act is given. These spaces are Sobolev spaces. A characterization of distributions belonging to these spaces is given, etc. Another property of the singular representations is that they persist in the limit as \(q\to\infty\). In connection with all this, in the end of the paper (in accordance with its philosophy), the authors study some problems of harmonic analysis for complementary series of \(\text{SO}_0(1,q)\), namely, the restriction of a spherical complementary series representation to the subgroup \(\text{SO}_0(1, q-1)\) and the tensor product of two such representations. For example, the restriction mentioned above can include discretely spherical complementary series representations of \(\text{SO}_0(1,q- 1)\). They can be realized in a natural way on irreducible subquotients of a space of distributions concentrated at an equator \(S^{q-2}\) of the sphere \(S^{q-1}\) or, in duality, on irreducible subquotients of a filtration of transversal jets.