Branching laws for minimal holomorphic representations (Q2460017)

The author investigates one of the most important problems in harmonic analysis and in representation theory -- the decomposition of group representations into irreducible ones. When a given representation arises as the restriction of an irreducible representation of a bigger group, the decomposition is referred to as a branching law. The author studies the branching law for the restriction from \(SU(n,m)\) to \(SO(n,m)\) of the minimal representation in the analytic continuation of the scalar holomorphic discrete series. The subspace of \(L := S(O(n)\times O(m))\)-invariants is considered and the spectral decomposition for the action of the Casimir element of the Lie algebra of \(H\) is studied. The diagonalisation gives a unitary isomorphism between the subspace of \(L\)-invariants and an \(L^2\)-space with a Hilbert basis given by certain continuous dual Hahn polynomials. The decomposition on the group level is identified with this spectral decomposition. The Plancherel measure turns out to have point masses precisely when \(n- m > 2\). Under these conditions an irreducible representation of \(SO(n,m)\), identified with a parabolically induced representation, is constructed. Finally, a unitary embedding into the representation space for the minimal representation of \(SU(n,m)\) that realises one of the discrete points in the spectrum is obtained.