F-method for symmetry breaking operators (Q2636747)

The author first gives geometric criteria for finite multiplicity and uniformly bounded multiplicity of the local and non-local symmetric breaking operators, and next, by extending the so-called F-method known for local operators to non-local ones, he obtains the functional identities and the explicit residue formulas of regular symmetric breaking operators on rank one orthogonal groups. Let \(G\) be a real reductive linear Lie group and \(P\) a parabolic subgroup of \(G\). For a finite dimensional representation \(\lambda\) of \(P\) on \(V\), let \({\mathcal V}=G\times_P V\) be the associated homogeneous vector bundle over \(G/P\). Then \(G\) acts continuously on the space \(C^\infty(G/P,{\mathcal V})\) of smooth sections. Similarly, we define \(C^\infty(G'/P',{\mathcal W})\) for a reductive subgroup \(G'\) of \(G\), a parabolic subgroup \(P'\) of \(G'\), and a finite dimensional representation of \(P'\) on \(W\). We assume \(P'\subset P\cap G\). Then the space of (non-local) symmetric breaking operators is given by \[ \text{Hom}_{G'}(C^\infty(G/P,{\mathcal V}), C^\infty(G'/P',{\mathcal W})). \] As a subspace of \(G'\)-intertwining differential operators, the local symmetric breaking operators is given by \[ \text{Diff}_{G'}(C^\infty(G/P,{\mathcal V}), C^\infty(G'/P',{\mathcal W})). \] For these spaces, geometric equivalent conditions that \(\dim <\infty\) (finite multiplicity) and \(\sup_{V}\sup_{W}\dim <\infty\) (uniformly bounded multiplicity) are obtained (Theorems 2.3 and 2.7). Let \({\mathfrak g}={\mathfrak n}^- + {\mathfrak l} + {\mathfrak n}^+\) be the Gelfand-Naimark decomposition of the complexification \(\mathfrak g\) of the Lie algebra \({\mathfrak g}_{\mathbb R}\) of \(G\). By extending F-method to non-local operators, the Fourier transform from \({\mathcal S}'({\mathfrak n}_{\mathbb R}^-)\) to \({\mathcal S}'({\mathfrak n}_{\mathbb R}^+)\) characterizes non-local symmetric breaking operators under the assumption that \(P'N^-P=G\) (Theorem 3.4). On rank one orthogonal groups, by using F-method, the author shows that Kummer's relation on Gauss hypergeometric functions \({}_2F_1\) corresponds to functional equations for symmetric breaking operators, and moreover, the fact that the Gegenbauer polynomial of even degree is given by \({}_2F_1\) corresponds to a residue formula for regular symmetric breaking operators (Theorems 5.6 and 5.2).