A triple ratio on the unitary Stiefel manifold. (Q1419568)

Let \(G\) be the identity component of the automorphism group of a bounded Hermitian symmetric domain \(D=G/K\) in \(\mathbb C^n\). It is known that the action of \(G\) extends to the Shilov boundary \(S\) of \(D\) and that \(G\) operates transitively and almost \(2\)-transitively on \(S\). In particular, the diagonal action of \(G\) on \(S\times S\) has a dense open orbit on \(S\times S\). Therefore it is natural to consider the \(G\)-action on the triple \(S\times S\times S\) and to ask for characteristic invariants of this action. In a previous paper the author and \textit{B. Ørstedt} solved this problem for domains \(D\) of tube type [Transform. Groups 6, No. 4, 303--320 (2001; Zbl 1078.53076)], showing that in that case there are only finitely many \(G\)-orbits in \(S\times S\times S\) and constructing a characteristic invariant for the action. The case of the unit ball in \(\mathbb C^n\) was already solved by \textit{E. Cartan} for \(n=2\) [Comment. Math. Helv. 4, 158--171 (1932; JFM 58.0671.01)] and later for general \(n\) by \textit{A. Koranyi} and \textit{H. M. Reimann} [Enseign. Math., II. Sér. 33, 291--300 (1987; Zbl 0638.22005)]. In the present paper the problem is studied for the unit ball \(D\) in \(\mathbb C^{p\times q}\) with \(p> q\). In this case \(G=PSU(p,q)\), and the Shilov boundary of \(D\) is the unitary Stiefel manifold \(S=\{\sigma\in\mathbb C^{p\times q}\mid \sigma^*\sigma={\mathbf 1}_q\}\). Two elements \(\xi\) and \(\sigma\) of \(S\) are called transversal to each other if \(\xi -\sigma\) is injective. Let \(S^3_T\) denote the set of those elements in \(S\times S\times S\) whose components are pairwise transversal to each other. The author gives a complete classification of the \(G\)-orbits in \(S^3_T\) in terms of the GL\((q,\mathbb C)\)-action on \(T_q=\{z\in \mathbb C^{q\times q}\mid \operatorname{Im} z\in\Omega_q\}\), where \(\Omega_q\) denotes the set of positive-definite \((q,q)\)-matrices.