Inequalities defining orbit spaces (Q1066210)

Let \(W\) be a finite dimensional real representation space of a compact Lie group \(K\), and let \(p_ 1,\dots, p_ m\) be polynomial functions on \(W\) which generate the algebra \({\mathbb R}[W]^ K\) of \(K\)-invariant polynomial functions on \(W\) (such a finite set of generators does always exist by a theorem of Hilbert and Hurwitz). Denote by \(I\) the ideal of relations of \(p_ 1,\dots, p_ n\) in \({\mathbb R}[Y_ 1,\dots,Y_ n]\), and by \(Z\subseteq {\mathbb R}^ m\) the corresponding irreducible real algebraic set. Let \(p=(p_ 1,\dots, p_ m): W\to {\mathbb R}^ m\). Then \(p^*\) induces an isomorphism of \({\mathbb R}[Z]\) with \({\mathbb R}[W]^ K\), and the image \(X=p(W)\subseteq Z\) of \(p\) is a semialgebraic set. Moreover, by a theorem of G. Schwarz \(p\) induces a homeomorphism \(\bar p: W/K\to X\), such that \(X\subset Z\) may be regarded as the orbit space of \(K\) on \(W\). The main result of the paper under review is the following characterization of \(X\) as a subset of \(Z\): Let \((\;,\;)\) be a \(K\)-invariant inner product on \(W\), and let \(^{\sim}(w)=\{(dp_ i(w),dp_ j(w)\}_{i,j}\). Then \(^{\sim}\) is a symmetric \(m\times m\)-matrix with entries in \({\mathbb R}[W]^ K\). There exists a unique matrix valued function \(\text{Grad}\) on \(Z\) such that \(^{\sim}(w)=\text{Grad}(p(w))\), and we have \(X=\{z\in Z:\text{Grad}(z)\) is positive semidefinite\(\}\), which gives a finite number of inequalities describing \(X\) in \(Z\). The proof makes use of results of Kempf-Ness, Dadok-Kac and Luna on orbits of the complexification of \(K\) on \(W\otimes_{{\mathbb R}}{\mathbb C}\). Several applications of this result are discussed, it is determined, when \(X=Z\), and, more generally, the boundary of \(X\) in \(Z\) is described. Also other methods to obtain inequalities defining \(X\) are discussed. Finally, an equivariant version of Hilbert's 17th problem for real reductive groups is discussed. Positive results are established in the compact case, and some counterexamples are given in the non-compact case.