A group theoretic proof of a compactness lemma and existence of nonradial solutions for semilinear elliptic equations (Q2658546)

The paper is concerned with compact embeddings \(H^1_{0,G}(\Omega)\hookrightarrow L^p(\Omega)\) where \(\Omega\subset M\) is a domain in a Riemannian manifold \(M\), \(G\subset \mathrm{Iso}(M)\) is a closed subgroup of the group of isometries, and \(H^1_{0,G}(\Omega)\) is the space of \(G\)-invariant functions in \(H^1_0(\Omega)\). The authors consider in particular the cases \(M=\mathbb{R}^n\), \(M\) a Hadamard manifold, or \(M\) a symmetric space of noncompact type. In the case where \(\Omega\subset\mathbb{R}^n\) is an unbounded domain which is invariant under \(G\subset O(n)\) their embedding result reduces to the known case that the group orbits \(Gx=\{gx:g\in G\}\) have dimension at least one if \(|x|\) is large. In some theorems and examples in section 3 the spaces \(H^1_{0,G}(\mathbb{R}^n)\) are unfortunately trivial. As a consequence, the proof of the existence of nonradial solutions of certain nonlinear scalar field equations in dimensions \(n=3\) or \(n=5\) are not correct. In dimensions \(n=4\) or \(n\ge6\) the existence of nonradial solutions has been proved in [the reviewer and \textit{M. Willem}, J. Funct. Anal. 117, No. 2, 447--460 (1993; Zbl 0790.35021)], dimension \(n=5\) is treated in [\textit{S. Lorca} and \textit{P. Ubilla}, Nonlinear Anal., Theory Methods Appl., Ser. A, Theory Methods 58, No. 7--8, 961--968 (2004; Zbl 1127.35015)], the general case \(n\ge2\) in [\textit{M. Musso} et al., J. Eur. Math. Soc. (JEMS) 14, No. 6, 1923--1953 (2012; Zbl 1263.35198)].