Existence and phase separation of entire solutions to a pure critical competitive elliptic system (Q1703150)

Let \(\alpha,\beta\in (1,2]\), with \(\alpha+\beta=\frac{2N}{N-2}=:2^*\), \(N\geq4\), and let \(\mu_1,\mu_2\in (0,\infty)\), \(\lambda \in (-\infty,0)\). The authors consider the following weakly coupled critical elliptic system \[ \begin{cases} -\Delta u=\mu_1|u|^{2^*-2}u+\lambda \alpha|u|^{\alpha-2}|v|^\beta u,\\ -\Delta u=\mu_2|v|^{2^*-2}u+\lambda \beta|v|^{\beta-2}|u|^\alpha v,\\ u,v\in D^{1,2}(\mathbb{R}^N), \end{cases}\tag{S} \] and investigate the existence of fully nontrivial solutions satisfying certain invariance properties. In particular, given a closed subgroup \(\Gamma\) of the group of linear isometries such that \(\Gamma p\notin \mathbb{S}^N\) and \(\Gamma p\) has positive dimension for each \(p\in \mathbb{S}^N\), the authors prove that system \((S)\) admits a positive solution \((u,v)\) which is fully nontrivial (i.e., both \(u\) and \(v\) are nonzero) and \(\Gamma\)-invariant (i.e,\(|\text{det}\tilde{\gamma}'(x)|^{1/2^*}u(\tilde{\gamma}x)=u(x)\) for each \(\gamma \in \Gamma\) and \(x\in \mathbb{R}^N\), where \(\tilde{\gamma}=\sigma \circ \gamma^{-1}\circ \sigma^{-1}\) and \(\sigma:\mathbb{S}^N\rightarrow \mathbb{R}^N\cup\{\infty\}\) is the stereographic projection). Moreover, they also prove that there exists \(\lambda^*<0\), depending on \(\alpha,\beta,\mu_1,\mu_2\), such that no fully nontrivial synchronized solution (that is a solution of the form \((sw,tw)\), \(s,t\in \mathbb{R}\), \(w\in D^{1,2}(\mathbb{R}^N)\)) can exist if \(\lambda<\lambda^*\). In the case \(\mu_1=\mu_2\), \(\alpha=\beta\) and \(\lambda \leq -\frac{\mu_1}{\alpha}\), the existence of infinitely many fully nontrivial \(\Gamma\)-invariant solutions which are not conformally equivalent is also established. The authors prove these results by studying the restriction to the set \[ \mathcal{N}^\Gamma:=\big\{(u,v)\in (D^{1,2}(\mathbb{R}^N))^2\setminus{(0,0)}: \partial E_u(u,v)u=\partial E_v(u,v)v=0, \;\gamma(u,v)=(u,v), \;\forall \gamma\in \Gamma\big\}. \] Under the above assumptions on \(\Gamma\), it is proved that \(\mathcal{N}^\Gamma\) is a closed \(C^1\)-submanifold of \(D^{1,2}(\mathbb{R}^N)^2\) and that \(E_{\mid \mathcal{N}^\Gamma}\) admits a global minimizer on \(\mathcal{N}^\Gamma\) which turns out to be a critical point of \(E\). For \(\mu_1=\mu_2\), \(\alpha=\beta\) and \(\lambda \leq -\frac{\mu_1}{\alpha}\), by adapting a \(C^1\) Ljusternik-Schnirelman result on manifolds due to A. Szulkin, the authors obtain infinitely many critical points of \(E_{\mid \mathcal{N}^\Gamma}\) which are also critical points of \(E\). In addiction, the authors analyze the phase separation of positive solutions as \(\lambda\rightarrow -\infty\). More precisely, for a sequence \(\lambda_k\) in \((-\infty,0)\) with \(\lambda_k\rightarrow-\infty\), it is shown that, up to a subsequence, the positive fully nontrivial solution \((u_k,v_k)\) of system \((S)\) corresponding to \(\lambda=\lambda_k\) converges strongly in \((D^{1,2}(\mathbb{R}^N))^2\) to \((u_\infty,v_\infty)\), where \(u_\infty,v_\infty\) are continuous functions such that \(u_\infty,v_\infty\geq 0\), \(u_\infty\cdot v_\infty\equiv 0\), the sets \(\Omega_1:=\{x\in \mathbb{R}^N: u_\infty(x)>0\}\), \(\Omega_2:=\{x\in \mathbb{R}^N: u_\infty(x)>0\}\) are \(\Gamma\)-invariant, connected and satisfy \(\overline{\Omega_1\cup \Omega_2}=\mathbb{R}^N\), and \[ \;u_\infty \;\text{solves} \;-\Delta u=\mu_1 |u|^{2^*-2}u \;\text{in} \;D_0^{1,2}(\Omega_1), \;\;\;v_\infty \;\text{solves} \;\;-\Delta v=\mu_1 |v|^{2^*-2}v \;\text{in} \;D_0^{1,2}(\Omega_2). \] When \(\Gamma=O(m)\times O(n)\), a more precise description of the domains \(\Omega_1,\Omega_2\) is given.