Regularity of continuous CR maps in arbitrary dimension. (Q1396346)

The authors prove two main results: Let \(M\subset W\subset{\mathbb C}^n\) and \(M'\subset W'\subset{\mathbb C}^n (n\geq 2)\) be real analytic smooth closed real hypersurfaces of finite type, where \(W,W'\) are open sets. If \(f\colon M\to M'\) is a continuous non-constant CR mapping then \(f\) extends holomorphically to a neighborhood of \(M\). In particular, a proper holomorphic mapping \(f\colon D\to D'\) extends holomorphically to a neighborhood of \(\overline D\), if \(D,D'\) are bounded domains in \({\mathbb C}^n\) with real analytic smooth boundary and if \(f\) already extends continuously to \(\overline D\). We now describe some main ideas in the paper. Since \(M\) is real analytic and of finite type, a result of \textit{M. S. Baouendi} and \textit{F. Treves} [Duke Math. J. 51, 77--107 (1984; Zbl 0564.32011)] says that all continuous CR functions on \(M\), including \(f\), extend holomorphically to a fixed open set \(V^-\) having \(M\) as part of boundary. The authors first show that \(\Sigma\), the set of points in \(M\) near which \(f\) extends holomorphically, is dense in \(M\). By a result of \textit{M. S. Baouendi} and \textit{L. P. Rothschild} [J. Differ. Geom. 36, No.1, 75--88 (1992; Zbl 0770.32009)], \(f\) is biholomorphic near a dense open set of \(M\). Now the authors employ the machinery of Segre varieties. Assume for the sake of contradiction that there exists \(z_0\in M\setminus \Sigma\). The authors find suitable neighborhoods \(U_1\ni z_0\) and \(U_1'\ni f(z_0)\), and extend the graph of \(f\) over \(U_1\cap V^-\) to a pure \(n\)-dimensional analytic set \(F\), closed in \((U_1\setminus (M\setminus\Sigma)) \times U_1'\), for which the projection \(\pi\colon F\to\pi(F)=U\) is proper (\(U\) open in \(U_1\)). This construction depends on a result of \textit{K. Diederich} and \textit{S. M. Webster} [Duke Math. J. 47, 835--843 (1980; Zbl 0451.32008)] that associated to the real analytic hypersurface \(M'\), the Segre map \(z\to Q_z\) is proper for \(z\) in some open set \(U_1'\). Given \(z_j\) in \(\Sigma\) and \(w_j\) in the Segre variety \(Q_{z_j}\), let \(S_j\) be the irreducible component of \( F\cap [(Q_{w_j}\cap U_1)\times U_1']\) containing the germ of graph of \(f| _{Q_{w_j}}\) at \((z_j, f(z_j))\). For a suitable sequence \((z_j,w_j)\), each \(S_j\) turns out to be large in the sense that \(S_j\) is closed in \((Q_{w_j}\cap U_1)\times U_1'\) and does not intersect \((M\setminus\Sigma)\times U_1'\), and that the projection \(\pi\colon S_j\to \pi(S_j) \subset Q_{w_j}\) remains proper. Let \(cl(S_j)\) be the cluster set of \(\{S_j\}\), i.e., the set of accumulation points of \(\{z_j\}\) with \(z_j\in S_j\). Roughly speaking, a result of Pinchuk and Verma [cited above] yields the desired contradiction, if \(cl(S_j)\) intersects \(U\times U_1'\). The authors however cannot establish the latter directly, and instead they construct a sequence of analytic sets \(\sigma_j(\subset\pi(S_j))\) of positive dimension, which are closed in the above \(U_1\), shrunk possibly, so that \(cl(\sigma_j)\subset M\setminus \Sigma\). Next the authors formulate the following conjecture, which is of independent interest. Conjecture. Given a sequence of closed complex analytic sets \(A_\nu\subset W\subset{\mathbb C}^n\) of pure dimension \(p\geq 1\), the cluster set \(cl(A_\nu)\) is not contained in any real analytic CR manifold of finite type in the open set \(W\). A significant part of the paper is devoted to the proof of this conjecture under some additional conditions, which is sufficient for the proof of the above local result. The main techniques in this paper depend heavily on the method of Segre varieties, which was first used by \textit{S. M. Webster} [Invent. Math. 43, 53--68 (1977; Zbl 0348.32005)] to the mapping problem shortly after works of \textit{Lewy} [Acad. Naz. Linc. 35, 1--8 (1977)] and \textit{S. I. Pinchuk} [Math. USSR-Sb. 27, 375--392 (1975; Zbl 0366.32010)]. Earlier step-by-step results of \textit{K. Diederich} and \textit{S. Pinchuk} [Indiana Univ. Math. J. 44, No. 4, 1089--1126 (1995; Zbl 0857.32015), Doc. Math., J. DMV, Extra Vol. ICM Berlin 1998, vol. II, 703--712 (1998; Zbl 0914.32008)], and Pinchuk-Verma [cited above] are used extensively. Needless to say, the proofs are very technical and details are considerable. The authors give a much simpler proof for the global result stated at the beginning. For related results on extendibility of holomorphic maps, see \textit{M. S. Baouendi} and \textit{L. P. Rothschild} [Invent. Math. 93, No. 3, 481--500 (1988; Zbl 0653.32020)], \textit{K. Diederich} and \textit{J. E. Fornaess} [Math Ann. 282, 681--700 (1988; Zbl 0661.32025)], \textit{K. Diederich; J. E. Fornaess} and \textit{Z. Ye} [J. Geom. Anal. 4, 539--552 (1994; Zbl 0864.32014)], and \textit{X. Huang} [Commun. Partial Differ. Equations 21, No. 11--12, 1781--1828 (1996; Zbl 0886.32010)].