The automorphism group and limit set of a bounded domain I: the finite type case (Q2309116)

By definition a bounded pseudoconvex domain \(\Omega \subset\mathbb{C}^d\) has a finite type, if its boundary \(\partial \Omega\) is smooth and has finite type (in the sense of D'Angelo). This article is Part I of a series of papers by the author studying the groups \(\mathrm{Aut}(\Omega)\) and the limit sets \(\mathcal L (\Omega)\). For bounded domains, \(\mathcal L(\Omega )\) is non-empty if and only if \(\mathrm{Aut}(\Omega)\) is non-compact. A family of examples are the so-called generalized ellipses. The main result of this paper shows that some interesting properties of generalized ellipses may be extended to pseudoconvex domains of finite type. This main result states that if for an orbit of the automorphism group \(\mathrm{Aut}(\Omega)\) its limit set \(\mathcal L(\Omega)\) contains at least two distinct points, then the automorphism group has finitely many components and it is the almost direct product of a compact group and connected Lie group, which is locally isomorphic to \(\mathrm{Aut}(\mathbf{B}_k)\), where \(\mathbf{B}^k \subset\mathbb{C}^k\) is the unit ball. Also it is proved that the limit set here is a smooth submanifold diffeomorphic to the sphere of dimension \(2k-1\). One of the main tools in this article is the geometry of the Kobayashi metric on finite type domains. Also two corollaries are proved. The first result involves determining an automorphism from its \(k\)-jet. And the second one is a variant of the Tits alternative theorem. Suppose \(\Omega\subset\mathbb{C}^d\) is a bounded pseudoconvex domain with real analytic boundary. If \(H \subset\mathrm{Aut}(\Omega)\) is a subgroup, then either \(H\) contains a free group or it has a finite index solvable subgroup. The proof uses the fact that every bounded pseudoconvex domain with real-analytic boundary is of finite type. At the end of the paper there is an appendix with references describing some basic properties of semisimple Lie groups and the symmetric spaces they act on.