Minimal co-volume hyperbolic lattices. I: The spherical points of a Kleinian group (Q2389212)

The paper presents a main step in a program to identify the Kleinian group of smallest co-volume, or equivalently the orientable hyperbolic 3-orbifold of smallest volume. The candidate is a 2-fold extension \(\Gamma_0\) of the orientation-preserving subgroup (tetrahedral group) in the group generated by the reflections in the faces of the hyperbolic Coxeter tetrahedron of type 3 - 5 - 3, with \(\text{Vol}(\mathbb H^3/\Gamma_0) = 0,03905 \ldots\) (the singular set of the orbifold \(\mathbb H^3/\Gamma_0\) is the quotient of the 1-skeleton of this Coxeter tetrahedron by its obvious 2-fold symmetry); by results of \textit{T. Chinburg} and \textit{E. Friedman}, \(\Gamma_0\) is known to be the smallest co-volume \textit{arithmetic} Kleinian group [Invent. Math. 86, 507--527 (1986; Zbl 0643.57011)]. In the present paper, it is shown that the two Kleinian groups of smallest co-volume which contain a tetrahedral, octahedral or icosahedral subgroup (that is, a spherical triangle subgroup of type (2,3,3), (2,3,4) or (2,3,5), fixing a point in hyperbolic 3-space) are the groups \(\Gamma_0\) and a second group \(\Gamma_1\) with \(\text{Vol}(\mathbb H^3/\Gamma_1) = 0,0408 \ldots\) (see a paper by the reviewer [Rend. Ist. Mat. Univ. Trieste 32, Suppl. 2, 149--161 (2001; Zbl 1012.57026)], for pictures of the singular sets of these and some other small volume hyperbolic 3-orbifolds which are the candidates for the smallest hyperbolic 3-orbifolds). In previous work [J. Differ. Geom. 49, No. 3, 411--435 (1998; Zbl 0989.57010)], the authors had shown that, if a Kleinian group \(\Gamma\) contains an elliptic element of finite order \(\geq 4\) with a simple axis, then \(\text{Vol}(\mathbb H^3/\Gamma) > \text{Vol}(\mathbb H^3/\Gamma_1)\). Hence the basic missing case necessary for determining the Kleinian group of minimal co-volume is that of a Kleinian group with torsion only of order two and three; this case is considered in a preprint by Marshall and Martin which constitutes the second part of the present paper. Roughly, the proofs in the present paper are based on the following ideas. It is known that the spectrum of distances between the axes of elliptic rotations in Kleinian groups is initially discrete (i.e., for small distances), and in the present paper a similar result is shown for the distances of fixed points of conjugate spherical triangle subgroups of a Kleinian group. This requires the main work of the present paper, involving complex calculations in hyperbolic trigonometry and motivated by a paper by \textit{D. A. Derevnin} and \textit{A. D. Mednykh} considering the case of icosahedral fixed points [Sov. Math., Dokl. 37, No. 3, 614--617 (1988); translation from Dokl. Akad. Nauk SSSR 300, No. 1, 27--30 (1988; Zbl 0713.30044)]. Then the configurations which are too small compared with the minimal candidates can be eliminated by arithmetic arguments (in fact, all examples of sufficiently small co-volume turn out to be arithmetic).