The Tits alternative for \(\text{Out}(F_n)\). II: A Kolchin type theorem. (Q2388267)

Let \(F_n\) be the free group of rank \(n\) and \(\text{Out}(F_n)\) the group of outer automorphisms of \(F_n\). It is shown that \(\text{Out}(F_n)\) has many common properties with both linear groups and mapping class groups of surfaces. For \(n=2\) the three groups are, in fact, the same group, and for all ranks \(\text{Out}(F_n)\) contains the mapping class group of any punctured surface with fundamental group \(F_n\) and \(\text{GL}(n,\mathbb{Z})\) is a quotient of \(\text{Out}(F_n)\). A group satisfies the ``Tits alternative'' if every subgroup is either solvable-by-finite or contains a nonabelian free subgroup. The Tits alternative does hold for finitely generated linear groups [\textit{J. Tits}, J. Algebra 20, 250-270 (1972; Zbl 0236.20032)] and for mapping class groups [\textit{N. V. Ivanov}, Dokl. Akad. Nauk SSSR 275, 786-789 (1984; Zbl 0586.20026); \textit{J. McCarthy}, Trans. Am. Math. Soc. 291, 583-612 (1985; Zbl 0579.57006); \textit{J. S. Birman, A. Lubotzky} and \textit{J. McCarthy}, Duke Math. J. 50, 1107-1120 (1983; Zbl 0551.57004)]. The authors in a series of three papers prove the Tits alternative for \(\text{Out}(F_n)\). In the first paper [Ann. Math. (2) 151, No. 2, 517-623 (2000; Zbl 0984.20025)] they gave an extensive introduction to all three papers and outlined the all procedure of the proof. There they considered individual automorphisms with primary emphasis on those with exponential growth rates and reduced the problem to deciding whether subgroups consisting entirely of ``unipotent polynomially-growing'' automorphisms satisfy the Tits alternative. In the second paper (the paper under review) they develop different techniques. An element of \(\text{Out}(F_n)\) of polynomial growth is `unipotent' if the induced automorphism of \(H_1(F_n,\mathbb{Z})=\mathbb{Z}^n\) is unipotent. The authors prove that every subgroup of polynomially growing outer automorphisms has a finite index unipotent subgroup, so they avoid difficulties arising from finite order phenomena. Let \(\text{Rose}_n\) be the bouquet of \(n\) circles. A marked graph is a graph equipped with a homotopy equivalence from \(\text{Rose}_n\). A homotopy equivalence \(f\colon G\to G\) on a marked graph \(G\) induces an outer automorphism of the fundamental group of \(G\) and therefore an element of \(\text{Out}(F_n)\), this element has the \(f\) as a `representative'. Let \(\emptyset=G_0\varsubsetneq G_1\varsubsetneq\cdots\varsubsetneq G_k=G\) be a filtration of marked graph \(G\) where \(G_i\) is obtained from \(G_{i-1}\) by adding a single edge \(e_i\). A homotopy equivalence \(f\colon G\to G\) is `upper triangular' (with respect to the filtration) if \(f(e_i)=v_ie_iu_i\), where \(v_i,u_i\) are closed paths in \(G_{i-1}\). It is proved that an outer automorphism is unipotent if and only if it has a representative that is upper triangular with respect to some filtered marked graph \(G\). For any filtered marked graph \(G\) it is proved that the set, say \(\mathcal Q\), of upper triangular homotopy equivalences of \(G\) up to homotopy relative to the vertices of \(G\) is a group. There is a natural map from \(\mathcal Q\) to \(\text{UPG}(F_n)\), the set of unipotent outer automorphisms of \(F_n\). A unipotent subgroup of \(\text{Out}(F_n)\) is `filtered' if it lifts to a subgroup of \(\mathcal Q\). `Free factor systems' are considered and a natural action of \(\text{Out}(F_n)\) on free factor systems is obtained. For a subgroup \(\mathcal H\) of \(\text{Out}(F_n)\) a free factor system \(\mathcal F\) is \(\mathcal H\)-invariant if each element of \(\mathcal H\) fixes \(\mathcal F\). A subgraph \(K\) of a marked real graph determines a free factor system \(\mathcal F(K)\). The main theorem in this paper is: Theorem: Every finitely generated unipotent subgroup \(\mathcal H\) of \(\text{Out}(F_n)\) is filtered. For any \(\mathcal H\)-invariant free factor system \(\mathcal F\), the marked filtered graph \(G\) can be chosen so that \(\mathcal F(G_r)=\mathcal F\) for some filtration element \(G_r\). The number of edges of \(G\) can be taken to be bounded by \(\tfrac{3n}{2}-1\) for \(n>1\). This theorem is parallel to the Kolchin Theorem concerning linear groups [\textit{J.-P. Serre}, Lie algebras and Lie groups, Berlin: Springer-Verlag (1992; Zbl 0742.17008)]. As a Corollary it is proved that: Every unipotent subgroup \(\mathcal H\) of \(\text{Out}(F_n)\) either contains a nonabelian free group or is solvable. This corollary is used to prove the full Tits alternative. Theorem: Let \(\mathcal H\) be any subgroup of \(\text{Out}(F_n)\). Then either \(\mathcal H\) is virtually solvable, or contains a nonabelian free group. In the third paper the authors strengthen the previous result by proving: A solvable subgroup of \(\text{Out}(F_n)\) has a finitely generated Abelian subgroup of index at most \(3^{5n^2}\). A result proved also by \textit{E. Alibegović} [Geom. Dedicata 92, 87-93 (2002; Zbl 1041.20024)].