On the groups of unitriangular automorphisms of relatively free groups. (Q1937740)

Let \(F_n\) be the free group of rank \(n\) and \(X_n=\{x_1,x_2,\dots,x_n\}\) a basis of \(F_n\). As usual, for \(n<m\), the group \(F_n\) naturally embeds into \(F_m\). If \(F_{\aleph_0}\) is the free group of countable infinite rank with basis \(X_{\aleph_0}=\{x_1,x_2,\dots,x_n,\dots\}\), then \(F_{\aleph_0}\) can be considered as the union \(F_{\aleph_0}=\bigcup_{n=1}^\infty F_n\). For any variety \(\mathcal C\) of groups let \(\mathcal C(F_n)\) denote the verbal subgroup of \(F_n\) corresponding to \(\mathcal C\) and let \(F_n(\mathcal C)=F_n/\mathcal C(F_n)\) be the relatively free group of rank \(n\) in \(\mathcal C\). Putting \(y_i=x_i\mathcal C(F_n)\) it is obtained a standard basis \(Y_n=\{y_1,y_2,\dots,y_n\}\) of \(F_n(\mathcal C)\), which gives the possibility, for \(n<m\), to consider \(F_n(\mathcal C)\) as a subgroup of \(F_m(\mathcal C)\). In this article the authors study the structure of a natural subgroup of the automorphism group of the relatively free group \(F_n(\mathcal C)\) for arbitrary variety \(\mathcal C\) and every \(n\geq 2\). Denote \(G_n=F_n(\mathcal C)\). For a fixed basis \(Y_n=\{y_1,y_2,\dots,y_n\}\) of \(G_n\) it is possible to define an automorphism of \(G_n\) by a mapping of the form \(\varphi\colon y_1\to y_1\), \(y_i\to u_iy_i\) for \(i=2,\dots,n\), where \(u_i=u_i(y_1,\dots,y_{i-1})\) is an arbitrary element of \(G_{i-1}\). Therefore for every \((u_2,\dots,u_n)\in G_1\times\cdots\times G_{n-1}\) it is defined an automorphism of \(G_n\) called a unitriangular automorphism. The unitriangular automorphisms of \(G_n\) form a subgroup of the all automorphism group and let denote it by \(U_n\). The subgroup \(U_n\) is independent, up to isomorphisms, of the choice of the basis \(Y_n\). The main results of the paper are included in the following theorems. Theorem A. Let \(G_n\) be a rank \(n\geq 2\) relatively free group in an arbitrary variety \(\mathcal C\) of groups. Then the group \(U_n\) of unitriangular automorphisms of \(G_n\) admits a normal series \(1=N_0\leq N_1\leq\cdots\leq N_{n-1}=U_n\) in which the quotients are isomorphic to relative free groups in \(\mathcal C\); namely, \(N_i/N_{i-1}\simeq G_{n-i}\) for \(i=1,\dots,n-1\). Moreover, the quotients of this series split, and so \(U_n\) is the product of disjoint subgroups \(U_n=U_n^{(n)}\cdots U_n^{(2)}\simeq G_{n-1}\cdots G_1\), where \(U_n^{(i)}\simeq G_{i-1}\) consists of one-row unitriangular automorphisms corresponding to \(y_i\), that is, of unitriangular automorphisms acting identically on all basis elements \(y_l\) except possibly for \(y_i\). In particular, \(U_n\) lies in \(\mathcal C^{n-1}\). Theorem B. Let \(G_n\) be a rank \(n\geq 2\) relatively free group in an arbitrary variety \(\mathcal C\) of groups. The following hold: \(U_1\) is trivial, while \(U_2\) is a cyclic group of order equal to the exponent of \(\mathcal C\). These groups admit faithful matrix representations. For \(n\geq 3\), if \(G_{n-1}\) is a nilpotent group then so is \(U_n\). If \(G_{n-1}\) is nilpotent-by-finite then \(U_n\) is linear over \(\mathbb Z\). Theorem C. Let \(G_n\) be a rank \(n\geq 3\) relatively free group in an arbitrary variety \(\mathcal C\) of groups different from the variety of all groups. If \(G_{n-1}\) is not a nilpotent-by-finite group then the group \(U_n=UT\Aut G_n\) of unitriangular automorphisms of \(G_n\) admits no faithful matrix representation over any field. Moreover, the authors introduce the concept of length \(\ell(\varphi)\) of an automorphism \(\varphi\) of an arbitrary relatively free group and estimate the length of the inverse automorphism \(\varphi^{-1}\) in terms of the length of \(\varphi\). Theorem D. Let \(A_n\) be a rank \(n\) free Abelian group. Every \(\varphi\in\Aut A_n\) satisfies \(\ell(\varphi^{-1})\leq\ell(\varphi)^{n-1}\) and the exponent \(n-1\) cannot be decreased. Let \(F_n\) be a rank \(n\) absolutely free group. Every unitriangular automorphism \(\varphi\in U\Aut F_n\) satisfies \(\ell(\varphi^{-1})\leq\ell(\varphi)^{n-1}\) and the exponent \(n-1\) cannot be decreased. In [Turk. J. Math. 31, Suppl., 105-111 (2007; Zbl 1139.20032)] it is proved by \textit{A. Yu. Ol'shanskiĭ} that if a relatively free group \(G_n\) is neither free nor nilpotent-by-finite then \(\Aut G_n\) is not linear. As the authors notify this result and the used method of proof does not give any information on the possible linearity of the group \(U_n=UT\Aut G_n\).