Diophantine questions in the class of finitely generated nilpotent groups (Q285579)

In the paper under review, the author shows several negative results about solvability of Diophantine questions in the class of finitely generated nilpotent groups. All of them are based on the following classical undecidability theorem by Matijasevich: there is a polynomial \(D(x_1, \ldots ,x_k)\) over the integers (of certain degree \(d\), and certain number of variables \(k\)) such that there is no algorithm which, given an integer \(c\in \mathbb{Z}\) decides whether the Diophantine equation \(D(x_1, \ldots ,x_k)=c\) has an integral solution or not. For every polynomial \(D(x_1, \ldots ,x_k)\), the author manages to construct a finitely generated nilpotent group of class 2 \(G(D)\) with the following property: for every integer \(c\in \mathbb{Z}\) there is an element \(w=w(c)\in G(D)\) such that \(w\) is a commutator of two elements in \(G(D)\) if and only if the equation \(D(x_1, \ldots ,x_k)=c\) has an integral solution. Therefore, the commutator problem (given \(w\in G\) decide if there exists \(x,y\in G\) such that \([x,y]=w\)) is undecidable in the group \(G(D)\). One can naturally define the Diophantine problem for an arbitrary group \(G\): find an algorithm which, given an equation over \(G\), say \(u(G,x_1, \ldots ,x_k)\in G*F(\{x_1, \ldots ,x_k\})\), decides whether \(u=1\) has a solution in \(G\) (i.e., whether there exists \(h_1,\ldots ,h_k\in G\) such that \(u(G,h_1, \ldots ,h_k)=1\) in \(G\)). The above group \(G\) happens to be the first known example of a finitely generated nilpotent group with undecidable Diophantine problem for the class of quadratic equations. In the paper, another finitely generated nilpotent group \(H\) of class 2 is presented having undecidable endomorphism problem: there is no algorithm such that, given two elements \(x,y\in H\) decides whether one can be mapped to the other by some endomorphism of \(H\). Finally, the author also proves that the retract problem is undecidable for the family of finitely generated nilpotent groups of class 2: there is no algorithm which, given such a group \(G\) and a subgroup \(H\leq G\), decides whether \(H\) is a retract of \(G\) or not (i.e., whether there exists a homomorphism from \(G\) to \(H\) restricting to the identity of \(H\)).