Separating invariants of three nilpotent \(3\times 3\) matrices (Q2226397)

This paper studies the invariants of nilpotent matrices over an algebraically closed field of characteristic zero. Let \(\mathcal{N}_n\) be the space of \(n\times n\) nilpotent matrices; let \(\mathcal{N}_n^d\) the space of \(d\)-tuples of \(n\times n\) nilpotent matrices; and let \(\mathcal{O}(\mathcal{N}_n^d)\) be the coordinate ring. The general linear group \(\mathrm{GL}_n\) acts on \(\mathcal{N}_n^d\) by simultaneous conjugation, and so also acts on the coordinate ring. The invariants studied are elements of \(\mathcal{O}(\mathcal{N}_n^d)^{\mathrm{GL}_n}\), i.e., polynomial functions from \(d\)-tuples of \(n\times n\) nilpotent matrices to the field with the property that \[ f(gA_1g^{-1},\ldots,gA_dg^{-1})=f(A_1,\ldots,A_d) \] for all \(g\in \mathrm{GL}_n\) and \(A_1,\ldots,A_d\in\mathcal{N}_n\). The authors study minimal generating sets for \(\mathcal{O}(\mathcal{N}_3^d)^{\mathrm{GL}_n}\), as an algebra, and minimal separating sets. Two elements \(u,v\) of \(\mathcal{N}_n^d\) are separated if there is an invariant map \(f\in \mathcal{O}(\mathcal{N}_n^d)^{\mathrm{GL}_n}\) such that \(f(u)\ne f(v)\); and a subset \(S\) of \(\mathcal{O}(\mathcal{N}_n^d)^{\mathrm{GL}_n}\) is separating if it separates all such \(u,v\). A generating set will be separating, but a separating set need not generate. The main theorem of the paper constructs minimal generating and separating sets in \(\mathcal{O}(\mathcal{N}_3^3)^{\mathrm{GL}_n}\). The former consists of 39 elements each of degree at most 6, and the latter consists of 26 elements each of degree at most~5. The authors also prove that the elements of \(\mathcal{O}(\mathcal{N}_3^d)^{\mathrm{GL}_n}\) of degree at most 6 form a separating set for any \(d\).