On the Cohen-Macaulay property of modular invariant rings (Q1291113)

Let \(G \subseteq Gl(V)\) denote a finite group acting on a vector space \(V\) of dimension \(n\) over a field \(K.\) Then \(G\) acts on the symmetric algebra \(R\) of the dual of \(V.\) If the characteristic of \(K\) does not divide the order of \(G\) the ring of invariants \(R^G\) is always a Cohen-Macaulay ring as shown by \textit{M.~Hochster} and \textit{J.~A.~Eagon} [Am. J. Math. 93, 1020-1058 (1971; Zbl 0244.13012)]. This is not true in the modular case, i.e. when the order of G is a multiple of the characteristic, as shown by examples of \textit{M.-J.~Bertin} [C. R. Acad. Sci., Paris, Sér. A 264, 653-656 (1967; Zbl 0147.29503)]. The main problem of the paper is to contribute to a clarification of the Cohen-Macaulayness of \(R^G\) in the modular case. In the first section it is shown that the regularity of sequences \(f_1, \cdots, f_n \in R^G\) is related to the group cohomology \(H^\star(G,R)\) with values in the polynomial ring \(R,\) viewed as an \(R^G\)-module. It turns out that `large' annihilators of elements of the cohomology destroy the Cohen-Macaulayness of \(R^G.\) As an application there is a result saying that if \(H \subseteq G\) is a strongly \(p\)-embedded subgroup, then \(\text{depth} R^H = \text{depth} R^G.\) In fact this is a partial converse to a result by \textit{H. E. A. Campbell, I. Hughes} and \textit{R. D. Pollack} [Can. Math. Bull. 34, No. 1, 42-47 (1991; Zbl 0695.14008)]. In the second section the author's techniques are used in order to prove the non-Cohen-Macaulayness of \(R^G\) provided there is an integer \(r > 0\) such that \(H^r(G,R) \not= 0\) and any element \(\sigma \in G\) of order \(p\) has \(\text{rank} (\sigma - 1) \geq r +2.\) As a corollary it follows that in the modular case the ring of sufficiently large vector invariants is not Cohen-Macaulay. The third section is concerned mainly with the group \(H^1(R,G)\) permitting a more accurate analysis of the annihilators. It provides the second main result saying that whenever \(G\) has a normal sugroup of index \(p\) containing all bireflections (that is an element \(1 \not= \sigma \in G\) such that \(\text{rank} (\sigma -1) \leq 2\)) then \(R^G\) is not a Cohen-Macaulay ring. Consequently if \(G\) is a \(p\)-group such that \(R^G\) is a Cohen-Macaulay ring, then \(G\) is generated by bireflections. Altogether there is a deep analysis of the non-Cohen-Macaulayness of invariant rings in the paper. In fact, all known non-Cohen-Macaulay invariant rings can be understood in terms of the author's methods.