Hyperelliptic Schottky problem and stable modular forms (Q288673)

The topic of this paper is the hyper-elliptic Schottky problem, i.e., the determination of the locus \(\mathrm{Hyp}_{g}\) of Jacobians of hyper-elliptic curves of genus \(g\) inside the moduli space \(\mathcal{A}_{g}\) of principally polarized \(g\)-dimensional abelian varieties. This problem faces many technical problems for every fixed genus \(g\), so the idea is to work on the limit. Indeed, the Satake compactification \(\mathcal{A}_{g}^{S}\) of \(\mathcal{A}_{g}\) is precisely the boundary component of \(\mathcal{A}_{g+1}^{S}\), so that the limit \(\mathcal{A}_{\infty}\) is defined as an ind-scheme. The functions (or more generally, sections of line bundles) on \(\mathcal{A}_{\infty}\) are stable modular forms, that is sequences \((F_{g})_{g=0}^{\infty}\) of Siegel modular forms, in which \(F_{g}\) has degree \(g\) (and the same weight \(n\) for all \(g\)) and is the image of the next function \(F_{g+1}\) under the Siegel operator. The main aim of this paper is to find those stable modular forms whose common vanishing is the hyper-elliptic locus, in the sense that for every such stable modular form \((F_{g})_{g=0}^{\infty}\), the \(g\)-th function \(F_{g}\) on \(\mathcal{A}_{g}\) (or \(\mathcal{A}_{g}^{S}\)) vanishes on \(\mathrm{Hyp}_{g}\). The result turns out to be differences stable theta series of even, unimodular, positive definite integral quadratic forms of the same rank. Additional analysis of those stable theta series, as well as some results about the stable behavior of the subsets \(\mathrm{Hyp}_{g}\) inside \(\mathcal{A}_{g}\), are also given. We can now describe the organization of the paper, as well as its main results, in more detail. The Introduction in Section 1 includes the basic definitions and states the main results. Section 2 reviews the theory of ind-varieties, as well as the additional properties required for obtaining the structure of a commutative monoid (multiplication, line bundles, splitting, sections, characters, etc.). It also contains a general result, which lies in the heart of the proof of the main theorem: Given a submonoid \(N\) of a commutative monoid \(M\) that is split with respect to a line bundle \(L\), the ideal consisting of those sections of powers of \(L\) that vanish along \(N\) inside the ring of sections of all power of \(L\) over \(M\) is generated by homogenous differences of characters. Section 3 concentrates on the specific ind-variety \(\mathcal{A}_{\infty}\), which is a commutative ind-monoid, and the ring in question is the ring of stable modular forms (with the appropriate choice of \(L\), which is the classical one). The theta series of the quadratic forms with the aforementioned properties are shown to be characters, where the evenness and unimodularity conditions are important since the ind-variety in question is of level 1. Section 4 investigates the hyper-elliptic locus in the Satake compactification, and proves the main result using the tools from Sections 2 and 3 and a result of Freitag about stable modular forms being generated by theta series. An auxiliary result, which is referred to as transversality and is interesting in its own right, is the statement that \(\mathrm{Hyp}_{g}\) is the intersection of \(\mathrm{Hyp}_{g+1}^{S}\) with the indecomposable locus \(\mathcal{A}_{g}^{ind}\) also scheme-theoretically. This is far from trivial, especially since a previous joint paper of the author with his Ph.D. advisor [\textit{G. Codogni} and \textit{N. I. Shepherd-Barron}, Compos. Math. 150, No. 4, 679--690 (2014; Zbl 1304.14037)] shows that an equivalent assertion does not hold for the moduli spaces of curves, and another paper by Shepherd-Barron proves the such an assertion does hold for \(n\)-gonal curves with \(n\geq3\). The proof of the transversality theorem uses an additional (4,8) level structure and a result of Salvati-Manni. Section 5 delves more into the details of the stable theta series involved. Using the Weierstrass identification of \(\mathrm{Hyp}_{g}\) with the moduli space \(\mathcal{B}_{g}^{\circ}\) of \(2g+2\) distinct points in \(\mathcal{P}^{1}\) up to re-orderings and projective maps and some of its extensions, an unpublished criterion of Weissauer is proved. It states that if the inequality \(\frac{n}{k}<8+\frac{4}{g}\) holds and the restriction of a modular form \(F_{g}\) of weight \(n\) and degree \(g\) to \(\mathrm{Hyp}_{g}^{S}\) vanishes to order \(k\) along \(\mathrm{Hyp}_{g-1}\) then \(F_{g}\) vanishes on the larger space \(\mathrm{Hyp}_{g}\). This combines with some results about lattices to obtain the vanishing of differences of theta series of lattices assuming that the rank of the lattices and the minimal non-zero norms in the lattice are \((8,2)\), \((16,2)\), \((24,4)\), \((32,4)\), or \((48,6)\). The first and this case contribute nothing, and the second case produces only the older result of Poor (which reduces to the classical Schottky equation when \(g=4\)), but the last two cases are expected to produce many equations (even though only a few are known explicitly). Finally, Section 6 proves, using a result of Venkov about Niemeier lattices, an extension of the previous result to pairs of rank 24 lattices with the same number of roots (there are 5 such equations). The paper finishes by citing two results of the author's Ph.D. thesis regarding the properties of these differences of theta functions in rank 24 in degrees (or genera) 4 and 5.