Localization and the quantization conjecture (Q2564631)

Let \(M\) be a compact quantizable symplectic manifold acted on in a Hamiltonian fashion by a compact connected Lie group \(K\) preserving the structure and suppose that \(0\) is a regular value of the moment map so that the reduced space \(M_{\text{red}}\) acquires a structure of a symplectic orbifold. Pick a compatible almost complex structure on \(M\) --- this is always possible; it descends to a compatible almost complex structure on \(M_{\text{red}}\). Let \(\mathcal L\) be a \(K\)-equivariant prequantum bundle for \(M\), the \(K\)-action being compatible with the moment map, and denote by \(\mathcal L_{\text{red}}\) the corresponding orbifold line bundle on the reduced space. When the almost complex structure is a genuine complex structure and \(\mathcal L\) a holomorphic line bundle, geometric quantization endows the cohomology groups \(\text H^j(M,\mathcal L)\) with structures of representation spaces for \(K\), and the Riemann-Roch numbers \(RR^K(\mathcal L)\) and \(RR(\mathcal L_{\text{red}})\) are defined as usual by \(RR^K(\mathcal L) = \sum (-1)^j \dim \text H^j(M,\mathcal L)^K\) and \(RR(\mathcal L_{\text{red}}) =\sum (-1)^j \dim \text H^j(M_{\text{red}},\mathcal L_{\text{red}})\). When the structure is only almost complex, Riemann-Roch numbers \(RR^K(\mathcal L)\) and \(RR(\mathcal L_{\text{red}})\) can still be defined by means of a direct image construction in \(K\)-theory involving a Spin\(^c\) Dirac operator, cf. [\textit{J. J. Duistermaat}, The heat kernel Lefschetz fixed point formula for the Spin\(^c\) Dirac operator (1996; Zbl 0858.58045); \textit{V. Guillemin}, Progr. Math. 123, 305-334 (1994; Zbl 0869.58017); \textit{M. Vergne}, Duke Math. J. 82, No. 1, 143-179 (1996; Zbl 0855.58033); 181-194 (1996; Zbl 0855.58034)]. This direct image construction has been used already by \textit{M. F. Atiyah} and \textit{F. Hirzebruch} in the proof of the differentiable Riemann-Roch theorem [Bull. Am. Math. Soc. 65, 276-281 (1959; Zbl 0142.40901)]; it involves, in the smooth context, a suitable Dirac operator as a replacement for the algebro-geometric Dolbeault operator, cf. e. g. D.18 in the book of \textit{H. B. Lawson jun.} and \textit{M.-L. Michelsohn} [Spin geometry, Princeton Math. Ser. 38 (1989; Zbl 0688.57001)]. What is referred to as the quantization conjecture in the title is the statement that, under appropriate circumstances, the two Riemann-Roch numbers coincide. In [Invent. Math. 67, 515-538 (1982; Zbl 0503.58018)] \textit{V. Guillemin} and \textit{S. Sternberg} verified equality of the two numbers under the additional hypothesis that \(M\) be a positive Kähler manifold and that the line bundle \(\mathcal L\) be holomorphic and sufficiently positive. There are counterexamples showing that the quantization conjecture cannot hold in full generality. In the paper under review, which extends earlier results of the authors for the special case where \(K\) has rank one [Q. J. Math., Oxf. II. Ser. 47, No. 186, 165-185 (1996; Zbl 0870.53022)], the quantization conjecture is shown to hold in many circumstances including more or less those of the cited result of V. Guillemin and S. Sternberg. (More precisely, one of the sufficient hypotheses in the paper says that the power of the requisite line bundle coming into play should be sufficiently high but it is not clear whether the given bound coincides with the bound required in the Guillemin-Sternberg result.) Among the tools is the authors' residue formula [Topology 34, No. 2, 291-327 (1995; Zbl 0833.55009)], applied to the Todd class of \(M_{\text{red}}\), the Todd class being given by Kawasaki's Riemann-Roch formula for orbifolds when \(K\) does not act freely on the zero locus. This suffices when \(M\) is actually a complex manifold; in the more general case where \(M\) carries only an almost complex structure, \textit{M. Vergne}'s extension [Duke Math. J. 82, No. 3, 637-652 (1996; Zbl 0874.57029)] of Kawasaki's Riemann-Roch theorem is used, with the appropriate \(K\)-theory interpretation of the Riemann-Roch numbers. A formula for the \(K\)-invariant Riemann-Roch number is obtained by means of the holomorphic Lefschetz formula, which is a special case of the Atiyah-Segal-Singer equivariant index formula. The resulting formulas for the two Riemann-Roch numbers are then shown to coincide in many circumstances. Several other papers have recently appeared or will appear which extend the Guillemin-Sternberg result, and in which the main tool is localization. These include the cited papers of Jeffrey and Kirwan [loc. cit.], of V. Guillemin [loc. cit.], and of M. Vergne [Duke Math. J. 82, 143-179, 181-194 (1996), loc. cit.], and \textit{E. Meinrenken} [J. Am. Math. Soc. 9, No. 2, 373-389 (1996; Zbl 0851.53020)]. By means of a different method based on \textit{E. Lerman}'s technique of symplectic cutting [Math. Res. Lett. 2, No. 3, 247-258 (1995; Zbl 0835.53034)], \textit{H. Duistermaat, V. Guillemin, E. Meinrenken} and \textit{S. Wu} proved a version of the quantization conjecture for circle actions in [ibid. 2, No. 3, 259-266 (1995; Zbl 0839.58026)]. The same technique enabled \textit{E. Meinrenken} [Adv. Math. (to appear)] to establish the quantization conjecture for arbitrary nonabelian groups provided the almost complex structure is positive with respect to the symplectic structure. In the paper under review, positivity is not assumed and the quantization conjecture is shown to hold in some situations where the almost complex structure is not necessarily positive, for \(G\) a torus and for certain nonabelian groups. Essentially the same result for the case of a torus action may be found in the already cited papers [Duke Math. J. 82, 143-179; 181-194 (1996)] of \textit{M. Vergne}. The known counterexamples to the quantization conjecture are in fact for the case of an action of a nonabelian group and non-positive almost complex structure. In a somewhat different direction, the Guillemin-Sternberg result has been extended by \textit{R. Sjamaar} [Ann. Math., II. Ser. 141, No. 1, 87-129 (1995 Zbl 0827.32030)] for the case where \(0\) is not a regular value, under certain additional hypotheses which ensure that the reduced space acquires the structure of a Kähler quotient. A generalization thereof, based on the technique of symplectic cutting, has recently been obtained by E. Meinrenken and R. Sjamaar [dg-ga/9707023] for genuinely singular quotients which are not necessarily Kähler. A leisurely introduction to these and related matters may be found in \textit{R. Sjamaar}'s survey article [Bull. Am. Math. Soc., New Ser. 33, No. 3, 327-338 (1996; Zbl 0857.58021)].