A family quantization formula for symplectic manifolds with boundary (Q1609671)

Let \(M\) be a compact symplectic manifold and let \(L\to M\) be a Hermitian line bundle with a Hermitian connection whose curvature \(\omega\) is the symplectic form on \(M\). For an almost complex structure on \(M\) compatible with \(\omega\) one can define a \(\text{Spin}^c\)-Dirac operator \(D^L\) on \(\Omega^{0,*}(M,L)\). Then the geometric quantization \(Q(M,L)\) of \((M,\omega)\) is the virtual vector space \( Q(M,L) = \Omega^{0,{\text{even}}}\cap\ker D^L - \Omega^{0,{\text{odd}}}\cap\ker D^L\). Now, suppose that a compact connected Lie group \(G\) with Lie algebra \({\mathfrak g}\) acts symplectically on \(L\to M\) and with moment map \(\mu\colon M\to\mathfrak g^*\). If \(0\in\mathfrak g^*\) is a regular value of \(\mu\) and \(G\) acts freely on \(\mu^{-1}(0)\), then \(M_G=\mu^{-1}(0)/G\) is a symplectic manifold, known as the Marsden-Weinstein or symplectic reduction of \(M\) by \(G\) at 0, and it inherits a prequantum line bundle, \(L_G\to M_G\), and the geometric quantization \(Q(M_G,L_G)\) can be constructed. The famous Guillemin-Sternberg conjecture, which has become known as ``quantization commutes with reduction'', or \([Q,R]=0\) theorem, is that \(\dim Q(M,L)^G = \dim Q(M_G,L_G)\), where \(Q(M,L)^G\) is the \(G\)-fixed part of \(Q(M,L)\) which is now a virtual representation of \(G\). The conjecture has been proved by several different methods under various different hypotheses. \textit{V. Guillemin} and \textit{S. Sternberg} [Invent. Math. 67, 515-538 (1982; Zbl 0503.58018)] themselves proved it in a special case when \(M\) is Kähler. \textit{Y. Tian} and \textit{W. Zhang} [Invent. Math. 132, No. 2, 229-259 (1998; Zbl 0944.53047)] gave a direct analytic proof of the conjecture and of some other related results, using a deformation of the \(\text{Spin}^c\)-Dirac operator by using the Clifford action of the Hamilton vector field, of the norm-squared of the moment map, following ideas of Kirwan and Witten. In [Geom. Funct. Anal. 9, No. 3, 596-640 (1999; Zbl 0970.53046)] Tian and Zhang extended their results to the case of symplectic manifolds with boundary and they also gave a general quantization formula that works for both regular and singular reduction. \textit{W. Zhang} [Int. Math. Res. Not. 1999, No. 19, 1043-1056 (1999; Zbl 0939.53044)] generalized and proved the Guillemin-Sternberg conjecture for the family of fibers of a smooth fibration. The proof combined the analytical approach developed by Y. Tian and W. Zhang in the previous papers with the homotopy invariance property of the index bundle. In this paper, the authors generalize the previous results on the family quantization formula to the case of symplectic manifolds with boundary. If \(Z\to M\overset{\pi}{\to}B\) is a smooth fibration with compact connected fibers with boundary \(Z\) and compact base \(B\) and \(G\) acts fiberwise on \(M\) in a Hamiltonian way, then the corresponding index bundle of the fiberwise \(\text{Spin}^c\)-Dirac operator is an element in the \(K\)-group \(K(B)\). The authors obtain a version of the geometric quantization conjecture with values in \(K(B)\).