Euler products of Selberg zeta functions in the critical strip (Q2675252)

Suppose \(\Gamma\) is a congruence subgroup of the modular group, and let \(\rho\) be a finite-dimensional unitary representation of \(\Gamma.\) The main result of the paper under review is to extend the region of convergence of the Euler products of the Selberg zeta functions beyond the boundary \(\Re( s) = 1\), if they are attached with a nontrivial irreducible unitary representation. The region is determined by the size of the lowest eigenvalue of the Laplacian, and it extends to \(\Re s \geqslant 3/4\) under Selberg's eigenvalue conjecture. In addition they establish a relation between the behavior of partial Euler products in the critical strip and the estimate of the error term in the prime geodesic theorem. Let \(\zeta(s)\) denote the Riemann zeta function. If the Riemann Hypothesis (RH) for \(\zeta(s)\) is assumed, then we have (Mertens, Ramanujan) \[ \prod_{p \leqslant x} (1-p^{-s})^{-1} = -\zeta(s) \exp \left(\mathrm{Li}(\vartheta(x)^{1-s})+\frac{2s x^{\frac{1}{2}-s}}{(2s-1) \log x} + \frac{S_{s}(x)}{\log x}+O \left(\frac{x^{\frac{1}{2}-s}}{(\log x)^{2}} \right) \right)\tag{1} \] in \(1/2 < s < 1\), where \(\vartheta(x) = \sum \limits_{p \leqslant x} \log p\) is Tchebyshev's function and \[ S_{s}(x) = -s \sum_{\zeta(\rho) = 0, \, \Re \rho = \frac{1}{2}} \frac{x^{\rho-s}}{\rho(\rho-s)}.\tag{2} \] Let \(\Gamma \subset \mathrm{PSL}(2, \mathbb{R})\) be a cofinite Fuchsian group acting discontinuously on the upper half-plane \(\mathbb{H}\). There is a lucid analogy between prime numbers and primitive (prime) closed geodesics on \(\Gamma \backslash \mathbb{H}\). Any matrix in a hyperbolic conjugacy class in \(\Gamma\) is conjugate over \(\mathrm{SL}(2, \mathbb{R})\) to a matrix of the form \(\begin{pmatrix} \lambda && 0 \\ 0 && \lambda^{-1} \end{pmatrix}\) with \(\lambda > 1\). We then denote the norm of such a hyperbolic conjugacy class to be \(N(p) = \lambda^{2}\). The Selberg zeta function of \(\Gamma\) attached with its finite-dimensional unitary representation \(\rho\) is defined as the following Euler product: \[ Z_{\Gamma}(s, \rho) = \prod_{p} \prod_{n = 0}^{\infty} \det \left(I_{\dim \rho}-\rho(p) N(p)^{-s-n} \right), \] Denote \[ \Theta_{\Gamma}(x, \rho) = \sum_{N(p) \leqslant x} \mathrm{tr}(\rho(p)) \log N(p) \] and \[ \Psi_{\Gamma}(x, \rho) = \sum_{N(p)^{k} \leqslant x} \mathrm{tr}(\rho(p)^{k}) \log N(p), \] where \(\sum_{N(p)^{k} \leqslant x}\) stands for the sum over all pairs of \((p, k)\) with \(p\) as above and positive integers \(k\) such that \(N(p)^{k} \leqslant x\). Let \[ \begin{aligned} Z_{\Gamma, x}(s, \rho)& = \prod_{N(p) \leqslant x} \prod_{n = 0}^{\infty} \det(I_{\dim \rho}-\rho(p) N(p)^{-s-n}), \\ \zeta_{\Gamma, x}(s, \rho)& = \prod_{N(p) \leqslant x} \det(I_{\dim \rho}-\rho(p) N(p)^{-s})^{-1}. \end{aligned}\tag{3} \] Finally let \(\lim_{x \to \infty} \zeta_{\Gamma, x}(s, \rho) = \zeta_{\Gamma}(s, \rho) = Z_{\Gamma}(s+1, \rho)/Z_{\Gamma}(s, \rho)\) for \(\Re s > 1.\) The two main results are: Theorem. Let \(\Gamma\) be an arithmetic group. For \(s = \sigma+it\) with \(\sigma > 1/2\) we then have \[ \zeta_{\Gamma, x}(s) = \varepsilon_{\Gamma}(s) \, \zeta_{\Gamma}(s) \exp \bigg(\mathrm{Li}(\Theta_{\Gamma}(x)^{1-s}) + \sum_{\frac{1}{2} < s_{j} < 1} \mathrm{Li}(x^{s_{j}-s}) - \frac{x^{-s}}{\log x} \sum_{\frac{1}{2} < s_{j} < 1} \frac{x^{s_{j}}}{s_{j}}+O(x^{\frac{1}{2}-\sigma}(\log x)^{2}) \bigg), \] where \(\varepsilon_{\Gamma}(s) = \pm 1\) and Theorem. Let \(\Gamma\) be an arithmetic group and \(\rho\) any unitary representation (not necessarily nontrivial). For \(\Re(s) > 1/2\) we have \[ \zeta_{\Gamma, x}(s, \rho) = \varepsilon_{\Gamma}(s, \rho) \, \zeta_{\Gamma}(s, \rho) \times \exp \bigg(\sum_{\frac{1}{2} < s_{j} \leqslant 1}\mathrm{Li}(x^{s_{j}-s}) + \frac{x^{\frac{1}{2}-s}}{\log x} \sum_{|t_{j}| \leqslant T}\frac{x^{it_{j}}}{s_{j}}+O \left(\frac{x^{1-\sigma}}{T} \log x \right) \bigg) \] provided \(1 \leqslant T \leqslant \sqrt{x}/\log x\), where \(\varepsilon_{\Gamma}(s, \rho) = \pm 1\) (see [\textit{K. Conrad}, Can. J. Math. 57, No. 2, 267--297 (2005; Zbl 1068.11058), Theorem 3.3]).