Conjectures P1--P15 for hyperbolic Coxeter groups of rank 3 (Q2214123)

This paper studies the Lusztig \(\mathbf{a}\)-function and the Kazhdan-Lusztig cells of certain weighted hyperbolic Coxeter groups of rank 3. Specifically, these are the Coxeter groups with three generators \(r,s,t\) where \(m_{rt}=2\), \(m_{rs}^{-1}+m_{st}^{-1}<1/2\), and \(\{m_{rs}, m_{st}\}\neq \{\infty\}\). For each such group, the authors explicitly define a certain subset \(D\) of the group and use \(D\) to describe the right and into two-sided cells of the group. The description crucially depends on a function \(\mathbf{a'}: D\rightarrow \mathbb{Z}_{\ge 0}\), which has a simple combinatorial definition but is shown to agree with the \(\mathbf{a}\)-function on \(D\). (Note that to define the \(\mathbf{a}\)-function of a general Coxeter group one needs to assume that a certain boundedness conjecture holds for the group. The conjecture is known to hold for all Coxeter groups of rank 3 and it is recalled in its precise form in the paper). Combined with known facts from the literature, the results of the paper complete the determination of Kazhdan-Lusztig cells for all weighted Coxeter groups of rank 3. The results mentioned above are deduced for the groups under consideration from Lusztig's conjectures ``P1--P15'', the proofs of which occupy a large part of the paper. These well-known conjectures mainly concern the \(\mathbf{a}\)-function, cells and so-called distinguished involution. Following [\textit{X. Xie}, Adv. Math. 379, Article ID 107565, 63 p. (2021; Zbl 1498.20015)], the authors formulate a refinement ``(P1--P15)\({}_{\ge N}\)'' of P1--P15 amenable to proofs by induction on \(\mathbf{a}\)-values, and they reduce the proofs of (P1--P15) \({}_{\ge N}\) to those of three properties about the set \(D\). These properties are stated in Section 3 and proved in Sections 6--8 with careful, extensive case analysis.