On a separation property of positive definite functions on locally compact groups (Q1566392)

Let \(G\) be a locally compact group and let \(P(G)\) be the set of continuous positive definite functions \(\phi\) on \(G\) with \(\phi(0)=1\). One says that \(G\) has the \(H\)-separation property, where \(H\) is a closed subgroup of \(G\), if for every \(x\in G\setminus H\) there exists \(\phi\in P(G)\) which is \(1\) on \(H\) and such that \(\phi(x)\neq 1\). When \(G\) has the \(H\)-property for each closed subgroup \(H\), then \(G\) is called a group with separation property. It is known that every SIN-group \(G\) (i.e. if \(G\) has a basis of invariant neighborhoods \(V\) of the identity then \(V=xVx^{-1}\) for every \(x\in G\)) has the separation property. The main result of a recent paper by the authors [J. Funct. Anal. 175, 89-110 (2000; Zbl 0953.43002)] says that conversely if \(G\) is an almost connected group having the separation property, then \(G\) must be a SIN-group. In this paper, the authors further investigate these separation properties. In particular, under the hypothesis that \(G\) possesses an almost connected open normal subgroup \(M\) and a sequence \(M=H_0\subset H_1\subset\dots\subset H_m=G\) of subgroups \(H_j\) such that \(H_{j-1}\) is normal in \(H_j\) and \(H_j/H_{j-1}={\mathbb Z^{d_j}}\) for some \(d_j\in{\mathbb N}\), \(1\leq j\leq m\), the authors show that \(G\) has the separation property (if and) only if \(G\) is a SIN-group (Theorem 1.7). An application of this theorem yields that if \(G\) is a compactly generated nilpotent group then \(G\) cannot have the separation property unless \(G\) is a SIN-group. Some illustrating examples are also given.