Non-unitary representations of nilpotent groups. I: Cohomologies, extensions and neutral cocycles (Q490726)

The paper deals with the general problem of decomposition of non-unitary representations. It is known that general finite-dimensional non-unitary representations do not decompose into the sums of irreducible ones, and so one considers non-decomposable representations as building blocks of non-unitary representations. Let \(H_+,H_-,H=H_++H_-\) be Hilbert spaces, and let \([\cdot,\cdot]=(x_+,y_+)-(x_-,y_-)\) be an ``indefinite scalar product'' on \(H\). If \(\mathrm{dim}H_-=k<\infty\) then this is called a J-unitary representation on Pontryagin \(\Pi_k\)-spaces. A representation of a connected nilpotent Lie group \(G\) on \(H\) is J-unitary if the operators \(\pi(g)\) preserve this scalar product. In fact it is known by work of Naimark that the study of J-unitary representations can be restricted to representations of triangular form \[ \pi(g)= \begin{bmatrix} \lambda(g) & \xi(g) & \gamma(g) \\ 0 & U(g) & \xi(g^{-1})^\ast \\ 0 & 0 & \lambda(g^{-1})^\ast \\ \end{bmatrix} \] where \(H=L\oplus\mathfrak{H}\oplus M\) with \(L,M\) \(k-\) dimensional neutral spaces, \(\mathfrak{H}\) a positive subspace, \(\lambda=\pi|_L\) and \(U\) the unitary representation defined by \(\pi\) in \(\mathfrak{H}\). The basic object of the paper are the extensions \[ \mathfrak{c}(\lambda,U,\xi)=\begin{bmatrix} \lambda & \xi \\ 0 & U \\ \end{bmatrix} \] of a finite-dimensional representation \(\lambda\) of \(G\) by a unitary representation \(U\). These extensions are examined through the connecting 1-cocycles, that is continuous functions \(\xi:G\longrightarrow B(\mathfrak{H},L)\) satisfying \(\xi(gh)=\lambda(g)\xi(h)+\xi(g)U(h)\) for \(g,h\in G\), \(\lambda,U\) representations of \(G\) on Banach spaces \(L,\mathfrak{H}\) respectively. Each connecting 1-cocycle defines a representation \(\pi=\mathfrak{c}(\lambda,U,\xi)\) acting on \(L+\mathfrak{H}\) by \(\pi(g)(x\oplus y)=(\lambda(g)x+\xi(g)y)\oplus U(g)y\). The authors study in particular the space \(\mathcal{H}^1(\lambda,U)\), which is the quotient of 1-cocycles over coboundaries \(\xi\), i.e, the functions satisfying \(\xi(g)=\lambda(g)T-TU(g)\) for some \(T\in B(\mathfrak{H},L)\). It is shown that if \(L\) is finite-dimensional, \(U\) is unitary, and there is \(g\in G\) such that the spectra of \(\lambda(g)\) and \(U(g)\) don't intersect then the spaces \(\mathcal{H}^1\) are trivial. Given this criterion, the authors pass to the study of decompositions of extensions \(\mathfrak{c}(\lambda,U,\xi)\) into a sum of subrepresentations which are not extensions of \(\lambda\). They give sufficient conditions of the decomposability of \(\mathfrak{c}(\lambda,U,\xi)\) and prove that there exists a finite-dimensional subspace \(\mathcal{L}\), containing \(L\) and an infinite sequence of pairs of invariant subspaces \((X_n,Y_n)\) such that \(L\oplus\mathfrak{H}=X_n+Y_n\) for all \(n\). In the next sections, they consider the problem of decomposing \(\mathfrak{c}(\lambda,U,\xi)\) into primary components and prove that such a decomposition always exists for commutative groups while this is not true for all nilpotent groups. The paper ends with a study of neutral cocycles appearing in J-unitary representations of \(\Pi_k\)-spaces. These \((\lambda,U)\)-cocycles \(\xi\) are such that \(\xi(g)\xi(h^{-1})\) is a \((\lambda,\lambda^\sharp)\)-coboundary. It is proved that the set of all neutral 1-cocycles is dense in \(Z^1(\lambda,U)\) if \(\lambda=\iota\) and \(U\) has no fixed vectors.