Continuous family of invariant subspaces for \(R\)-diagonal operators (Q1608543)

The main result of the paper under review says that every \(R\)-diagonal operator has the upper triangular form property. Every circular operator in the sense of D. Voiculescu's free probability theory is in particular \(R\)-diagonal, so that the results of the paper by \textit{K. Dykema} and \textit{U. Haagerup} [Geom. Funct. Anal. 11, 693-741 (2001; Zbl 1023.47005)] are improved in the present paper. To describe in more detail the contents of the paper, we mention that its background is twofold: firstly, the invariant subspace problem relative to a von Neumann algebra \({\mathcal A}\) (i.e., whether or not for every operator \(x\in{\mathcal A}\) there exists an orthogonal projection \(p\in{\mathcal A}\) such that \(0\neq p\neq 1\) and the range of \(p\) is invariant under \(x\)). The second theme of the paper is free probability theory, where the \(R\)-diagonal operators were introduced. The freeness is a noncommutative version of the idea of independence from classical probability theory. A scalar (noncommutative) probability space is a pair \(({\mathcal A},\varphi)\), where \({\mathcal A}\) is a unital \(W^*\)-algebra and \(\varphi\colon{\mathcal A}\to{\mathbb C}\) is a normal tracial faithful state of \({\mathcal A}\). An operator-valued probability space is a similar object, where \({\mathbb C}\) is replaced by a unital subalgebra \({\mathcal D}\) of \({\mathcal A}\), while the state \(\varphi\) is replaced by a conditional expectation \(E\colon{\mathcal A}\to{\mathcal D}\). An element \(a\) of a (scalar or operator-valued) probability space is \(R\)-diagonal with amalgamation over \({\mathcal D}\) (or simply \(R\)-diagonal if \({\mathcal D}={\mathbb C}\)) if, roughly speaking, for every enlarged probability space defined by the data \((\widetilde{\mathcal D}\subseteq\widetilde{\mathcal A},\widetilde{E})\) and every unitary \(u\in\widetilde{\mathcal A}\) the elements \(a\) and \(ua\) are identically \({\mathcal D}\)-distributed provided the sets \(\{a,a^*\}\subseteq{\mathcal A}\subseteq\widetilde{\mathcal A}\) and \(\{u,u^*\}\subseteq\widetilde{\mathcal A}\) are free. Several characterizations of this concept are described in Theorem 3 of the paper. As we mentioned at the very beginning, it is based in fact on an extension of the notion of circular operator that can be found in the book by \textit{D. V. Voiculescu, K. J. Dykema} and \textit{A. Nica} [``Free random variables. A noncommutative probability approach to free products with applications to random matrices, operator algebras and harmonic analysis on free groups'' (CRM Monograph Series 1, American Mathematical Society, Providence) (1992; Zbl 0795.46049)]. Instead of the definition of the latter notion, we just recall that every circular operator generates the \(\text{II}_1\) factor \(\text{L}({\mathbb F}_2)\) corresponding to the free group \({\mathbb F}_2\) with two generators. We should point out the new characterization of \(R\)-diagonality in terms of freeness that can be found in Theorem 4 of the present paper. We now pass to the other theme of the paper, i.e., the invariant subspace problem relative to a von Neumann algebra. To this end, we briefly describe the upper triangular form property referred to in the main result of the paper (see the very beginning of the present review). For a Hilbert space operator \(x\) and a nonnegative real number \(s\), denote by \(V_s\) the closure of the set of all vectors \(v\) such that the local spectral radius of \(x\) at \(v\) is at most \(s\), that is, \(\limsup_{k\to\infty}\|x^k\|^{1/k}\leq s\). Then the closed subspace \(V_s\) is invariant under the commutant of \(x\), hence the orthogonal projection \(p_s\) on \(V_s\) belongs to the von Neumann algebra generated by \(x\). Now, if \(0=s_0\leq s_1<s_2<\cdots<s_{n-1}\), then \(V_{s_1}\subseteq V_{s_2}\subseteq\cdots\subseteq V_{s_n}\) is a nest of invariant subspaces for \(x\), thus leading to a representation of \(x\) as a triangular operator matrix. If the spectrum of the \(i\)-th diagonal entry of that triangular matrix is contained in the set \(\{z\in{\mathbb C}\mid s_{i-1}\leq|z|\leq s_i\}\) for \(i=1,2,\dots,n\) whenever \(s_0,s_1,\dots,s_n\) are as above and \(n\) is an arbitrary positive integer, then we say that \(x\) has the upper triangular form property. This is just a property that every \(R\)-diagonal operator has, according to the main result of the paper under review (Theorem 11). Towards the end of the paper, a couple of significant applications of this fact are included. Firstly, one recovers a result of \textit{U. Haagerup} and \textit{F. Larsen} [J. Funct. Anal. 176, 331-367 (2000; Zbl 0984.46042)] that explicitly describes the Brown spectral distribution measure (introduced in the paper by \textit{L. G. Brown} [in: Geometric methods in operator algebras, Proc. US-Jap. Semin., Kyoto/Jap. 1983 (Pitman Res. Notes Math. Ser. 123) 1-35 (1986; Zbl 0646.46058)]) of \(R\)-diagonal operators. As a second application, a new proof is presented for the fact that D. Voiculescu's \(S\)-transform is multiplicative with respect to the free multiplicative convolution of probability measures on \([0,\infty)\). The paper concludes with a section discussing an asymptotic freeness conjecture that might provide a setting where Brown measures of a sequence of random matrices converge weakly to the Brown measure of the limit distribution.