Cases of equality in the Riesz rearrangement inequality (Q1918522)

The Riesz rearrangement inequality of the title is the assertion that for nonnegative measurable functions \(f\), \(g\), \(h\) in \(\mathbb{R}^n\), we have \[ \int_{\mathbb{R}^n\times\mathbb{R}^n} f(y)g(x- y)h(x)dx dy\leq \int_{\mathbb{R}^n\times\mathbb{R}^n} f^*(y)g^*(x- y)h^*(x)dx dy, \] where \(*\) denotes symmetric decreasing rearrangement in \(\mathbb{R}^n\). For \(n=1\) the inequality is due to F. Riesz in 1930, and for general \(n\) to Sobolev in 1938. The paper under review is devoted to the identification of the triples \(f\), \(g\), \(h\) for which equality holds. The author gives a complete description when \(f\), \(g\), \(h\) are characteristic functions of sets \(A\), \(B\), and \(C\) of finite positive measure: If the volume radius of one of the sets, say \(C\), is \(\geq\) the sum of the volume radii of \(A\) and \(B\), then equality holds if \(A+B\subset C\), modulo sets of measure zero. But if the volume radius of each of the sets is less than the sum of the volume radii of the other two, then \(A\), \(B\), and \(C\) must be suitable similitudes of a single ellipsoid. The former case follows fairly easily from known results related to the Brunn-Minkowski inequality, but proof of the latter case requires a good deal with inventiveness with symmetrization techniques and some new hands-on criteria for deciding when certain types of sets are ellipsoids. Via ``layer-cake'' representations [see, for example the book ``Analysis'', by \textit{E. H. Lieb} and \textit{M. Loss}, Am. Math. Soc. (1996)] and her characterization in the characteristic function case, the author deduces that for general functions \(f\), \(g\), \(h\), if at least two of the symmetric decreasing rearrangements are strictly decreasing, then, except in trivial cases, equality holds in Riesz's inequality iff there is a measure-preserving linear transformation \(T\) and constants \(a\), \(b\), \(c\) with \(c=a+b\), such that \(f= f^*\circ(T+ a)\), \(g= g^*\circ(T+ b)\), and \(h= h^*\circ(T+ c)\). As a corollary, the author is able to identify the optimizers in a weak-type Young inequality. In the reviewer's opinion, this paper constitutes a very significant contribution to the theory of symmetrization. Some related uniqueness problems are still open. For example, the author notes that her proof does not identify optimizers in the generalization of Riesz's inequality by Brascamp, Lie and Luttinger (1974): \[ \int_{\mathbb{R}^{nk}} \prod^m_{i=1} f_i\Biggl(\sum^k_{j=1} a_{ij}x_j\Biggr) dx_1\cdots dx_k\leq \int_{\mathbb{R}^{nk}} \prod^m_{i=1} f^*_i\Biggl(\sum^k_{j= 1} a_{ij}x_j\Biggr) dx_1\cdots dx_k. \]