Controlled learning of pointwise nonlinearities in neural-network-like architectures (Q6992576)

The subject of this interesting paper is controlled learning of pointwise nonlinearities in neural-network-like architectures.\N\NLet us motivate this idea. Modern computer vision and signal processing algorithms, for example, rely very much on the following types of computational modules. (a) Pointwise nonlinearities, which are typically shared across signal components, and (b) linear transforms. Examples are convolutions, filterbanks, wavelet transforms, and any linear layer of a neural network and pointwise nonlinearities, which are typically shared across signal components.\N\NIt is known that neural networks are being increasingly integrated into signal-processing algorithms, often with substantial performance benefits. This is because neural networks enjoy many of the same fundamental operations as in classic signal processing in the sense that one builds these networks by stacking linear modules (such as the convolutional layers of the network) and (pointwise) nonlinearities known as activations.\N\NThe authors provide a general variational framework for the training of freeform nonlinearities subject to some slope constraints. The regularization that the authors add to the traditional training loss, in fact penalizes the second-order total variation of each trainable activation. The slope constraints allow the authors remarkably to impose properties such as 1-Lipschitz stability, firm non-expansiveness, and monotonicity/invertibility. These properties are crucial to ensure the proper functioning of certain classes of signal-processing algorithms.\N\NThe authors establish rigorously that the global optimum of the stated constrained-optimization problem is achieved with nonlinearities that are adaptive nonuniform linear splines. They also show how to solve the resulting function-optimization problem numerically by representing the nonlinearities in a suitable (nonuniform) B-spline basis. Finally the authors use their framework with the data-driven design of (weakly) convex regularizers for the denoising of certain images and the resolution of certain inverse problems.\N\NThis is an interesting well written paper with an excellent set of references.