Controlled learning of pointwise nonlinearities in neural-network-like architectures
Spline approximation (41A15) Algorithms for approximation of functions (65D15) Artificial neural networks and deep learning (68T07) Convex programming (90C25) Applications of mathematical programming (90C90) Image processing (compression, reconstruction, etc.) in information and communication theory (94A08) Signal theory (characterization, reconstruction, filtering, etc.) (94A12)
The subject of this interesting paper is controlled learning of pointwise nonlinearities in neural-network-like architectures.\N\NLet us motivate this idea. Modern computer vision and signal processing algorithms, for example, rely very much on the following types of computational modules. (a) Pointwise nonlinearities, which are typically shared across signal components, and (b) linear transforms. Examples are convolutions, filterbanks, wavelet transforms, and any linear layer of a neural network and pointwise nonlinearities, which are typically shared across signal components.\N\NIt is known that neural networks are being increasingly integrated into signal-processing algorithms, often with substantial performance benefits. This is because neural networks enjoy many of the same fundamental operations as in classic signal processing in the sense that one builds these networks by stacking linear modules (such as the convolutional layers of the network) and (pointwise) nonlinearities known as activations.\N\NThe authors provide a general variational framework for the training of freeform nonlinearities subject to some slope constraints. The regularization that the authors add to the traditional training loss, in fact penalizes the second-order total variation of each trainable activation. The slope constraints allow the authors remarkably to impose properties such as 1-Lipschitz stability, firm non-expansiveness, and monotonicity/invertibility. These properties are crucial to ensure the proper functioning of certain classes of signal-processing algorithms.\N\NThe authors establish rigorously that the global optimum of the stated constrained-optimization problem is achieved with nonlinearities that are adaptive nonuniform linear splines. They also show how to solve the resulting function-optimization problem numerically by representing the nonlinearities in a suitable (nonuniform) B-spline basis. Finally the authors use their framework with the data-driven design of (weakly) convex regularizers for the denoising of certain images and the resolution of certain inverse problems.\N\NThis is an interesting well written paper with an excellent set of references.
- A characterization of proximity operators
- A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems
- A Greedy Approach to $\ell_{0,\infty}$-Based Convolutional Sparse Coding
- A primal-dual splitting method for convex optimization involving Lipschitzian, proximable and linear composite terms
- A representer theorem for deep neural networks
- A survey on modern trainable activation functions
- A wavelet tour of signal processing. The sparse way.
- Adapting to Unknown Smoothness via Wavelet Shrinkage
- An EM algorithm for wavelet-based image restoration
- Analysis of multiresolution image denoising schemes using generalized Gaussian and complexity priors
- Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
- Compressed sensing
- Continuous-Domain Solutions of Linear Inverse Problems With Tikhonov Versus Generalized TV Regularization
- Fields of experts
- Firmly nonexpansive mappings and maximally monotone operators: correspondence and duality
- From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images
- scientific article; zbMATH DE number 3537179 (Why is no real title available?)
- scientific article; zbMATH DE number 2107836 (Why is no real title available?)
- scientific article; zbMATH DE number 3295108 (Why is no real title available?)
- Insights Into Analysis Operator Learning: From Patch-Based Sparse Models to Higher Order MRFs
- Learning Convex Regularizers for Optimal Bayesian Denoising
- Learning weakly convex regularizers for convergent image-reconstruction algorithms
- Locally adaptive regression splines
- Proximal splitting methods in signal processing
- Sparse and redundant representations. From theory to applications in signal and image processing.
- Sparsest piecewise-linear regression of one-dimensional data
- Sparsity and incoherence in compressive sampling
- Spatially adaptive wavelet thresholding with context modeling for image denoising
- Splines are universal solutions of linear inverse problems with generalized TV regularization
- The alternating descent conditional gradient method for sparse inverse problems
- The Lasso problem and uniqueness
- Thresholding estimators for linear inverse problems and deconvolutions
- Variational networks: an optimal control approach to early stopping variational methods for image restoration
This page was built for publication: Controlled learning of pointwise nonlinearities in neural-network-like architectures
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6992576)
