Parseval proximal neural networks

DOI10.1007/S00041-020-09761-7zbMATH Open1489.68224arXiv1912.10480OpenAlexW2994632629MaRDI QIDQ785901FDOQ785901

Authors: Marzieh Hasannasab, Johannes Hertrich, Sebastian Neumayer, S. Setzer, Gerlind Plonka, Gabriele Steidl 

Publication date: 12 August 2020

Published in: The Journal of Fourier Analysis and Applications (Search for Journal in Brave)

Abstract: The aim of this paper is twofold. First, we show that a certain concatenation of a proximity operator with an affine operator is again a proximity operator on a suitable Hilbert space. Second, we use our findings to establish so-called proximal neural networks (PNNs) and stable tight frame proximal neural networks. Let $mathcalH$ and $mathcalK$ be real Hilbert spaces, $binmathcalK$ and $TinmathcalB(mathcalH,mathcalK)$ have closed range and Moore-Penrose inverse $T^{d}agger$. Based on the well-known characterization of proximity operators by Moreau, we prove that for any proximity operator $extProxcolonmathcalKomathcalK$ the operator $T^{d}agger,extProx(Tcdot+b)$ is a proximity operator on $mathcalH$ equipped with a suitable norm. In particular, it follows for the frequently applied soft shrinkage operator $extProx=S_{lambda}colonel{l}_{2}ightarrowel{l}_2$ and any frame analysis operator $TcolonmathcalHoel{l}_2$ that the frame shrinkage operator $T^{d}agger,S_{l}ambda,T$ is a proximity operator on a suitable Hilbert space. The concatenation of proximity operators on $mathbb{R}^d$ equipped with different norms establishes a PNN. If the network arises from tight frame analysis or synthesis operators, then it forms an averaged operator. Hence, it has Lipschitz constant 1 and belongs to the class of so-called Lipschitz networks, which were recently applied to defend against adversarial attacks. Moreover, due to its averaging property, PNNs can be used within so-called Plug-and-Play algorithms with convergence guarantee. In case of Parseval frames, we call the networks Parseval proximal neural networks (PPNNs). Then, the involved linear operators are in a Stiefel manifold and corresponding minimization methods can be applied for training. Finally, some proof-of-the concept examples demonstrate the performance of PPNNs.

Full work available at URL: https://arxiv.org/abs/1912.10480

Recommendations

• Deep neural network structures solving variational inequalities
• NETT: solving inverse problems with deep neural networks
• Nonlinear Power Method for Computing Eigenvectors of Proximal Operators and Neural Networks
• Proximal splitting methods in signal processing
• Proximity for sums of composite functions

zbMATH Keywords

proximal operatorsframe shrinkageadverserial robustnessaveraged operatorsLipschitz neural networksoptimization on Stiefel manifolds

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Nonconvex programming, global optimization (90C26)

Cites Work

• A feasible method for optimization with orthogonality constraints
• Convex analysis and monotone operator theory in Hilbert spaces
• First-order methods in optimization
• Title not available (Why is that?)
• Title not available (Why is that?)
• Signal Recovery by Proximal Forward-Backward Splitting
• An iterative thresholding algorithm for linear inverse problems with a sparsity constraint
• Proximité et dualité dans un espace hilbertien
• Weak convergence theorems for nonexpansive mappings in Banach spaces
• Operator splittings, Bregman methods and frame shrinkage in image processing
• Variable metric forward-backward algorithm for minimizing the sum of a differentiable function and a convex function
• Incremental proximal methods for large scale convex optimization
• On the Equivalence of Soft Wavelet Shrinkage, Total Variation Diffusion, Total Variation Regularization, and SIDEs
• First order algorithms in variational image processing
• An introduction to frames and Riesz bases
• A MULTISCALE WAVELET-INSPIRED SCHEME FOR NONLINEAR DIFFUSION
• Monotone operator theory in convex optimization
• On the rotational invariant \(L_1\)-norm PCA
• A Convergent Image Fusion Algorithm Using Scene-Adapted Gaussian-Mixture-Based Denoising

Cited In (21)

• Designing stable neural networks using convex analysis and ODEs
• Variational models for signal processing with graph neural networks
• Motion detection in diffraction tomography by common circle methods
• Inertial stochastic PALM and applications in machine learning
• Approximation of Lipschitz Functions Using Deep Spline Neural Networks
• Convergence of deep convolutional neural networks
• Deep solution operators for variational inequalities via proximal neural networks
• NESTANets: stable, accurate and efficient neural networks for analysis-sparse inverse problems
• Sparse additive function decompositions facing basis transforms
• On \(\alpha\)-firmly nonexpansive operators in \(r\)-uniformly convex spaces
• Proximal neural networks and stochastic normalizing flows for inverse problems
• Convolutional proximal neural networks and plug-and-play algorithms
• A Bregman stochastic method for nonconvex nonsmooth problem beyond global Lipschitz gradient continuity
• Compressive sensing and neural networks from a statistical learning perspective
• Frame soft shrinkage operators are proximity operators
• Neural-network-based regularization methods for inverse problems in imaging
• Stabilizing Invertible Neural Networks Using Mixture Models
• Robustness and exploration of variational and machine learning approaches to inverse problems: an overview
• Designing rotationally invariant neural networks from PDEs and variational methods
• Stochastic Gauss-Seidel type inertial proximal alternating linearized minimization and its application to proximal neural networks
• Learning weakly convex regularizers for convergent image-reconstruction algorithms

This page was built for publication: Parseval proximal neural networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q785901)