Approximation of Lipschitz Functions Using Deep Spline Neural Networks
From MaRDI portal
Abstract: Lipschitz-constrained neural networks have many applications in machine learning. Since designing and training expressive Lipschitz-constrained networks is very challenging, there is a need for improved methods and a better theoretical understanding. Unfortunately, it turns out that ReLU networks have provable disadvantages in this setting. Hence, we propose to use learnable spline activation functions with at least 3 linear regions instead. We prove that this choice is optimal among all component-wise -Lipschitz activation functions in the sense that no other weight constrained architecture can approximate a larger class of functions. Additionally, this choice is at least as expressive as the recently introduced non component-wise Groupsort activation function for spectral-norm-constrained weights. Previously published numerical results support our theoretical findings.
Recommendations
- Error bounds for approximations with deep ReLU networks
- Regularisation of neural networks by enforcing Lipschitz continuity
- Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations
- Provable approximation properties for deep neural networks
- Spline representation and redundancies of one-dimensional ReLU neural network models
Cites work
- A representation method for PWL functions oriented to parallel processing
- A representer theorem for deep neural networks
- CLIP: cheap Lipschitz training of neural networks
- Convolutional proximal neural networks and plug-and-play algorithms
- Deep learning architectures. A mathematical approach
- Deep Neural Networks With Trainable Activations and Controlled Lipschitz Constant
- Higher-order total variation approaches and generalisations
- Parseval proximal neural networks
- Regularisation of neural networks by enforcing Lipschitz continuity
- Stabilizing Invertible Neural Networks Using Mixture Models
Cited in
(6)- Regularisation of neural networks by enforcing Lipschitz continuity
- Stable parameterization of continuous and piecewise-linear functions
- On the number of regions of piecewise linear neural networks
- Learning weakly convex regularizers for convergent image-reconstruction algorithms
- ABBA neural networks: coping with positivity, expressivity, and robustness
- Data-driven regularization with weak convexity for robust image reconstruction
This page was built for publication: Approximation of Lipschitz Functions Using Deep Spline Neural Networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6104312)
