Better approximations of high dimensional smooth functions by deep neural networks with rectified power units
From MaRDI portal
Publication:5162006
DOI10.4208/CICP.OA-2019-0168zbMATH Open1474.65031arXiv1903.05858MaRDI QIDQ5162006FDOQ5162006
Authors: Bo Li, Shanshan Tang, Haijun Yu 
Publication date: 1 November 2021
Published in: Communications in Computational Physics (Search for Journal in Brave)
Abstract: Deep neural networks with rectified linear units (ReLU) are getting more and more popular due to their universal representation power and successful applications. Some theoretical progress regarding the approximation power of deep ReLU network for functions in Sobolev space and Korobov space have recently been made by [D. Yarotsky, Neural Network, 94:103-114, 2017] and [H. Montanelli and Q. Du, SIAM J Math. Data Sci., 1:78-92, 2019], etc. In this paper, we show that deep networks with rectified power units (RePU) can give better approximations for smooth functions than deep ReLU networks. Our analysis bases on classical polynomial approximation theory and some efficient algorithms proposed in this paper to convert polynomials into deep RePU networks of optimal size with no approximation error. Comparing to the results on ReLU networks, the sizes of RePU networks required to approximate functions in Sobolev space and Korobov space with an error tolerance , by our constructive proofs, are in general times smaller than the sizes of corresponding ReLU networks constructed in most of the existing literature. Comparing to the classical results of Mhaskar [Mhaskar, Adv. Comput. Math. 1:61-80, 1993], our constructions use less number of activation functions and numerically more stable, they can be served as good initials of deep RePU networks and further trained to break the limit of linear approximation theory. The functions represented by RePU networks are smooth functions, so they naturally fit in the places where derivatives are involved in the loss function.
Full work available at URL: https://arxiv.org/abs/1903.05858
Recommendations
- PowerNet: efficient representations of polynomials and smooth functions by deep neural networks with rectified power units
- Approximation of smoothness classes by deep rectifier networks
- Error bounds for approximations with deep ReLU networks
- Nonlinear approximation and (deep) ReLU networks
- Optimal approximation of piecewise smooth functions using deep ReLU neural networks
sparse gridsdeep neural networkhigh dimensional approximationrectified linear unitrectified power unitrectified quadratic unit
Artificial neural networks and deep learning (68T07) Algorithms for approximation of functions (65D15) Stability and convergence of numerical methods for initial value and initial-boundary value problems involving PDEs (65M12) Error bounds for initial value and initial-boundary value problems involving PDEs (65M15)
Cites Work
- Numerical integration using sparse grids
- Spectral methods. Algorithms, analysis and applications.
- A logical calculus of the ideas immanent in nervous activity
- When are quasi-Monte Carlo algorithms efficient for high dimensional integrals?
- A Sparse Grid Stochastic Collocation Method for Partial Differential Equations with Random Input Data
- Approximation by superpositions of a sigmoidal function
- High dimensional polynomial interpolation on sparse grids
- Title not available (Why is that?)
- A sparse finite element method with high accuracy. I
- Optimal nonlinear approximation
- Sparse spectral approximations of high-dimensional problems based on hyperbolic cross
- Sparse grids for the Schrödinger equation
- A Fast Learning Algorithm for Deep Belief Nets
- Sparse finite elements for stochastic elliptic problems --- higher order moments
- Efficient spectral sparse grid methods and applications to high-dimensional elliptic problems
- A nonhomogeneous kinetic model of liquid crystal polymers and its thermodynamic closure approximation
- Error analysis of the dynamically orthogonal approximation of time dependent random PDEs
- Approximation by Ridge Functions and Neural Networks
- Why Are High-Dimensional Finance Problems Often of Low Effective Dimension?
- The hyperbolic cross space approximation of electronic wavefunctions
- Optimally scaled and optimally conditioned vandermonde and Vandermonde-like matrices
- Approximation properties of a multilayered feedforward artificial neural network
- On the expressive power of deep architectures
- Solving high-dimensional partial differential equations using deep learning
- Model reduction with memory and the machine learning of dynamical systems
- A dynamic-solver-consistent minimum action method: with an application to 2D Navier-Stokes equations
- Efficient spectral-element methods for the electronic Schrödinger equation
- An adaptive sparse grid algorithm for elliptic PDEs with lognormal diffusion coefficient
- The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems
- Optimal approximation of piecewise smooth functions using deep ReLU neural networks
- Error bounds for approximations with deep ReLU networks
- Neural Networks for Localized Approximation
- Title not available (Why is that?)
- Sparse grid discontinuous Galerkin methods for high-dimensional elliptic equations
- Efficient spectral sparse grid methods and applications to high-dimensional elliptic equations II. Unbounded domains
- Deep Potential: A General Representation of a Many-Body Potential Energy Surface
- Approximations by orthonormal mapped Chebyshev functions for higher-dimensional problems in unbounded domains
- Exponential convergence of the deep neural network approximation for analytic functions
- Deep ReLU networks and high-order finite element methods
- ReLU deep neural networks and linear finite elements
- New error bounds for deep ReLU networks using sparse grids
- A nodal sparse grid spectral element method for multi-dimensional elliptic partial differential equations
- Data-driven, physics-based feature extraction from fluid flow fields using convolutional neural networks
- PowerNet: efficient representations of polynomials and smooth functions by deep neural networks with rectified power units
- Finding the Origin of Noise Transients in LIGO Data with Machine Learning
Cited In (21)
- Convergence rate analysis for deep Ritz method
- Constructive deep ReLU neural network approximation
- Approximation rates for neural networks with encodable weights in smoothness spaces
- Solving parametric partial differential equations with deep rectified quadratic unit neural networks
- Discovering phase field models from image data with the pseudo-spectral physics informed neural networks
- Exponential ReLU DNN expression of holomorphic maps in high dimension
- A functional equation with polynomial solutions and application to neural networks
- Efficient spectral methods for quasi-equilibrium closure approximations of symmetric problems on unit circle and sphere
- Optimal approximation of infinite-dimensional holomorphic functions
- The robust physics-informed neural networks for a typical fourth-order phase field model
- A convergent deep learning algorithm for approximation of polynomials
- Two-layer networks with the \(\text{ReLU}^k\) activation function: Barron spaces and derivative approximation
- Integral representations of shallow neural network with Rectified Power Unit activation function
- Neural Control of Parametric Solutions for High-Dimensional Evolution PDEs
- Solving Allen-Cahn and Cahn-Hilliard Equations using the Adaptive Physics Informed Neural Networks
- Sparse approximation of triangular transports. I: The finite-dimensional case
- Approximation results for gradient flow trained shallow neural networks in \(1d\)
- PowerNet: efficient representations of polynomials and smooth functions by deep neural networks with rectified power units
- High Order Deep Neural Network for Solving High Frequency Partial Differential Equations
- Neural networks with ReLU powers need less depth
- Approximation of smoothness classes by deep rectifier networks
Uses Software
This page was built for publication: Better approximations of high dimensional smooth functions by deep neural networks with rectified power units
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5162006)
