Higher-order quasi-Monte Carlo training of deep neural networks
From MaRDI portal
Publication:5015302
DOI10.1137/20M1369373zbMATH Open1492.65043arXiv2009.02713OpenAlexW3216945074MaRDI QIDQ5015302FDOQ5015302
Authors: Marcello Longo, Siddhartha Mishra, T. Konstantin Rusch, Christoph Schwab 
Publication date: 7 December 2021
Published in: SIAM Journal on Scientific Computing (Search for Journal in Brave)
Abstract: We present a novel algorithmic approach and an error analysis leveraging Quasi-Monte Carlo points for training deep neural network (DNN) surrogates of Data-to-Observable (DtO) maps in engineering design. Our analysis reveals higher-order consistent, deterministic choices of training points in the input data space for deep and shallow Neural Networks with holomorphic activation functions such as tanh. These novel training points are proved to facilitate higher-order decay (in terms of the number of training samples) of the underlying generalization error, with consistency error bounds that are free from the curse of dimensionality in the input data space, provided that DNN weights in hidden layers satisfy certain summability conditions. We present numerical experiments for DtO maps from elliptic and parabolic PDEs with uncertain inputs that confirm the theoretical analysis.
Full work available at URL: https://arxiv.org/abs/2009.02713
Recommendations
- Quasi-Monte Carlo sampling for solving partial differential equations by deep neural networks
- Deep learning in high dimension: neural network expression rates for generalized polynomial chaos expansions in UQ
- Deep UQ: learning deep neural network surrogate models for high dimensional uncertainty quantification
- Deep neural network surrogates for nonsmooth quantities of interest in shape uncertainty quantification
- Constructive deep ReLU neural network approximation
machine learningparametric PDEshigh-dimensional approximationdeep neural networkshigher-order quasi-Monte Carlo
Monte Carlo methods (65C05) Artificial neural networks and deep learning (68T07) Numerical integration (65D30) PDEs with randomness, stochastic partial differential equations (35R60) Algorithms for approximation of functions (65D15)
Cites Work
- Reduced basis methods for partial differential equations. An introduction
- Deep learning
- On the distribution of points in a cube and the approximate evaluation of integrals
- Title not available (Why is that?)
- Sparse adaptive approximation of high dimensional parametric initial value problems
- Computational higher order quasi-Monte Carlo integration
- Low-discrepancy point sets obtained by digital constructions over finite fields
- Higher order QMC Petrov-Galerkin discretization for affine parametric operator equations with random field inputs
- Convergence rates of best \(N\)-term Galerkin approximations for a class of elliptic SPDEs
- Title not available (Why is that?)
- High-dimensional integration: The quasi-Monte Carlo way
- Certified reduced basis methods for parametrized partial differential equations
- Breaking the curse of dimensionality in sparse polynomial approximation of parametric PDEs
- Advanced Lectures on Machine Learning
- Higher order quasi-Monte Carlo integration for holomorphic, parametric operator equations
- Solving high-dimensional partial differential equations using deep learning
- Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
- Deep learning observables in computational fluid dynamics
- Optimal approximation with sparsely connected deep neural networks
- Solving the Kolmogorov PDE by means of deep learning
- Machine learning for fast and reliable solution of time-dependent differential equations
- Reconstructing a neural net from its output
- Richardson extrapolation of polynomial lattice rules
- Multilevel higher-order quasi-Monte Carlo Bayesian estimation
- Shape holomorphy of the stationary Navier-Stokes equations
- Domain uncertainty quantification in computational electromagnetics
- Iterative surrogate model optimization (ISMO): an active learning algorithm for PDE constrained optimization with deep neural networks
- Foundations of machine learning
- Deep ReLU networks and high-order finite element methods
- Shape holomorphy of the Calderón projector for the Laplacian in \(\mathbb{R}^2\)
- Higher order quasi-Monte Carlo integration for Bayesian PDE inversion
- Convergence rates of high dimensional Smolyak quadrature
- Deep neural network expression of posterior expectations in Bayesian PDE inversion
- Enhancing Accuracy of Deep Learning Algorithms by Training with Low-Discrepancy Sequences
- A multi-level procedure for enhancing accuracy of machine learning algorithms
- Exponential ReLU DNN expression of holomorphic maps in high dimension
- A quasi-Monte Carlo data compression algorithm for machine learning
Cited In (10)
- Convergence Analysis of a Quasi-Monte CarloBased Deep Learning Algorithm for Solving Partial Differential Equations
- Deep learning based on randomized quasi-Monte Carlo method for solving linear Kolmogorov partial differential equation
- An overview on deep learning-based approximation methods for partial differential equations
- Wavenumber-Explicit Parametric Holomorphy of Helmholtz Solutions in the Context of Uncertainty Quantification
- Numerical analysis of physics-informed neural networks and related models in physics-informed machine learning
- The deep arbitrary polynomial chaos neural network or how deep artificial neural networks could benefit from data-driven homogeneous chaos theory
- Computing ground states of Bose-Einstein condensation by normalized deep neural network
- Adaptive quadratures for nonlinear approximation of low-dimensional PDEs using smooth neural networks
- Quasi-Monte Carlo sampling for solving partial differential equations by deep neural networks
- Greedy training algorithms for neural networks and applications to PDEs
Uses Software
This page was built for publication: Higher-order quasi-Monte Carlo training of deep neural networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5015302)
