Ensemble Kalman inversion: a derivative-free technique for machine learning tasks
From MaRDI portal
Publication:5197869
DOI10.1088/1361-6420/AB1C3AzbMATH Open1430.68266arXiv1808.03620OpenAlexW2949400887WikidataQ127968194 ScholiaQ127968194MaRDI QIDQ5197869FDOQ5197869
Authors: Nikola B. Kovachki, A. M. Stuart 
Publication date: 20 September 2019
Published in: Inverse Problems (Search for Journal in Brave)
Abstract: The standard probabilistic perspective on machine learning gives rise to empirical risk-minimization tasks that are frequently solved by stochastic gradient descent (SGD) and variants thereof. We present a formulation of these tasks as classical inverse or filtering problems and, furthermore, we propose an efficient, gradient-free algorithm for finding a solution to these problems using ensemble Kalman inversion (EKI). Applications of our approach include offline and online supervised learning with deep neural networks, as well as graph-based semi-supervised learning. The essence of the EKI procedure is an ensemble based approximate gradient descent in which derivatives are replaced by differences from within the ensemble. We suggest several modifications to the basic method, derived from empirically successful heuristics developed in the context of SGD. Numerical results demonstrate wide applicability and robustness of the proposed algorithm.
Full work available at URL: https://arxiv.org/abs/1808.03620
Recommendations
- Ensemble Kalman filter for neural network-based one-shot inversion
- Convergence acceleration of ensemble Kalman inversion in nonlinear settings
- Well posedness and convergence analysis of the ensemble Kalman inversion
- Gradient flow structure and convergence analysis of the ensemble Kalman inversion for nonlinear forward models
- Adaptive Tikhonov strategies for stochastic ensemble Kalman inversion
deep learningmachine learningderivative-free optimizationensemble Kalman filteringensemble Kalman inversion
Bayesian inference (62F15) Learning and adaptive systems in artificial intelligence (68T05) Artificial neural networks and deep learning (68T07) Signal detection and filtering (aspects of stochastic processes) (60G35)
Cites Work
- Title not available (Why is that?)
- AlexNet
- Reducing the Dimensionality of Data with Neural Networks
- Gaussian processes for machine learning.
- Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies?
- Title not available (Why is that?)
- Title not available (Why is that?)
- Ridge Regression: Biased Estimation for Nonorthogonal Problems
- Data. A collection of problems from many fields for the student and research worker
- Machine learning. A probabilistic perspective
- Nonparametric stochastic approximation with large step-sizes
- Title not available (Why is that?)
- A Stochastic Approximation Method
- Title not available (Why is that?)
- Computational Methods for Inverse Problems
- Diffuse Interface Models on Graphs for Classification of High Dimensional Data
- Title not available (Why is that?)
- Stochastic Estimation of the Maximum of a Regression Function
- Title not available (Why is that?)
- Title not available (Why is that?)
- Ensemble Kalman methods for inverse problems
- A Bayesian level set method for geometric inverse problems
- Minimizing finite sums with the stochastic average gradient
- Analysis of the Ensemble Kalman Filter for Inverse Problems
- Data Assimilation
- FitNets
- Entropy-SGD: biasing gradient descent into wide valleys
- Uncertainty Quantification in Graph-Based Classification of High Dimensional Data
- Analysis of the Ensemble and Polynomial Chaos Kalman Filters in Bayesian Inverse Problems
Cited In (42)
- Efficient Bayesian physics informed neural networks for inverse problems via ensemble Kalman inversion
- Quantification and reduction of uncertainties in a wind turbine numerical model based on a global sensitivity analysis and a recursive Bayesian inference approach
- Continuous limits of residual neural networks in case of large input data
- Convergence acceleration of ensemble Kalman inversion in nonlinear settings
- Recent trends on nonlinear filtering for inverse problems
- EnKSGD: a class of preconditioned black box optimization and inversion algorithms
- Analysis of stochastic gradient descent in continuous time
- Deep limits of residual neural networks
- Affine Invariant Interacting Langevin Dynamics for Bayesian Inference
- Efficient estimation of hydraulic conductivity heterogeneity with non-redundant measurement information
- Consensus‐based sampling
- Fokker--Planck Particle Systems for Bayesian Inference: Computational Approaches
- Subsampling in ensemble Kalman inversion
- Tikhonov Regularization within Ensemble Kalman Inversion
- A Stabilization of a Continuous Limit of the Ensemble Kalman Inversion
- $l_p$ Regularization for Ensemble Kalman Inversion
- Iterated Kalman methodology for inverse problems
- Ensemble Inference Methods for Models With Noisy and Expensive Likelihoods
- Gradient flow structure and convergence analysis of the ensemble Kalman inversion for nonlinear forward models
- Continuous Time Limit of the Stochastic Ensemble Kalman Inversion: Strong Convergence Analysis
- Hierarchical ensemble Kalman methods with sparsity-promoting generalized gamma hyperpriors
- Adaptive Tikhonov strategies for stochastic ensemble Kalman inversion
- Adaptive regularisation for ensemble Kalman inversion
- Localized ensemble Kalman inversion
- Mean-field and kinetic descriptions of neural differential equations
- Iterative ensemble Kalman methods: a unified perspective with some new variants
- On convergence rates of adaptive ensemble Kalman inversion for linear ill-posed problems
- Neural dynamical operator: continuous spatial-temporal model with gradient-based and derivative-free optimization methods
- Identifying a fractional order and a time-dependent coefficient in a time-fractional diffusion wave equation
- Sequential Kalman tuning of the \(t\)-preconditioned Crank-Nicolson algorithm: efficient, adaptive and gradient-free inference for Bayesian inverse problems
- Covariance-modulated optimal transport and gradient flows
- Harmonic analysis on graphs via Bratteli diagrams and path-space measures
- Online learning of both state and dynamics using ensemble Kalman filters
- Large-eddy simulation-based shape optimization for mitigating turbulent wakes of a bluff body using the regularized ensemble Kalman method
- Calibrate, emulate, sample
- Derivative-Free Bayesian Inversion Using Multiscale Dynamics
- Ensemble Kalman method for learning turbulence models from indirect observation data
- Ensemble Gradient for Learning Turbulence Models from Indirect Observations
- Affine-invariant ensemble transform methods for logistic regression
- Discrete gradients for computational Bayesian inference
- Interacting Langevin Diffusions: Gradient Structure and Ensemble Kalman Sampler
- Adaptive Ensemble Kalman Inversion with Statistical Linearization
Uses Software
This page was built for publication: Ensemble Kalman inversion: a derivative-free technique for machine learning tasks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5197869)
