Autodifferentiable ensemble Kalman filters
From MaRDI portal
Publication:5089722
DOI10.1137/21M1434477zbMATH Open1493.62499arXiv2107.07687OpenAlexW3184600788MaRDI QIDQ5089722FDOQ5089722
Authors: Yuming Chen, Daniel Sanz-Alonso, Rebecca Willett 
Publication date: 15 July 2022
Published in: SIAM Journal on Mathematics of Data Science (Search for Journal in Brave)
Abstract: Data assimilation is concerned with sequentially estimating a temporally-evolving state. This task, which arises in a wide range of scientific and engineering applications, is particularly challenging when the state is high-dimensional and the state-space dynamics are unknown. This paper introduces a machine learning framework for learning dynamical systems in data assimilation. Our auto-differentiable ensemble Kalman filters (AD-EnKFs) blend ensemble Kalman filters for state recovery with machine learning tools for learning the dynamics. In doing so, AD-EnKFs leverage the ability of ensemble Kalman filters to scale to high-dimensional states and the power of automatic differentiation to train high-dimensional surrogate models for the dynamics. Numerical results using the Lorenz-96 model show that AD-EnKFs outperform existing methods that use expectation-maximization or particle filters to merge data assimilation and machine learning. In addition, AD-EnKFs are easy to implement and require minimal tuning.
Full work available at URL: https://arxiv.org/abs/2107.07687
Recommendations
- Online learning of both state and dynamics using ensemble Kalman filters
- A hybrid ensemble transform particle filter for nonlinear and spatially extended dynamical systems
- A dynamical systems framework for intermittent data assimilation
- Performance of ensemble Kalman filters in large dimensions
- Efficient data assimilation for spatiotemporal chaos: a local ensemble transform Kalman filter
Markov processes: estimation; hidden Markov models (62M05) Artificial neural networks and deep learning (68T07) Computational aspects of data analysis and big data (68T09) Computational methods for problems pertaining to geophysics (86-08)
Cites Work
- Title not available (Why is that?)
- Pattern recognition and machine learning.
- Title not available (Why is that?)
- Particle Markov Chain Monte Carlo Methods
- Title not available (Why is that?)
- Filtering complex turbulent systems.
- Importance sampling: intrinsic dimension and computational cost
- Title not available (Why is that?)
- The ensemble Kalman filter for combined state and parameter estimation
- Title not available (Why is that?)
- Estimation of high-dimensional prior and posterior covariance matrices in Kalman filter vari\-ants
- Large sample asymptotics for the ensemble Kalman filter
- Optimal filtering and the dual process
- Filter accuracy for the Lorenz 96 model: fixed versus adaptive observation operators
- Relation between two common localisation methods for the EnKF
- An ensemble Kalman filter and smoother for satellite data assimilation
- Estimation of parameterized spatio-temporal dynamic models
- State and parameter estimation in stochastic dynamical models
- Analysis of the ensemble Kalman filter for inverse problems
- Iterative ensemble Kalman methods: a unified perspective with some new variants
- Data-driven science and engineering. Machine learning, dynamical systems, and control
- Data Assimilation
- Numerical Gaussian processes for time-dependent and nonlinear partial differential equations
- Understanding the Ensemble Kalman Filter
- On the stability and the uniform propagation of chaos properties of ensemble Kalman-Bucy filters
- Deterministic mean-field ensemble Kalman filtering
- Long-time asymptotics of the filtering distribution for partially observed chaotic dynamical systems
- Convergence of the square root ensemble Kalman filter in the large ensemble limit
- Probabilistic Forecasting and Bayesian Data Assimilation
- Collected matrix derivative results for forward and reverse mode algorithmic differentiation
- Importance Sampling and Necessary Sample Size: An Information Theory Approach
- Supervised learning from noisy observations: combining machine-learning techniques with data assimilation
- Machine learning for prediction with missing dynamics
- Ensemble Kalman methods for high-dimensional hierarchical dynamic space-time models
Cited In (17)
- Affine-mapping based variational ensemble Kalman filter
- EnKSGD: a class of preconditioned black box optimization and inversion algorithms
- Learning about structural errors in models of complex dynamical systems
- A data-driven non-linear assimilation framework with neural networks
- Quantum Mechanics for Closure of Dynamical Systems
- Machine learning-based conditional mean filter: a generalization of the ensemble Kalman filter for nonlinear data assimilation
- CGNSDE: conditional Gaussian neural stochastic differential equation for modeling complex systems and data assimilation
- A framework for machine learning of model error in dynamical systems
- Using data assimilation to train a hybrid forecast system that combines machine-learning and knowledge-based components
- \(\Phi\)-DVAE: physics-informed dynamical variational autoencoders for unstructured data assimilation
- Deep learning-enhanced ensemble-based data assimilation for high-dimensional nonlinear dynamical systems
- Discrepancy Modeling Framework: Learning Missing Physics, Modeling Systematic Residuals, and Disambiguating between Deterministic and Random Effects
- Hierarchical ensemble Kalman methods with sparsity-promoting generalized gamma hyperpriors
- Machine learning: deepest learning as statistical data assimilation problems
- Statistical variational data assimilation
- Reduced-order autodifferentiable ensemble Kalman filters
- Accurate deep learning-based filtering for chaotic dynamics by identifying instabilities without an ensemble
Uses Software
This page was built for publication: Autodifferentiable ensemble Kalman filters
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5089722)
