Autodifferentiable ensemble Kalman filters
From MaRDI portal
Publication:5089722
Abstract: Data assimilation is concerned with sequentially estimating a temporally-evolving state. This task, which arises in a wide range of scientific and engineering applications, is particularly challenging when the state is high-dimensional and the state-space dynamics are unknown. This paper introduces a machine learning framework for learning dynamical systems in data assimilation. Our auto-differentiable ensemble Kalman filters (AD-EnKFs) blend ensemble Kalman filters for state recovery with machine learning tools for learning the dynamics. In doing so, AD-EnKFs leverage the ability of ensemble Kalman filters to scale to high-dimensional states and the power of automatic differentiation to train high-dimensional surrogate models for the dynamics. Numerical results using the Lorenz-96 model show that AD-EnKFs outperform existing methods that use expectation-maximization or particle filters to merge data assimilation and machine learning. In addition, AD-EnKFs are easy to implement and require minimal tuning.
Recommendations
- Online learning of both state and dynamics using ensemble Kalman filters
- A hybrid ensemble transform particle filter for nonlinear and spatially extended dynamical systems
- A dynamical systems framework for intermittent data assimilation
- Performance of ensemble Kalman filters in large dimensions
- Efficient data assimilation for spatiotemporal chaos: a local ensemble transform Kalman filter
Cites work
- scientific article; zbMATH DE number 5919872 (Why is no real title available?)
- scientific article; zbMATH DE number 5822909 (Why is no real title available?)
- scientific article; zbMATH DE number 4211245 (Why is no real title available?)
- scientific article; zbMATH DE number 3567782 (Why is no real title available?)
- scientific article; zbMATH DE number 2106098 (Why is no real title available?)
- An ensemble Kalman filter and smoother for satellite data assimilation
- Analysis of the ensemble Kalman filter for inverse problems
- Collected matrix derivative results for forward and reverse mode algorithmic differentiation
- Convergence of the square root ensemble Kalman filter in the large ensemble limit
- Data Assimilation
- Data-driven science and engineering. Machine learning, dynamical systems, and control
- Deterministic mean-field ensemble Kalman filtering
- Ensemble Kalman methods for high-dimensional hierarchical dynamic space-time models
- Estimation of high-dimensional prior and posterior covariance matrices in Kalman filter vari\-ants
- Estimation of parameterized spatio-temporal dynamic models
- Filter accuracy for the Lorenz 96 model: fixed versus adaptive observation operators
- Filtering complex turbulent systems.
- Importance Sampling and Necessary Sample Size: An Information Theory Approach
- Importance sampling: intrinsic dimension and computational cost
- Iterative ensemble Kalman methods: a unified perspective with some new variants
- Large sample asymptotics for the ensemble Kalman filter
- Long-time asymptotics of the filtering distribution for partially observed chaotic dynamical systems
- Machine learning for prediction with missing dynamics
- Numerical Gaussian processes for time-dependent and nonlinear partial differential equations
- On the stability and the uniform propagation of chaos properties of ensemble Kalman-Bucy filters
- Optimal filtering and the dual process
- Particle Markov Chain Monte Carlo Methods
- Pattern recognition and machine learning.
- Probabilistic Forecasting and Bayesian Data Assimilation
- Relation between two common localisation methods for the EnKF
- State and parameter estimation in stochastic dynamical models
- Supervised learning from noisy observations: combining machine-learning techniques with data assimilation
- The ensemble Kalman filter for combined state and parameter estimation
- Understanding the Ensemble Kalman Filter
Cited in
(17)- Affine-mapping based variational ensemble Kalman filter
- EnKSGD: a class of preconditioned black box optimization and inversion algorithms
- A data-driven non-linear assimilation framework with neural networks
- Learning about structural errors in models of complex dynamical systems
- Quantum Mechanics for Closure of Dynamical Systems
- Machine learning-based conditional mean filter: a generalization of the ensemble Kalman filter for nonlinear data assimilation
- CGNSDE: conditional Gaussian neural stochastic differential equation for modeling complex systems and data assimilation
- A framework for machine learning of model error in dynamical systems
- Using data assimilation to train a hybrid forecast system that combines machine-learning and knowledge-based components
- Deep learning-enhanced ensemble-based data assimilation for high-dimensional nonlinear dynamical systems
- \(\Phi\)-DVAE: physics-informed dynamical variational autoencoders for unstructured data assimilation
- Discrepancy Modeling Framework: Learning Missing Physics, Modeling Systematic Residuals, and Disambiguating between Deterministic and Random Effects
- Hierarchical ensemble Kalman methods with sparsity-promoting generalized gamma hyperpriors
- Machine learning: deepest learning as statistical data assimilation problems
- Statistical variational data assimilation
- Reduced-order autodifferentiable ensemble Kalman filters
- Accurate deep learning-based filtering for chaotic dynamics by identifying instabilities without an ensemble
This page was built for publication: Autodifferentiable ensemble Kalman filters
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5089722)
