Machine learning: deepest learning as statistical data assimilation problems

DOI10.1162/NECO_A_01094MaRDI QIDQ5157217zbMATH OpenOpenAlexDBLPWikidataFDO

Authors Henry D. I. Abarbanel, Paul. J. Rozdeba, Sasha Shirman

Publication date 12 October 2021

Published in Neural Computation (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1707.01415

Mathematics Subject Classification ID

Statistical aspects of big data and data science (62R07) Learning and adaptive systems in artificial intelligence (68T05) Artificial neural networks and deep learning (68T07)

Abstract: We formulate a strong equivalence between machine learning, artificial intelligence methods and the formulation of statistical data assimilation as used widely in physical and biological sciences. The correspondence is that layer number in the artificial network setting is the analog of time in the data assimilation setting. Within the discussion of this equivalence we show that adding more layers (making the network deeper) is analogous to adding temporal resolution in a data assimilation framework. How one can find a candidate for the global minimum of the cost functions in the machine learning context using a method from data assimilation is discussed. Calculations on simple models from each side of the equivalence are reported. Also discussed is a framework in which the time or layer label is taken to be continuous, providing a differential equation, the Euler-Lagrange equation, which shows that the problem being solved is a two point boundary value problem familiar in the discussion of variational methods. The use of continuous layers is denoted "deepest learning". These problems respect a symplectic symmetry in continuous time/layer phase space. Both Lagrangian versions and Hamiltonian versions of these problems are presented. Their well-studied implementation in a discrete time/layer, while respected the symplectic structure, is addressed. The Hamiltonian version provides a direct rationale for back propagation as a solution method for the canonical momentum.

Recommendations

• Machine learning from a continuous viewpoint. I
• Dynamical Systems andOptimal Control Approach to Deep Learning
• “FORCE” learning in recurrent neural networks as data assimilation
• Deep learning as optimal control problems: models and numerical methods
• Autodifferentiable ensemble Kalman filters

Cites work

• A first course in the calculus of variations
• A Limited Memory Algorithm for Bound Constrained Optimization
• A survey of computational physics. Introductory computational science. With CD-Rom
• Algorithm 778: L-BFGS-B
• Calculus of variations and optimal control theory. A concise introduction
• Deep learning
• Discrete mechanics and variational integrators
• Geometric Numerical Integration
• scientific article; zbMATH DE number 45100 (Why is no real title available?)
• scientific article; zbMATH DE number 47206 (Why is no real title available?)
• scientific article; zbMATH DE number 7434519 (Why is no real title available?)
• scientific article; zbMATH DE number 270742 (Why is no real title available?)
• scientific article; zbMATH DE number 3206496 (Why is no real title available?)
• Information and complexity in statistical modeling.
• Mechanical integrators derived from a discrete variational principle
• Numerical recipes. The art of scientific computing.
• On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming
• Predicting the future. Completing models of observed complex systems
• Some NP-complete problems in quadratic and nonlinear programming
• The ensemble Kalman filter for combined state and parameter estimation

Cited in

(21)

• A nonintrusive hybrid neural-physics modeling of incomplete dynamical systems: Lorenz equations
• Online learning of both state and dynamics using ensemble Kalman filters
• Supervised learning from noisy observations: combining machine-learning techniques with data assimilation
• Algorithms of data generation for deep learning and feedback design: a survey
• Least action principles and well-posed learning problems
• Deep learning as optimal control problems: models and numerical methods
• scientific article; zbMATH DE number 6975079 (Why is no real title available?)
• “FORCE” learning in recurrent neural networks as data assimilation
• Using data assimilation to train a hybrid forecast system that combines machine-learning and knowledge-based components
• Deep double descent: where bigger models and more data hurt*
• Reduced-dimension, biophysical neuron models constructed from observed data
• Machine learning of time series using time-delay embedding and precision annealing
• Understanding Deep Learning with Statistical Relevance
• Deep learning: a statistical viewpoint
• Hybrid analysis and modeling, eclecticism, and multifidelity computing toward digital twin revolution
• Inferring unknown unknowns: regularized bias-aware ensemble Kalman filter
• Machine learning-based conditional mean filter: a generalization of the ensemble Kalman filter for nonlinear data assimilation
• Variational data assimilation and its decoupled iterative numerical algorithms for Stokes-Darcy model
• Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observations
• Variational data assimilation with finite-element discretization for second-order parabolic interface equation
• Machine learning from a continuous viewpoint. I

Describes a project that uses

Uses Software

• LBFGS-B
• L-BFGS-B
• EnKF
• Ipopt

This page was built for publication: Machine learning: deepest learning as statistical data assimilation problems

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5157217)