Sparsity in long-time control of neural ODEs
From MaRDI portal
Publication:6099693
Abstract: We consider the neural ODE and optimal control perspective of supervised learning, with -control penalties, where rather than only minimizing a final cost (the emph{empirical risk}) for the state, we integrate this cost over the entire time horizon. We prove that any optimal control (for this cost) vanishes beyond some positive stopping time. When seen in the discrete-time context, this result entails an emph{ordered} sparsity pattern for the parameters of the associated residual neural network: ordered in the sense that these parameters are all beyond a certain layer. Furthermore, we provide a polynomial stability estimate for the empirical risk with respect to the time horizon. This can be seen as a emph{turnpike property}, for nonsmooth dynamics and functionals with -penalties, and without any smallness assumptions on the data, both of which are new in the literature.
Recommendations
- An optimal control framework for adaptive neural ODEs
- Turnpike in optimal control of PDEs, ResNets, and beyond
- Regularization and discretization error estimates for optimal control of ODEs with group sparsity
- Infinite horizon sparse optimal control
- Deep learning as optimal control problems: models and numerical methods
Cites work
- A proposal on machine learning via dynamical systems
- Deep learning
- Deep learning as optimal control problems: models and numerical methods
- Exact turnpike properties and economic NMPC
- scientific article; zbMATH DE number 845714 (Why is no real title available?)
- Infinite horizon sparse optimal control
- Interpolation and approximation via momentum ResNets and neural ODEs
- Linear Inversion of Band-Limited Reflection Seismograms
- Linear-quadratic control problems with \(\mathbf{L}^{\mathbf{1}}\)-control cost
- Maximum principle based algorithms for deep learning
- Mean-field sparse optimal control
- Normalizing flows for probabilistic modeling and inference
- Optimal approximation with sparsely connected deep neural networks
- Real analysis. Theory of measure and integration
- Sensitivity analysis of optimal control for a class of parabolic PDEs motivated by model predictive control
- Sparse and switching infinite horizon optimal controls with mixed-norm penalizations
- Sparse stabilization and control of alignment models
- Sparse stabilization and optimal control of the Cucker-Smale model
- Stable architectures for deep neural networks
- Structure-preserving deep learning
- Switching control
- The finite-time turnpike phenomenon for optimal control problems: stabilization by non-smooth tracking terms
- The turnpike property in finite-dimensional nonlinear optimal control
- Turnpike in Lipschitz-nonlinear optimal control
- Turnpike in optimal control of PDEs, ResNets, and beyond
- Turnpike properties in optimal control: an overview of discrete-time and continuous-time results
Cited in
(11)- On obtaining sparse semantic solutions for inverse problems, control, and neural network training
- Neural ODE Control for Classification, Approximation, and Transport
- Control of neural transport for normalising flows
- An optimal control framework for adaptive neural ODEs
- On dissipativity of cross-entropy loss in training ResNets. A turnpike towards architecture search
- Universal approximation of dynamical systems by semiautonomous neural ODEs and applications
- Local manifold approximation of dynamical system based on neural ordinary differential equation
- Cluster-based classification with neural ODEs via control
- A mathematical perspective on transformers
- Almost periodic turnpike phenomenon for time-dependent systems
- Optimal control of neural differential equations: the turnpike property
This page was built for publication: Sparsity in long-time control of neural ODEs
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6099693)
