Metric entropy limits on recurrent neural network learning of linear dynamical systems
From MaRDI portal
Abstract: One of the most influential results in neural network theory is the universal approximation theorem [1, 2, 3] which states that continuous functions can be approximated to within arbitrary accuracy by single-hidden-layer feedforward neural networks. The purpose of this paper is to establish a result in this spirit for the approximation of general discrete-time linear dynamical systems - including time-varying systems - by recurrent neural networks (RNNs). For the subclass of linear time-invariant (LTI) systems, we devise a quantitative version of this statement. Specifically, measuring the complexity of the considered class of LTI systems through metric entropy according to [4], we show that RNNs can optimally learn - or identify in system-theory parlance - stable LTI systems. For LTI systems whose input-output relation is characterized through a difference equation, this means that RNNs can learn the difference equation from input-output traces in a metric-entropy optimal manner.
Recommendations
Cites work
- scientific article; zbMATH DE number 3147888 (Why is no real title available?)
- scientific article; zbMATH DE number 1022658 (Why is no real title available?)
- scientific article; zbMATH DE number 1161567 (Why is no real title available?)
- A note on metric dimension and feedback in discrete time
- Affine symmetries and neural network identifiability
- Analysis of the generalization error: empirical risk minimization over deep artificial neural networks overcomes the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations
- Approximating nonlinear fading-memory operators using neural network models
- Approximation by superpositions of a sigmoidal function
- Data compression and harmonic analysis
- Deep Neural Network Approximation Theory
- Deep learning
- Fast projection methods for minimal design problems in linear system theory
- Foundations of time-frequency analysis
- Gradient descent learns linear dynamical systems
- High-dimensional statistics. A non-asymptotic viewpoint
- Introduction to mathematical systems theory. Discrete time linear systems, control and identification
- Multilayer feedforward networks are universal approximators
- Neural network identifiability for a family of sigmoidal nonlinearities
- On the computational power of neural nets
- On the metric complexity of casual linear systems: ε -Entropy and ε -Dimension for continuous time
- Optimal approximation with sparsely connected deep neural networks
- Pick's Theorem-What's the Big Deal?
- Reconstructing a neural net from its output
- Sparse components of images and optimal atomic decompositions
- The uncertainty principle
- Unconditional bases and bit-level compression
Cited in
(9)- Riemannian metrics for neural networks II: recurrent networks and learning symbolic data sequences
- Probing the relationship between latent linear dynamical systems and low-rank recurrent neural network models
- Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks
- Relative entropy minimizing noisy non-linear neural network to approximate stochastic processes
- Designing universal causal deep learning models: The geometric (Hyper)transformer
- Linear RNNs provably learn linear dynamical systems
- Approximation bounds for random neural networks and reservoir systems
- On the adaptation of recurrent neural networks for system identification
- A metric entropy bound is not sufficient for learnability
This page was built for publication: Metric entropy limits on recurrent neural network learning of linear dynamical systems
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2134114)
