Dimension reduction in recurrent networks by canonicalization
From MaRDI portal
Publication:2076953
Abstract: Many recurrent neural network machine learning paradigms can be formulated using state-space representations. The classical notion of canonical state-space realization is adapted in this paper to accommodate semi-infinite inputs so that it can be used as a dimension reduction tool in the recurrent networks setup. The so-called input forgetting property is identified as the key hypothesis that guarantees the existence and uniqueness (up to system isomorphisms) of canonical realizations for causal and time-invariant input/output systems with semi-infinite inputs. Additionally, the notion of optimal reduction coming from the theory of symmetric Hamiltonian systems is implemented in our setup to construct canonical realizations out of input forgetting but not necessarily canonical ones. These two procedures are studied in detail in the framework of linear fading memory input/output systems. Finally, the notion of implicit reduction using reproducing kernel Hilbert spaces (RKHS) is introduced which allows, for systems with linear readouts, to achieve dimension reduction without the need to actually compute the reduced spaces introduced in the first part of the paper.
Recommendations
- Reducing the Dimensionality of Data with Neural Networks
- Model reduction with memory and the machine learning of dynamical systems
- Realization theory of recurrent neural ODEs using polynomial system embeddings
- Dimensionality-reduction using connectionist networks
- Distributed sequence memory of multidimensional inputs in recurrent networks
Cites work
- A Lie-theorie approach to nonlinear discrete-time controllability via Ritt's formal differential groups
- A unified approach to the theory of sampling systems
- Accessibility and foliations with singularities
- Accessible Sets, Orbits, and Foliations with Singularities
- Bayesian filtering and smoothing
- CANONICAL STRUCTURE OF LINEAR DYNAMICAL SYSTEMS
- Clebsch optimal control formulation in mechanics
- Controllability of Nonlinear Discrete-Time Systems: A Lie-Algebraic Approach
- Controlled invariance for nonlinear systems
- Distributed sequence memory of multidimensional inputs in recurrent networks
- Echo state networks are universal
- Echo state networks trained by Tikhonov least squares are \(L^2(\mu)\) approximators of ergodic dynamical systems
- Embedding and approximation theorems for echo state networks
- Extensions of Lipschitz mappings into a Hilbert space
- Fading memory and the problem of approximating nonlinear operators with Volterra series
- Fading memory echo state networks are universal
- Foundations of machine learning
- Hamiltonian reduction by stages
- scientific article; zbMATH DE number 2129143 (Why is no real title available?)
- scientific article; zbMATH DE number 976356 (Why is no real title available?)
- scientific article; zbMATH DE number 1182386 (Why is no real title available?)
- scientific article; zbMATH DE number 1745905 (Why is no real title available?)
- scientific article; zbMATH DE number 205892 (Why is no real title available?)
- scientific article; zbMATH DE number 6125590 (Why is no real title available?)
- scientific article; zbMATH DE number 7306919 (Why is no real title available?)
- scientific article; zbMATH DE number 7164779 (Why is no real title available?)
- scientific article; zbMATH DE number 3227108 (Why is no real title available?)
- scientific article; zbMATH DE number 3297923 (Why is no real title available?)
- Linear Stochastic Systems
- Liquid state machines: motivation, theory, and applications
- Memory and forecasting capacities of nonlinear recurrent networks
- Memory in linear recurrent neural networks in continuous time
- Momentum maps and Hamiltonian reduction
- Nonautonomous dynamical systems
- Nonholonomic mechanics and control. With the collaboration of J. Baillieul, P. E Crouch, J. E. Marsden and D. Zenkov. With scientific input from P. S. Krishnaprasad and R. M. Murray
- Nonlinear memory capacity of parallel time-delay reservoir computers in the processing of multidimensional signals
- Orbits of Families of Vector Fields and Integrability of Distributions
- Quantification géométrique.
- Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations
- Realization theory of discrete-time nonlinear systems: Part I-The bounded case
- Reduction of symplectic manifolds with symmetry
- Reservoir computing approaches to recurrent neural network training
- Singular reduction of implicit Hamiltonian systems
- Statistical Inference for Probabilistic Functions of Finite State Markov Chains
- Stochastic nonlinear time series forecasting using time-delay reservoir computers: performance and universality
- Symmetries and conservation laws for Hamiltonian systems with inputs and outputs: A generalization of Noether's theorem
- Symmetries in Optimal Control
- Symmetry reduction of optimal control systems and principal connections
- The asymptotic performance of linear echo state neural networks
- The identification of nonlinear discrete-time fading-memory systems using neural network models
- The Optimal Momentum Map
- The structure of nonlinear control systems possessing symmetries
- The symplectic reduced spaces of a Poisson action.
- Time series analysis by state space methods.
- Topology and mechanics. I
- Universal discrete-time reservoir computers with stochastic inputs and linear readouts using non-homogeneous state-affine systems
Cited in
(7)- The Gaussian kernel on the circle and spaces that admit isometric embeddings of the circle
- Gaussian kernels on nonsimply connected closed Riemannian manifolds are never positive definite
- Learning strange attractors with reservoir systems
- Expressiveness and structure preservation in learning port-Hamiltonian systems
- Constrained dynamics, stochastic numerical methods and the modeling of complex systems. Abstracts from the workshop held May 26--31, 2024
- Generic and isometric embeddings in reservoir computers
- Forecasting causal dynamics with universal reservoirs
This page was built for publication: Dimension reduction in recurrent networks by canonicalization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2076953)
