Empirical risk minimization and complexity of dynamical models

DOI10.1214/19-AOS1876MaRDI QIDQ2215723zbMATH OpenFDO

Authors Kevin McGoff, Andrew B. Nobel

Publication date 14 December 2020

Published in The Annals of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1611.06173, https://projecteuclid.org/euclid.aos/1597370663

zbMATH Keywords

dynamical modelsempirical risk minimizationjoiningstopological entropy

Mathematics Subject Classification ID

Non-Markovian processes: estimation (62M09) Topological data analysis (62R40)

Abstract: A dynamical model consists of a continuous self-map $T:mathcalXomathcalX$ of a compact state space $mathcalX$ and a continuous observation function $f:mathcalXomathbbR$. This paper considers the fitting of a parametrized family of dynamical models to an observed real-valued stochastic process using empirical risk minimization. The limiting behavior of the minimum risk parameters is studied in a general setting. We establish a general convergence theorem for minimum risk estimators and ergodic observations. We then study conditions under which empirical risk minimization can effectively separate the signal from the noise in an additive observational noise model. The key, necessary condition in the latter results is that the family of dynamical models has limited complexity, which is quantified through a notion of entropy for families of infinite sequences. Close connections between entropy and limiting average mean widths for stationary processes are established.

Review text

One investigates the use of an empirical risk minimization approach to the problem of fitting a family of dynamical models to an observed stochastic process. If \(\mathcal{D}\) denotes a family of dynamical models and \(\mathbf{Y}\) is an observed stationary ergodic process, one defines \(R_n(\theta: x)\) as the empirical risk of the model with initial state \(x\) relative to the first observations of \(\mathbf{Y}\). The minimum risk estimates for \(\mathcal{D}\) are introduced as a sequence of measurable functions \(\theta _n\) fulfilling some property. Given the observations, one selects a parameter \(\hat{\theta}_n\) and an initial state \(\hat{x}_n\) such that the couple is an approximate minimizer for \(R_n(\theta: x)\). The minimum risk estimation involves the identification of an optimal or near optimal parameter state-pair, \((\hat{\theta}_n, \hat{x}_n)\). As the authors state, the principal goal of this paper is to characterize the limiting behavior of the minimum risk estimates \(\hat{\theta}_n\). The second section is devoted to a discussion on entropy for families of dynamical models and the third section to a discussion on the following dynamical models: gene regulatory networks, subcritical logistic family and ecology, symbolic dynamics and quasicrystals, toral rotations and almost periodicity. More basics on processes associated to dynamical models are shown in the fourth section. The main results of the article are presented in the fifth section. If the family of dynamical models, the loss function and the observations satisfy some special assumptions, one obtains a variational characterization of the limiting behavior of minimum risk estimates. A further result is concerned with the question if the minimum risk estimation can recover optimal parameters for an underlaying signal process \(\mathbf{V}\) rather that the observed process \(\mathbf{Y}\). Related work is discussed in the sixth section and proofs are contained in the last two sections. One reports that all other proofs can be found in the supplementary material, Appendices A--E (\url{doi:10.1214/19-AOS1875SUPP}).

Recommendations

• Empirical risk minimizing for dynamical data structures
• Efficient Estimation of Dynamical Systems
• scientific article; zbMATH DE number 4105912
• Empirical estimates with minimal \(d\)-risk for discrete exponential families
• On a method of empirical risk minimization

Cites work

• A Bernstein-type inequality for some mixing processes and dynamical systems with an application to learning
• A generalization of Ornstein's \(\overline d\) distance with applications to information theory
• An application of ergodic theory to probability theory
• An introduction to joinings in ergodic theory
• An Introduction to Symbolic Dynamics and Coding
• Asymptotic theory of nonlinear least squares estimation
• Beneath the noise, chaos
• Bernoulli shifts with the same entropy are isomorphic
• Central limit theorems for empirical and U-processes of stationary mixing sequences
• Chaos in the Lorenz equations: A computer assisted proof. III: Classical parameter values
• Concentration inequalities. A nonasymptotic theory of independence
• Consistency of maximum likelihood estimation for some dynamical systems
• Consistency of support vector machines for forecasting the evolution of an unknown ergodic dynamical system from observations with unknown noise
• Denoising deterministic time series
• Disjointness in ergodic theory, minimal sets, and a problem in diophantine approximation
• Equilibrium states and the ergodic theory of Anosov diffeomorphisms
• Ergodic theory on compact spaces
• Estimating a regression function
• Fast learning from \(\alpha\)-mixing observations
• Fifty years of entropy in dynamics: 1958 -- 2007
• Generalization bounds for non-stationary mixing processes
• How sampling reveals a process
• scientific article; zbMATH DE number 3745547 (Why is no real title available?)
• scientific article; zbMATH DE number 3463931 (Why is no real title available?)
• scientific article; zbMATH DE number 1223460 (Why is no real title available?)
• scientific article; zbMATH DE number 978565 (Why is no real title available?)
• scientific article; zbMATH DE number 1987699 (Why is no real title available?)
• scientific article; zbMATH DE number 2079141 (Why is no real title available?)
• scientific article; zbMATH DE number 1889829 (Why is no real title available?)
• scientific article; zbMATH DE number 776267 (Why is no real title available?)
• scientific article; zbMATH DE number 891082 (Why is no real title available?)
• Hypothesis testing for families of ergodic processes
• Introduction to Dynamical Systems
• Kernel density estimation for dynamical systems
• Memory-universal prediction of stationary random processes
• Metric entropy and the optimal prediction of chaotic signals
• Model selection for weakly dependent time series forecasting
• Nonlinear least-squares estimation
• Nonparametric time series prediction through adaptive model selection
• Nonuniform hyperbolicity. Dynamics of systems with nonzero Lyapunov exponents
• On classifying processes
• On the sample complexity of the linear quadratic regulator
• Prediction for discrete time series
• Prediction uncertainty and optimal experimental design for learning dynamical systems
• Probability in Banach spaces. Isoperimetry and processes
• Rates of convergence for empirical processes of stationary mixing sequences
• Separating signal from noise
• Sequential complexities and uniform martingale laws of large numbers
• Spectral properties of Schrödinger operators arising in the study of quasicrystals
• Stability bounds for stationary -mixing and -mixing processes
• Statistical estimation with bounded memory
• Statistical inference for dynamical systems: a review
• Stochastic optimal control. The discrete time case
• Substitution dynamical systems. Spectral analysis
• The Generalization Ability of Online Algorithms for Dependent Data
• The Poincaré-Bendixson theorem for monotone cyclic feedback systems
• Thermodynamic Formalism
• Topological Entropy
• Topological entropy of the transformation \(x\mapsto ax(1-x)\)
• Uniform convergence of Vapnik-Chervonenkis classes under ergodic sampling
• Weak convergence and empirical processes. With applications to statistics

Cited in

(5)

• Gibbs posterior convergence and the thermodynamic formalism
• scientific article; zbMATH DE number 5972347 (Why is no real title available?)
• Empirical risk minimization for dynamical systems and stationary processes
• scientific article; zbMATH DE number 7625198 (Why is no real title available?)
• Dynamical hypothesis tests and decision theory for Gibbs distributions

This page was built for publication: Empirical risk minimization and complexity of dynamical models

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2215723)