Equations of states in singular statistical estimation
From MaRDI portal
Publication:1784533
DOI10.1016/J.NEUNET.2009.08.002zbMATH Open1396.68106DBLPjournals/nn/Watanabe10arXiv0712.0653OpenAlexW2055505622WikidataQ51798993 ScholiaQ51798993MaRDI QIDQ1784533FDOQ1784533
Authors: Sumio Watanabe 
Publication date: 27 September 2018
Published in: Neural Networks (Search for Journal in Brave)
Abstract: Learning machines which have hierarchical structures or hidden variables are singular statistical models because they are nonidentifiable and their Fisher information matrices are singular. In singular statistical models, neither the Bayes a posteriori distribution converges to the normal distribution nor the maximum likelihood estimator satisfies asymptotic normality. This is the main reason why it has been difficult to predict their generalization performances from trained states. In this paper, we study four errors, (1) Bayes generalization error, (2) Bayes training error, (3) Gibbs generalization error, and (4) Gibbs training error, and prove that there are mathematical relations among these errors. The formulas proved in this paper are equations of states in statistical estimation because they hold for any true distribution, any parametric model, and any a priori distribution. Also we show that Bayes and Gibbs generalization errors are estimated by Bayes and Gibbs training errors, and propose widely applicable information criteria which can be applied to both regular and singular statistical models.
Full work available at URL: https://arxiv.org/abs/0712.0653
Recommendations
- Maximum likelihood estimation of singular systems of equations
- scientific article; zbMATH DE number 2060113
- State estimation of stochastic singular linear systems
- Estimation of random states in general linear models
- Statistical inference for partial differential equations
- Reparameterization for statistical state estimation applied to differential equations
- Publication:4955299
- scientific article; zbMATH DE number 1064646
- The Cramér-Rao inequality on singular statistical models
- State and parameter estimation in stochastic dynamical models
Cites Work
- Weak convergence and empirical processes. With applications to statistics
- Title not available (Why is that?)
- Singularities in mixture models and upper bounds of stochastic complexity.
- Stochastic complexities of reduced rank regression in Bayesian estimation
- Resolution of singularities of an algebraic variety over a field of characteristic zero. I
- Algebraic analysis for nonidentifiable learning machines
- Resolution of Singularities and Division of Distributions
- Lectures on Resolution of Singularities (AM-166)
- B-functions and holonomic systems. Rationality of roots of B-functions
- On the Problem in Model Selection of Neural Network Regression in Overrealizable Scenario
- Singularities Affect Dynamics of Learning in Neuromanifolds
- Title not available (Why is that?)
- On the Asymptotic Distribution of the Least-Squares Estimators in Unidentifiable Models
- Learning Coefficients of Layered Models When the True Distribution Mismatches the Singularities
Cited In (13)
- Learning Coefficients of Layered Models When the True Distribution Mismatches the Singularities
- Approximating cross-validatory predictive evaluation in Bayesian latent variable models with integrated IS and WAIC
- Consensus theory for mixed response formats
- Consideration on singularities in learning theory and the learning coefficient
- Bayesian fused lasso modeling via horseshoe prior
- Consensus theory for multiple latent traits and consensus groups
- A widely applicable Bayesian information criterion
- Consideration on the learning efficiency of multiple-layered neural networks with linear units
- Bayesian modeling of bacterial growth for multiple populations
- A limit theorem in singular regression problem
- Effects of additional data on Bayesian clustering
- Learning coefficient of generalization error in Bayesian estimation and Vandermonde matrix-type singularity
- Conditional vs marginal estimation of the predictive loss of hierarchical models using WAIC and cross-validation
This page was built for publication: Equations of states in singular statistical estimation
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1784533)
