Equations of states in singular statistical estimation
From MaRDI portal
Publication:1784533
Abstract: Learning machines which have hierarchical structures or hidden variables are singular statistical models because they are nonidentifiable and their Fisher information matrices are singular. In singular statistical models, neither the Bayes a posteriori distribution converges to the normal distribution nor the maximum likelihood estimator satisfies asymptotic normality. This is the main reason why it has been difficult to predict their generalization performances from trained states. In this paper, we study four errors, (1) Bayes generalization error, (2) Bayes training error, (3) Gibbs generalization error, and (4) Gibbs training error, and prove that there are mathematical relations among these errors. The formulas proved in this paper are equations of states in statistical estimation because they hold for any true distribution, any parametric model, and any a priori distribution. Also we show that Bayes and Gibbs generalization errors are estimated by Bayes and Gibbs training errors, and propose widely applicable information criteria which can be applied to both regular and singular statistical models.
Recommendations
- Maximum likelihood estimation of singular systems of equations
- scientific article; zbMATH DE number 2060113
- State estimation of stochastic singular linear systems
- Estimation of random states in general linear models
- Statistical inference for partial differential equations
- Reparameterization for statistical state estimation applied to differential equations
- Publication:4955299
- scientific article; zbMATH DE number 1064646
- The Cramér-Rao inequality on singular statistical models
- State and parameter estimation in stochastic dynamical models
Cites work
- scientific article; zbMATH DE number 1453027 (Why is no real title available?)
- scientific article; zbMATH DE number 6791314 (Why is no real title available?)
- Algebraic analysis for nonidentifiable learning machines
- B-functions and holonomic systems. Rationality of roots of B-functions
- Learning Coefficients of Layered Models When the True Distribution Mismatches the Singularities
- Lectures on Resolution of Singularities (AM-166)
- On the Asymptotic Distribution of the Least-Squares Estimators in Unidentifiable Models
- On the Problem in Model Selection of Neural Network Regression in Overrealizable Scenario
- Resolution of Singularities and Division of Distributions
- Resolution of singularities of an algebraic variety over a field of characteristic zero. I
- Singularities Affect Dynamics of Learning in Neuromanifolds
- Singularities in mixture models and upper bounds of stochastic complexity.
- Stochastic complexities of reduced rank regression in Bayesian estimation
- Weak convergence and empirical processes. With applications to statistics
Cited in
(15)- Learning Coefficients of Layered Models When the True Distribution Mismatches the Singularities
- Approximating cross-validatory predictive evaluation in Bayesian latent variable models with integrated IS and WAIC
- Consensus theory for mixed response formats
- Consideration on singularities in learning theory and the learning coefficient
- Bayesian fused lasso modeling via horseshoe prior
- A widely applicable Bayesian information criterion
- Asymptotic accuracy of Bayes estimation for latent variables with redundancy
- Learning coefficient of generalization error in Bayesian estimation and Vandermonde matrix-type singularity
- Asymptotic analysis of Bayesian generalization error with Newton diagram
- Consensus theory for multiple latent traits and consensus groups
- Effects of additional data on Bayesian clustering
- Bayesian modeling of bacterial growth for multiple populations
- Conditional vs marginal estimation of the predictive loss of hierarchical models using WAIC and cross-validation
- A limit theorem in singular regression problem
- Consideration on the learning efficiency of multiple-layered neural networks with linear units
This page was built for publication: Equations of states in singular statistical estimation
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1784533)
