Mutual information, metric entropy and cumulative relative entropy risk
Given a sample of size \(n\) distributed according \(P_{\theta^*}\) let \(\widehat P_t\) be an estimate of \(P_{\theta^*}\) based on the observations up to prime \(t\leq n\). The authors measure the total risk up to time \(n\) by the Kullback-Leibler divergence \(R_{n,\widehat P}(\theta^*)=D_{KL} (P^n_{\theta^*} \|\widehat P)\) where \(\widehat P\) is the product measure of the \(\widehat P_t\). Using this risk the authors introduce the minimax as well as the Bayes strategy. It turns out that the Bayes risk is the mutual information \(I (\Theta^*;Y^n)\) of the random parameter \(\Theta^*\) and the sample \(Y^n\). Let \(M_{n,\mu}\) be the marginal distribution of \(Y^n\) if \(\Theta^*\) has distribution \(\mu\). The main results in the first part of the paper concern two-sided bounds of \(I(\Theta^*;Y^n)\) and \(D_{KL}(P^n_{\theta^*}\| M_{n,\mu})\) in terms of Renyi distances \(I_\alpha (P_{\theta^*}, P_{\tilde\theta})\) of order \(\alpha\) and \(I_1 (P_{\theta^*},Q^{\tilde\theta})\) where \(Q_\theta\) is the distribution of \(Y\) given \(\theta\). For finite parameter set \(I(\Theta^*;Y^n)\) is shown to tend with exponential rate to the entropy \(H(\Theta^*)\). The asymptotics of the minimax risk based on the risk \(R_{n,\widehat P}(\theta^*)\) is studied for any \(\Theta\) in the second part of the paper. The authors use the concepts metric entropy and metric dimension to describe the topological structure of \(\Theta\) and to relate these properties to the minimax risk for large \(n\).
- A bound on the financial value of information
- A general minimax result for relative entropy
- A loss bound model for on-line stochastic prediction algorithms
- A source matching approach to finding minimax codes
- A strong version of the redundancy-capacity theorem of universal coding
- Approximation dans les espaces m�triques et th�orie de l'estimation
- Asymptotic methods in statistical decision theory
- Bounds on the sample complexity of Bayesian learning using information theory and the VC dimension
- Density estimation by stochastic complexity
- Differential geometry of curved exponential families. Curvatures and information loss
- Distribution estimation consistent in total variation and in two types of information divergence
- Entropies of several sets of real valued functions
- scientific article; zbMATH DE number 3883309 (Why is no real title available?)
- scientific article; zbMATH DE number 51427 (Why is no real title available?)
- scientific article; zbMATH DE number 107482 (Why is no real title available?)
- scientific article; zbMATH DE number 524352 (Why is no real title available?)
- scientific article; zbMATH DE number 3202900 (Why is no real title available?)
- scientific article; zbMATH DE number 3224161 (Why is no real title available?)
- Information contained in a sequence of observations
- Information-theoretic asymptotics of Bayes methods
- Jeffreys' prior is asymptotically least favorable under entropy risk
- Lower bounds on expected redundancy for nonparametric classes
- On density estimation in the view of Kolmogorov's ideas in approximation theory
- On estimating a density using Hellinger distance and some other strange facts
- On the consistency of Bayes estimates
- On the stochastic complexity of learning realizable and unrealizable rules
- Probability inequalities for likelihood ratios and convergence rates of sieve MLEs
- Rates of convergence for minimum contrast estimators
- Recent Developments in Nonparametric Density Estimation
- Some limit theorems for empirical processes (with discussion)
- Stochastic complexity and modeling
- The strong ergodic theorem for densities: Generalized Shannon-McMillan- Breiman theorem
- There is no universal source code for an infinite source alphabet
- Transformations of Wiener integrals under translations
- Information-theoretic determination of minimax rates of convergence
- Hölder's identity
- Mixing strategies for density estimation.
- Improved lower bounds for learning from noisy examples: An information-theoretic approach
- Simultaneous prediction of independent Poisson observables
- Loss of information of a statistic for a family of non-regular distributions. II: More general case
- Convergence rates of deep ReLU networks for multiclass classification
- Bayesian parametric inference in a nonparametric framework
- Optimal quantization of the support of a continuous multivariate distribution based on mutual information
- Predictability, complexity, and learning
- On the practical implication of mutual information for statistical decisionmaking
- Statistical Decision Problems and Bayesian Nonparametric Methods
- Adaptive Design Optimization: A Mutual Information-Based Approach to Model Discrimination in Cognitive Science
- A general minimax result for relative entropy
- scientific article; zbMATH DE number 1121816 (Why is no real title available?)
- scientific article; zbMATH DE number 1139796 (Why is no real title available?)
- Catching up Faster by Switching Sooner: A Predictive Approach to Adaptive Estimation with an Application to the AIC–BIC Dilemma
- Conjugate Priors Represent Strong Pre-Experimental Assumptions
- Fast rates for general unbounded loss functions: from ERM to generalized Bayes
- Prequential analysis of complex data with adaptive model reselection
- scientific article; zbMATH DE number 7370563 (Why is no real title available?)
- Quantifying information conveyed by large neuronal populations
- Entropy-SGD: biasing gradient descent into wide valleys
- Information aware max-norm Dirichlet networks for predictive uncertainty estimation
- Markov chain entropy games and the geometry of their Nash equilibria
This page was built for publication: Mutual information, metric entropy and cumulative relative entropy risk
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1383090)
