Statistical estimation of conditional Shannon entropy
From MaRDI portal
Abstract: The new estimates of the conditional Shannon entropy are introduced in the framework of the model describing a discrete response variable depending on a vector of d factors having a density w.r.t. the Lebesgue measure in R^d. Namely, the mixed-pair model (X,Y) is considered where X and Y take values in R^d and an arbitrary finite set, respectively. Such models include, for instance, the famous logistic regression. In contrast to the well-known Kozachenko -- Leonenko estimates of unconditional entropy the proposed estimates are constructed by means of the certain spacial order statistics (or k-nearest neighbor statistics where k=k_n depends on amount of observations n) and a random number of i.i.d. observations contained in the balls of specified random radii. The asymptotic unbiasedness and L^2-consistency of the new estimates are established under simple conditions. The obtained results can be applied to the feature selection problem which is important, e.g., for medical and biological investigations.
Recommendations
Cites work
- f -Divergence Inequalities
- A complete proof of universal inequalities for the distribution function of the binomial law
- A computationally efficient estimator for mutual information
- A Mathematical Theory of Communication
- Bayesian entropy estimation for countable discrete distributions
- Concentration inequalities and model selection. Ecole d'Eté de Probabilités de Saint-Flour XXXIII -- 2003.
- Efficient multivariate entropy estimation via \(k\)-nearest neighbour distances
- Ensemble Estimators for Multivariate Entropy Estimation
- Entropy-based inhomogeneity detection in fiber materials
- Estimation of Entropy and Mutual Information
- scientific article; zbMATH DE number 5957250 (Why is no real title available?)
- scientific article; zbMATH DE number 107482 (Why is no real title available?)
- scientific article; zbMATH DE number 1092005 (Why is no real title available?)
- scientific article; zbMATH DE number 825585 (Why is no real title available?)
- scientific article; zbMATH DE number 922427 (Why is no real title available?)
- Infinite family of approximations of the digamma function
- Lectures on the nearest neighbor method
- Limit theory for point processes in manifolds
- Logistic regression. A self-learning text. With contributions by Erica Rihl Pryor
- Measure theory. Vol. I and II
- Notes on the evolution of feature selection methodology
- On the estimation of entropy
- On the Kozachenko-Leonenko entropy estimator
- Practical guide to logistic regression
- Probability-1
- Real analysis. Theory of measure and integration
- Sample estimate of the entropy of a random vector
- Statistical estimation of the Shannon entropy
- The different paths to entropy
Cited in
(10)- Time-efficient estimation of conditional mutual information for variable selection in classification
- Statistical estimation of the Shannon entropy
- Estimation of the Shannon's entropy of several shifted exponential populations
- Feature selection based on statistical estimation of mutual information
- Entropy-based test for generalised Gaussian distributions
- Statistical estimation of mutual information for mixed model
- scientific article; zbMATH DE number 7149440 (Why is no real title available?)
- On Relevant Features Selection Based on Information Theory
- A new separation index and classification techniques based on Shannon entropy
- Statistical estimates of information characteristics
This page was built for publication: Statistical estimation of conditional Shannon entropy
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4967803)
