Distribution of mutual information from complete and incomplete data
From MaRDI portal
Abstract: Mutual information is widely used, in a descriptive way, to measure the stochastic dependence of categorical random variables. In order to address questions such as the reliability of the descriptive value, one must consider sample-to-population inferential approaches. This paper deals with the posterior distribution of mutual information, as obtained in a Bayesian framework by a second-order Dirichlet prior distribution. The exact analytical expression for the mean, and analytical approximations for the variance, skewness and kurtosis are derived. These approximations have a guaranteed accuracy level of the order O(1/n^3), where n is the sample size. Leading order approximations for the mean and the variance are derived in the case of incomplete samples. The derived analytical expressions allow the distribution of mutual information to be approximated reliably and quickly. In fact, the derived expressions can be computed with the same order of complexity needed for descriptive mutual information. This makes the distribution of mutual information become a concrete alternative to descriptive mutual information in many applications which would benefit from moving to the inductive side. Some of these prospective applications are discussed, and one of them, namely feature selection, is shown to perform significantly better when inductive mutual information is used.
Recommendations
- Bayesian treatment of incomplete discrete data applied to mutual information and feature selection
- Bayesian and quasi-Bayesian estimators for mutual information from discrete data
- Some applications for the useful mutual information
- Multivariate mutual information
- Mutual information and redundancy for categorical data
Cites work
- scientific article; zbMATH DE number 4088699 (Why is no real title available?)
- scientific article; zbMATH DE number 1196643 (Why is no real title available?)
- scientific article; zbMATH DE number 1222289 (Why is no real title available?)
- scientific article; zbMATH DE number 3436645 (Why is no real title available?)
- scientific article; zbMATH DE number 236854 (Why is no real title available?)
- scientific article; zbMATH DE number 3241743 (Why is no real title available?)
- scientific article; zbMATH DE number 3273551 (Why is no real title available?)
- Approximating discrete probability distributions with dependence trees
- Bayesian treatment of incomplete discrete data applied to mutual information and feature selection
- On the optimality of the simple Bayesian classifier under zero-one loss
- Pattern classification.
- Robust inference of trees
- Selection of relevant features and examples in machine learning
- The Analysis of Contingency Tables with Incompletely Classified Data
Cited in
(16)- Mutual information for the multinomial distribution
- Measuring Distances Between Variables by Mutual Information
- Bayesian treatment of incomplete discrete data applied to mutual information and feature selection
- Information entropy, continuous improvement, and US energy performance: a novel stochastic-entropic analysis for ideal solutions (SEA-IS)
- scientific article; zbMATH DE number 4001268 (Why is no real title available?)
- Robust inference of trees
- Multivariate mutual information
- Minimum mutual information and non-Gaussianity through the maximum entropy method: estimation from finite samples
- Evaluation of mutual information estimators for time series
- Bayesian and quasi-Bayesian estimators for mutual information from discrete data
- Tsallis conditional mutual information in investigating long range correlation in symbol sequences
- TCMI: a non-parametric mutual-dependence estimator for multivariate continuous distributions
- Relevance measures for subset variable selection in regression problems based on \(k\)-additive mutual information
- Estimation of mutual information by the fuzzy histogram
- Mutual information and redundancy for categorical data
- A unified definition of mutual information with applications in machine learning
This page was built for publication: Distribution of mutual information from complete and incomplete data
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q957142)
