Ensemble Estimation of Generalized Mutual Information With Applications to Genomics

DOI10.1109/TIT.2021.3100108zbMATH Open1486.94041arXiv1701.08083OpenAlexW2971969744MaRDI QIDQ4958280FDOQ4958280

Authors: Kevin R. Moon, Kumar Sricharan, Alfred O. III Hero 

Publication date: 7 September 2021

Published in: IEEE Transactions on Information Theory (Search for Journal in Brave)

Abstract: Mutual information is a measure of the dependence between random variables that has been used successfully in myriad applications in many fields. Generalized mutual information measures that go beyond classical Shannon mutual information have also received much interest in these applications. We derive the mean squared error convergence rates of kernel density-based plug-in estimators of general mutual information measures between two multidimensional random variables $mathbfX$ and $mathbfY$ for two cases: 1) $mathbfX$ and $mathbfY$ are continuous; 2) $mathbfX$ and $mathbfY$ may have any mixture of discrete and continuous components. Using the derived rates, we propose an ensemble estimator of these information measures called GENIE by taking a weighted sum of the plug-in estimators with varied bandwidths. The resulting ensemble estimators achieve the $1/N$ parametric mean squared error convergence rate when the conditional densities of the continuous variables are sufficiently smooth. To the best of our knowledge, this is the first nonparametric mutual information estimator known to achieve the parametric convergence rate for the mixture case, which frequently arises in applications (e.g. variable selection in classification). The estimator is simple to implement and it uses the solution to an offline convex optimization problem and simple plug-in estimators. A central limit theorem is also derived for the ensemble estimators and minimax rates are derived for the continuous case. We demonstrate the ensemble estimator for the mixed case on simulated data and apply the proposed estimator to analyze gene relationships in single cell data.

Full work available at URL: https://arxiv.org/abs/1701.08083

zbMATH Keywords

mutual information measuresensemble estimatorkernel density-based plug-in estimatorsmean squared error convergence rates

Mathematics Subject Classification ID

Measures of information, entropy (94A17)

This page was built for publication: Ensemble Estimation of Generalized Mutual Information With Applications to Genomics

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4958280)