Analysis of KNN Information Estimators for Smooth Distributions

DOI10.1109/TIT.2019.2945041MaRDI QIDQ5124400zbMATH OpenOpenAlexFDO

Authors Puning Zhao, Lifeng Lai

Publication date 29 September 2020

Published in IEEE Transactions on Information Theory (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1810.11571

Mathematics Subject Classification ID

Statistical aspects of information-theoretic topics (62B10) Classification and discrimination; cluster analysis (statistical aspects) (62H30) Measures of information, entropy (94A17)

Abstract: KSG mutual information estimator, which is based on the distances of each sample to its k-th nearest neighbor, is widely used to estimate mutual information between two continuous random variables. Existing work has analyzed the convergence rate of this estimator for random variables whose densities are bounded away from zero in its support. In practice, however, KSG estimator also performs well for a much broader class of distributions, including not only those with bounded support and densities bounded away from zero, but also those with bounded support but densities approaching zero, and those with unbounded support. In this paper, we analyze the convergence rate of the error of KSG estimator for smooth distributions, whose support of density can be both bounded and unbounded. As KSG mutual information estimator can be viewed as an adaptive recombination of KL entropy estimators, in our analysis, we also provide convergence analysis of KL entropy estimator for a broad class of distributions.

Recommendations

• Demystifying Fixed <inline-formula> <tex-math notation="LaTeX">$k$ </tex-math> </inline-formula>-Nearest Neighbor Information Estimators
• scientific article; zbMATH DE number 1240561
• Smoothed analysis of the k-means method
• Rates of convergence for the \(k\)-nearest neighbor estimators with smoother regression functions
• Efficiency behaviour of kernel-smoothed kernel distribution function estimators
• Information measures of kernel estimation
• A new smoothness quantification in kernel density estimation
• Distribution estimation with smoothed auxiliary information
• Kullback-Leibler informational measure in the distribution density estimation problem

Cited in

(1)

• A novel approach of dependence measure for complex signals

This page was built for publication: Analysis of KNN Information Estimators for Smooth Distributions

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5124400)