Rényi Divergence and Kullback-Leibler Divergence

DOI10.1109/TIT.2014.2320500MaRDI QIDQ2986250zbMATH OpenOpenAlexWikidataFDO

Authors Tim van Erven, Peter Harremoës

Publication date 16 May 2017

Published in IEEE Transactions on Information Theory (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1206.2459

Mathematics Subject Classification ID

Measures of information, entropy (94A17)

Abstract: R'enyi divergence is related to R'enyi entropy much like Kullback-Leibler divergence is related to Shannon's entropy, and comes up in many settings. It was introduced by R'enyi as a measure of information that satisfies almost the same axioms as Kullback-Leibler divergence, and depends on a parameter that is called its order. In particular, the R'enyi divergence of order 1 equals the Kullback-Leibler divergence. We review and extend the most important properties of R'enyi divergence and Kullback-Leibler divergence, including convexity, continuity, limits of $sigma$-algebras and the relation of the special order 0 to the Gaussian dichotomy and contiguity. We also show how to generalize the Pythagorean inequality to orders different from 1, and we extend the known equivalence between channel capacity and minimax redundancy to continuous channel inputs (for all orders) and present several other minimax results.

Cited in

(only showing first 100 items - show all)

• Differential privacy: getting more for less
• Rényi-Holevo inequality from \(\alpha\)-\(z\)-Rényi relative entropies
• Rényi divergence and the central limit theorem
• Uniqueness and characterization theorems for generalized entropies
• A joint Shannon cipher and privacy amplification approach to attaining exponentially decaying information leakage
• Infinite-dimensional gradient-based descent for alpha-divergence minimisation
• A Variational Characterization of Rényi Divergences
• Rényi's divergence as a chemical similarity criterion
• Divergence-based tests for the bivariate gamma distribution applied to polarimetric synthetic aperture radar
• On the convergence of phase space distributions to microcanonical equilibrium: dynamical isometry and generalized coarse-graining
• ordpy: a Python package for data analysis with permutation entropy and ordinal network methods
• Almost perfect quantum state transfer on isotropic and anisotropic Heisenberg spin chains with tailored site dependent exchange couplings
• The computational asymptotics of Gaussian variational inference and the Laplace approximation
• Limits on the efficiency of (ring) LWE based non-interactive key exchange
• On the hardness of module learning with errors with short distributions
• Fast rates for general unbounded loss functions: from ERM to generalized Bayes
• Scalable information inequalities for uncertainty quantification
• A note on quadratic transportation and divergence inequality
• On dynamic pricing under model uncertainty
• Information measures of kernel estimation
• Discounted fully probabilistic design of decision rules
• Common Information, Noise Stability, and Their Extensions
• Optimal experimental design for prediction based on push-forward probability measures
• scientific article; zbMATH DE number 5313608 (Why is no real title available?)
• Bounding Wasserstein Distance with Couplings
• Collision avoidance control for limited perception unmanned surface vehicle swarm based on proximal policy optimization
• Robust mean variance optimization problem under Rényi divergence information
• On the convergence of the maximum likelihood estimator for the transition rate under a 2-state symmetric model
• Rényi divergence minimization based co-regularized multiview clustering
• Rényi divergences from Euclidean quenches
• Measuring diversity in heterogeneous information networks
• On stochastic comparisons of finite -mixture models
• Generative model of autoencoders self-learning on images represented by count samples
• Nonuniform bounds in the Poisson approximation with applications to informational distances. II
• Log-mean distribution: applications to medical data, survival regression, Bayesian and non-Bayesian discussion with MCMC algorithm
• Multiple-source adaptation theory and algorithms
• PACMAN: PAC-style bounds accounting for the mismatch between accuracy and negative log-loss
• Spherical Gaussian leftover hash lemma via the Rényi divergence
• Characterization of time series via Rényi complexity-entropy curves
• scientific article; zbMATH DE number 7650126 (Why is no real title available?)
• Adjusted Rényi entropic value-at-risk
• Logarithmic divergences from optimal transport and Rényi geometry
• Ordering on the probability simplex of endmembers for hyperspectral morphological image processing
• Quantile-based cumulative Kullback–Leibler divergence
• Relative entropy via distribution of observables
• Quantum \(f\)-divergences via Nussbaum-Szkoła distributions and applications to \(f\)-divergence inequalities
• Relations between dissipated work in non-equilibrium process and the family of Rényi divergences
• Properties of noncommutative Rényi and Augustin information
• Transport-majorization to analytic and geometric inequalities
• Differentially private confidence intervals for proportions under stratified random sampling
• Zipf-Mandelbrot law, \(f\)-divergences and the Jensen-type interpolating inequalities
• Mutual singularities of overlapping attractor and repeller
• Label distribution learning by regularized sample self-representation
• Sub-domain adaptation learning methodology
• Sandwiched Rényi relative entropy on density operators
• Data-driven priors and their posterior concentration rates
• The information-theoretic treatment of spinless particles with the assorted diatomic molecular potential
• \(\alpha\)-variational inference with statistical guarantees
• Learning models with uniform performance via distributionally robust optimization
• Hierarchical micro-macro acceleration for moment models of kinetic equations
• Conditionally structured variational Gaussian approximation with importance weights
• Contagion in financial systems: a Bayesian network approach
• Rényi relative entropy from homogeneous Kullback-Leibler divergence Lagrangian
• FL-MAC-RDP: federated learning over multiple access channels with Rényi differential privacy
• Communication-efficient and privacy-preserving large-scale federated learning counteracting heterogeneity
• The sphere packing bound for memoryless channels
• Studying the impact of fluctuations, spikes and rare events in time series through a wavelet entropy predictability measure
• Variational representations of annealing paths: Bregman information under monotonic embedding
• An active contour model for texture image segmentation using Rényi divergence measure
• Cramér-Rao lower bounds arising from generalized Csiszár divergences
• Bayesian fractional posteriors
• Exponential decay of Rényi divergence under Fokker-Planck equations
• Information aware max-norm Dirichlet networks for predictive uncertainty estimation
• Thermodynamic utility of non-Markovianity from the perspective of resource interconversion
• Framework based on communicability to measure the similarity of nodes in complex networks
• Conformal mirror descent with logarithmic divergences
• Convergence rates of variational posterior distributions
• Sensitivity analysis for rare events based on Rényi divergence
• Optimal information, Jensen-RIG function and \(\alpha\)-Onicescu's correlation coefficient in terms of information generating functions
• Local permutation tests for conditional independence
• Convergence rates of deep ReLU networks for multiclass classification
• Price probabilities: a class of Bayesian and non-Bayesian prediction rules
• Decay of convolved densities via Laplace transform
• The Augustin capacity and center
• Uncertainty, information, and disagreement of economic forecasters
• Variational representations and neural network estimation of Rényi divergences
• Analysis of Langevin Monte Carlo via convex optimization
• Mosaics of combinatorial designs for information-theoretic security
• A primer on alpha-information theory with application to leakage in secrecy systems
• General constructions of fuzzy extractors for continuous sources
• Two fractional order cumulative residual time series measures based on Rényi entropy
• The Kullback-Leibler divergence between lattice Gaussian distributions
• The right complexity measure in locally private estimation: it is not the Fisher information
• Optimal control of probabilistic Boolean control networks: A scalable infinite horizon approach
• Stressing dynamic loss models
• Information fractal dimension of mass function
• Optimal insurance under maxmin expected utility
• Poisson approximation in \(\chi^2\) distance by the Stein-Chen approach
• Fault-tolerant fusion using \(\alpha\)-Rényi divergence for autonomous vehicle localization
• Consistency and convergence rate of phylogenetic inference via regularization

This page was built for publication: Rényi Divergence and Kullback-Leibler Divergence

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2986250)