Universal Estimation of Directed Information

DOI10.1109/TIT.2013.2267934MaRDI QIDQ5346283zbMATH OpenOpenAlexFDO

Authors Jiantao Jiao, Haim Permuter, Lei Zhao, Younghan Kim, Tsachy Weissman

Publication date 8 June 2017

Published in IEEE Transactions on Information Theory (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1201.2334

Mathematics Subject Classification ID

Information theory (general) (94A15) Measures of information, entropy (94A17)

Abstract: Four estimators of the directed information rate between a pair of jointly stationary ergodic finite-alphabet processes are proposed, based on universal probability assignments. The first one is a Shannon--McMillan--Breiman type estimator, similar to those used by Verd'u (2005) and Cai, Kulkarni, and Verd'u (2006) for estimation of other information measures. We show the almost sure and $L_1$ convergence properties of the estimator for any underlying universal probability assignment. The other three estimators map universal probability assignments to different functionals, each exhibiting relative merits such as smoothness, nonnegativity, and boundedness. We establish the consistency of these estimators in almost sure and $L_1$ senses, and derive near-optimal rates of convergence in the minimax sense under mild conditions. These estimators carry over directly to estimating other information measures of stationary ergodic finite-alphabet processes, such as entropy rate and mutual information rate, with near-optimal performance and provide alternatives to classical approaches in the existing literature. Guided by these theoretical results, the proposed estimators are implemented using the context-tree weighting algorithm as the universal probability assignment. Experiments on synthetic and real data are presented, demonstrating the potential of the proposed schemes in practice and the utility of directed information estimation in detecting and measuring causal influence and delay.

Cited in

(7)

• Causal inference for multivariate stochastic process prediction
• Directed Information on Abstract Spaces: Properties and Variational Equalities
• Estimating the Directed Information and Testing for Causality
• The Convex Mixture Distribution: Granger Causality for Categorical Time Series
• Estimating the directed information to infer causal relationships in ensemble neural spike train recordings
• Information rate analysis of a synaptic release site using a two-state model of short-term depression
• Posterior representations for Bayesian context trees: sampling, estimation and convergence

This page was built for publication: Universal Estimation of Directed Information

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5346283)