Directed Information on Abstract Spaces: Properties and Variational Equalities

DOI10.1109/TIT.2016.2604846MaRDI QIDQ2976481zbMATH OpenOpenAlexWikidataFDO

Authors Photios A. Stavrou, Charalambos D. Charalambous

Publication date 28 April 2017

Published in IEEE Transactions on Information Theory (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1302.3971

Mathematics Subject Classification ID

Variational inequalities (49J40) Measures of information, entropy (94A17)

Abstract: Directed information or its variants are utilized extensively in the characterization of the capacity of channels with memory and feedback, nonanticipative lossy data compression, and their generalizations to networks. In this paper, we derive several functional and topological properties of directed information for general abstract alphabets (complete separable metric spaces) using the topology of weak convergence of probability measures. These include convexity of the set of consistent distributions, which uniquely define causally conditioned distributions, convexity and concavity of directed information with respect to the sets of consistent distributions, weak compactness of these sets of distributions, their joint distributions and their marginals. Furthermore, we show lower semicontinuity of directed information, and under certain conditions we also establish continuity of directed information. Finally, we derive variational equalities for directed information, including sequential versions. These may be viewed as the analogue of the variational equalities of mutual information (utilized in Blahut-Arimoto algorithm). In summary, we extend the basic functional and topological properties of mutual information to directed information. These properties are discussed in the context of extremum problems of directed information.

Recommendations

• Directed information on abstract spaces: properties and extremum problems
• Information and \(\sigma\)-algebras
• scientific article; zbMATH DE number 3965962
• scientific article; zbMATH DE number 2169043
• A Unifying Variational Perspective on Some Fundamental Information Theoretic Inequalities
• scientific article; zbMATH DE number 3731917
• Universal Estimation of Directed Information
• Information geometry of the space of probability measures and barycenter maps
• scientific article; zbMATH DE number 3797668
• F‐Manifolds and geometry of information

Cited in

(4)

• Optimal estimation via nonanticipative rate distortion function and applications to time-varying Gauss-Markov processes
• Ergodic control-coding capacity of stochastic control systems: information signalling and hierarchical optimality of Gaussian systems
• Optimal Control and Signaling Strategies of Control-Coding Capacity of General Decision Models: Applications to Gaussian Models and Decentralized Strategies
• Signalling and control of nonlinear partially observable stochastic systems: information states, sufficient statistics, and applications

This page was built for publication: Directed Information on Abstract Spaces: Properties and Variational Equalities

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2976481)