Linear dependence structure of the entropy space

From MaRDI portal





The author studies linear dependence relations among Shannon's and McGill's information as well as the lattice-theoretic description of them to construct a possible mathematical foundation for information-theoretic multivariate analysis. Let \(\underline X\) be the Boolean lattice formed of marginal random vectors of an \(n\)-dimensional random variable \(X^n = (X_1,\ldots,X_n)\), and let \(\overline H(\underline X)\) (entropy space) be the vector space generated by all the entropies \(h(\alpha)\) \((\alpha\in\underline X)\). Then, the main results are summarized as follows: 1) The difference \(\Delta h(\alpha) = \displaystyle\sum_{\gamma\le \alpha} \mu(\gamma,\alpha)h(\gamma)\) \((\mu(\gamma,\alpha)\): Möbius function of \(\underline X\)) coincides with McGill's multiple mutual information. Similarly for the conditional difference \(\Delta_\sigma h(\alpha)\). 2) The subspace \(\overline H_0(\underline X)\) (correlative entropy space) of \(\overline H(\underline X)\) is defined as the set of all the elements in \(\overline H(\underline X)\) the values of which vanish for independent distributions. Such elements may be regarded as information-theoretic measures of multivariate analysis. 3) The set \(\underline H_1 = \{\Delta h(\alpha\mid r(\alpha)\ge 1\}\) \((r(\alpha)\): rank of \(\alpha)\) is a basis of \(\overline H(\underline X)\), whereas the set \(\underline H_2) = \{\Delta h(\alpha\mid r(\alpha)\ge 2\}\) is a basis of \(\overline H_0(\underline X)\). The dimension of \(\overline H(\underline X)\) is \(2^n - 1\), and that of \(\overline H_0(\underline X)\) is \(2^n - (n +1)\). 4) \(\Delta_\sigma h(\alpha)\) is expanded in terms of \(\Delta_\sigma h(\beta)\) as follows: \[ \Delta_\sigma h(\alpha) = \sum_{\gamma\le\sigma} \Delta h((\alpha\wedge\bar\sigma)\vee \gamma)\quad (\bar\sigma: \text{ complement of }\sigma). \] 5) There exists a probability distribution under which the values of all the elements of \(\underline H_2\) coincide with arbitrarily prescribed nonpositive values. 6) Any linear dependence relations among Shannon's (conditional) mutual information is generated by a particular type of linear dependence relations, i.e., those among chain-forming information on 3-dimensional sublattices. 7) It is shown that the above results are valid, with obvious modifications, for the dual lattice \(\underline X^*\) of \(\underline X\). In parallel with Watanabe's total correlation \(g(X^n) = \sum_{i=1}^n h(X_i) - h(X^n)\), the author defines the dual total correlation \[ g^*(\emptyset) = \sum_{i=1}^n h(\overline X_i) - (n-1)h(X^n), \] which is also nonnegative and vanishes if and only if \(X_1,\ldots, X_n\) are independent. As is well known, this property is a basis for multivariate-analytic interpretation of Watanabe's total correlation.











This page was built for publication: Linear dependence structure of the entropy space

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4082682)