Minimum mutual information and non-Gaussianity through the maximum entropy method: theory and properties (Q406088): Difference between revisions

Summary: The application of the Maximum Entropy (ME) principle leads to a minimum of the Mutual Information (MI), \(I(X,Y)\), between random variables \(X,Y\), which is compatible with prescribed joint expectations and given ME marginal distributions. A sequence of sets of joint constraints leads to a hierarchy of lower MI bounds increasingly approaching the true MI. In particular, using standard bivariate Gaussian marginal distributions, it allows for the MI decomposition into two positive terms: the Gaussian MI \((I_g)\), depending upon the Gaussian correlation or the correlation between `Gaussianized variables', and a non-Gaussian MI \((I_{ng})\), coinciding with joint negentropy and depending upon nonlinear correlations. Joint moments of a prescribed total order \(p\) are bounded within a compact set defined by Schwarz-like inequalities, where \(I_{ng}\) grows from zero at the `Gaussian manifold' where moments are those of Gaussian distributions, towards infinity at the set's boundary where a deterministic relationship holds. Sources of joint non-Gaussianity have been systematized by estimating \(I_{ng}\) between the input and output from a nonlinear synthetic channel contaminated by multiplicative and non-Gaussian additive noises for a full range of signal-to-noise ratio \((snr)\) variances. We have studied the effect of varying \(snr\) on \(I_g\) and \(I_{ng}\) under several signal/noise scenarios.