The mutual information of a stochastic binary channel: validity of the replica symmetry ansatz

DOI10.1088/0305-4470/32/10/007zbMATH Open0936.82023arXivcond-mat/9901146OpenAlexW3099980897MaRDI QIDQ4258288FDOQ4258288

Authors: Antonio Turiel, E. R. Korutcheva, Néstor Parga 

Publication date: 24 May 2000

Published in: Journal of Physics A: Mathematical and General (Search for Journal in Brave)

Abstract: We calculate the mutual information (MI) of a two-layered neural network with noiseless, continuous inputs and binary, stochastic outputs under several assumptions on the synaptic efficiencies. The interesting regime corresponds to the limit where the number of both input and output units is large but their ratio is kept fixed at a value $alpha$. We first present a solution for the MI using the replica technique with a replica symmetric (RS) ansatz. Then we find an exact solution for this quantity valid in a neighborhood of $alpha=0$. An analysis of this solution shows that the system must have a phase transition at some finite value of $alpha$. This transition shows a singularity in the third derivative of the MI. As the RS solution turns out to be infinitely differentiable, it could be regarded as a smooth approximation to the MI. This is checked numerically in the validity domain of the exact solution.

Full work available at URL: https://arxiv.org/abs/cond-mat/9901146

Recommendations

• Observation of a hexatic vortex glass in flux lattices of the high-<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:msub><mml:mrow><mml:mi mathvariant="italic">T</mml:mi></mml:mrow><mml:mrow><mml:mi mathvariant="ita
• A perturbative approach to nonlinearities in the information carried by a two layer neural network
• Maximization of mutual information in a linear noisy network: a detailed study
• Representation of Mutual Information Via Input Estimates

zbMATH Keywords

mutual informationGaussian sourcetwo-layered neural networkbinary communication channelcontinuous inputsnoisy perceptron channelreplica symmetric ansatzreplica techniquestochastic outputs

Mathematics Subject Classification ID

Neural nets applied to problems in time-dependent statistical mechanics (82C32)

Cited In (4)

• A perturbative approach to nonlinearities in the information carried by a two layer neural network
• Information restriction in two-step cascade: role of fidelity and fluctuations
• Mutual information as a two-point correlation function in stochastic lattice models
• Information dynamics of neural networks with the aid of supersymmetry fields as the microscopic thermal flow

This page was built for publication: The mutual information of a stochastic binary channel: validity of the replica symmetry ansatz

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4258288)