Thermodynamics of restricted Boltzmann machines and related learning dynamics (Q1990123)

There is an investigation on thermodynamic properties of realistic RBMs. One starts with the introduction of the restricted Boltzmann machine (RBM) and with the description of the associated learning procedure. In the next section, the static thermodynamical properties of the RBM are introduced. There is a discussion on statistical ensemble of RBMs and further the notions of singular value decomposition (SVD) and weight matrix W are defined. The presentation continues with the mean-field equations in the RS framework together with the corresponding phase-diagram. The nature of the ferromagnetic phase is extensively discussed. The next section is devoted to the investigation on the learning dynamics of the RBM in the thermodynamic limit. The set of dynamical equations is derived and further, the stability analysis of the learning process is performed. The linear regime for an RBM with binary units is considered first. The purpose of this analysis is to identify which deformation modes of the weight matrix are the most unstable and how they are related to the input data. Next, the dynamics and the steady-state regime of a nonlinear RBM are discussed. Results on numerical experiments are shown. The RBM model has been trained on a simple synthetic data, described in Appendix B, and on real world data, the well-known MNIST dataset which is composed of 70.000 handwritten digits. Appendix A: AT Line contains some computational details that are used.