A topological insight into restricted Boltzmann machines

DOI10.1007/S10994-016-5570-ZzbMATH Open1386.68134arXiv1604.05978OpenAlexW2337788193WikidataQ59460817 ScholiaQ59460817MaRDI QIDQ331673FDOQ331673

Authors: Decebal Constantin Mocanu, Elena Mocanu, Phuong H. Nguyen, Madeleine Gibescu, Antonio Liotta 

Publication date: 27 October 2016

Published in: Machine Learning (Search for Journal in Brave)

Abstract: Restricted Boltzmann Machines (RBMs) and models derived from them have been successfully used as basic building blocks in deep artificial neural networks for automatic features extraction, unsupervised weights initialization, but also as density estimators. Thus, their generative and discriminative capabilities, but also their computational time are instrumental to a wide range of applications. Our main contribution is to look at RBMs from a topological perspective, bringing insights from network science. Firstly, here we show that RBMs and Gaussian RBMs (GRBMs) are bipartite graphs which naturally have a small-world topology. Secondly, we demonstrate both on synthetic and real-world datasets that by constraining RBMs and GRBMs to a scale-free topology (while still considering local neighborhoods and data distribution), we reduce the number of weights that need to be computed by a few orders of magnitude, at virtually no loss in generative performance. Thirdly, we show that, for a fixed number of weights, our proposed sparse models (which by design have a higher number of hidden neurons) achieve better generative capabilities than standard fully connected RBMs and GRBMs (which by design have a smaller number of hidden neurons), at no additional computational costs.

Full work available at URL: https://arxiv.org/abs/1604.05978

Recommendations

• Restricted Boltzmann machines: a review
• Restricted Boltzmann machines
• Topological approaches to deep learning
• Restricted Boltzmann machines: introduction and review
• scientific article; zbMATH DE number 6542830
• Representational Power of Restricted Boltzmann Machines and Deep Belief Networks
• Restricted Boltzmann machines as models of interacting variables

zbMATH Keywords

deep learningcomplex networkssmall-world networksscale-free networkssparse restricted Boltzmann machines

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Small world graphs, complex networks (graph-theoretic aspects) (05C82)

Cites Work

• Power-law distributions in empirical data
• Reducing the Dimensionality of Data with Neural Networks
• Learning deep architectures for AI
• Random graphs and complex networks. Volume 1
• Emergence of Scaling in Random Networks
• Collective dynamics of `small-world' networks
• Exploring complex networks
• On Realizability of a Set of Integers as Degrees of the Vertices of a Linear Graph. I
• Training Products of Experts by Minimizing Contrastive Divergence
• The flip-the-state transition operator for restricted Boltzmann machines

Cited In (7)

• A factor graph model for unsupervised feature selection
• A brain-inspired algorithm for training highly sparse neural networks
• Title not available (Why is that?)
• Topologically ordered feature extraction based on sparse group restricted Boltzmann machines
• Title not available (Why is that?)
• The emergence of a concept in shallow neural networks
• Quick and robust feature selection: the strength of energy-efficient sparse training for autoencoders

Uses Software

• NetworkX
• darch
• ImageNet
• plfit
• MADE
• AlexNet

This page was built for publication: A topological insight into restricted Boltzmann machines

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q331673)