Learning in the machine: to share or not to share?

DOI10.1016/J.NEUNET.2020.03.016MaRDI QIDQ1980412zbMATH OpenOpenAlexWikidataFDO

Authors Jordan Ott, Erik Linstead, Nicholas LaHaye, Pierre Baldi

Publication date 8 September 2021

Published in Neural Networks (Search for Journal in Brave)

Copyright license Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International

Full work available at URL https://arxiv.org/abs/1909.11483

zbMATH Keywords

deep learningconvolutional neural networksweight-sharingbiologically plausible architectures

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07)

Abstract: Weight-sharing is one of the pillars behind Convolutional Neural Networks and their successes. However, in physical neural systems such as the brain, weight-sharing is implausible. This discrepancy raises the fundamental question of whether weight-sharing is necessary. If so, to which degree of precision? If not, what are the alternatives? The goal of this study is to investigate these questions, primarily through simulations where the weight-sharing assumption is relaxed. Taking inspiration from neural circuitry, we explore the use of Free Convolutional Networks and neurons with variable connection patterns. Using Free Convolutional Networks, we show that while weight-sharing is a pragmatic optimization approach, it is not a necessity in computer vision applications. Furthermore, Free Convolutional Networks match the performance observed in standard architectures when trained using properly translated data (akin to video). Under the assumption of translationally augmented data, Free Convolutional Networks learn translationally invariant representations that yield an approximate form of weight sharing.

Recommendations

• Convolutional neural networks analyzed via convolutional sparse coding
• On the behavior of convolutional nets for feature extraction
• A Note on the Regularity of Images Generated by Convolutional Neural Networks
• Convolutional neural network models of V1 responses to complex patterns
• Error bounds for approximations using multichannel deep convolutional neural networks with downsampling

Cites work

• scientific article; zbMATH DE number 6378127 (Why is no real title available?)
• Deep learning with dynamic spiking neurons and fixed feedback weights
• Learning in the machine: random backpropagation and the deep learning channel
• Learning in the machine: the symmetries of the deep learning channel
• Neocognition: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position
• The dropout learning algorithm

Cited in

(1)

• Multi-objective ranking to optimize CNN's encoding features: application to the optimization of tracer dose for scintigraphic imagery

Describes a project that uses

Uses Software

• MNIST
• TensorFlow
• Keras
• ImageNet
• GitHub
• AlexNet

This page was built for publication: Learning in the machine: to share or not to share?

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1980412)