Combinatorial designs for deep learning

DOI10.1002/JCD.21720MaRDI QIDQ6146764zbMATH OpenFDO

Authors Shoko Chisaki, Ryoh Fuji-Hara, Nobuko Miyamoto

Publication date 31 January 2024

Published in Journal of Combinatorial Designs (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1809.08404

zbMATH Keywords

deep learningdropoutdropout designsplit-block design

Mathematics Subject Classification ID

Combinatorial aspects of block designs (05B05) Learning and adaptive systems in artificial intelligence (68T05) Artificial neural networks and deep learning (68T07) Applications of design theory to circuits and networks (94C30)

Abstract: Deep learning is a machine learning methodology using multi-layer neural network. A multi-layer neural network can be regarded as a chain of complete bipartite graphs. The nodes of the first partita is the input layer and the last is the output layer. The edges of a bipartite graph function as weights which are represented as a matrix. The values of i-th partita are computed by multiplication of the weight matrix and values of (i-1)-th partita. Using mass training and teacher data, the weight parameters are estimated little by little. Overfitting (or Overlearning) refers to a model that models the `training data` too well. It then becomes difficult for the model to generalize to new data which were not in the training set. The most popular method to avoid overfitting is called dropout. Dropout deletes a random sample of activations (nodes) to zero during the training process. A random sample of nodes causes more irregular frequency of dropout edges. There is a similar sampling concept in the area of design of experiments. We propose a combinatorial design on dropout nodes from each partita which balances the frequency of edges. We analyze and construct such designs in this paper.

Recommendations

• A construction for circulant type dropout designs
• scientific article; zbMATH DE number 6378127
• The dropout learning algorithm
• The stochastic delta rule: faster and more accurate deep learning through adaptive weight noise
• Surprising properties of dropout in deep networks

Cites work

• Balanced and semi-balanced incomplete split-block designs
• Bipartite decomposition of complete multipartite graphs
• Bipartite designs
• Constructions for a certain type of balanced bipartite block designs
• scientific article; zbMATH DE number 6378127 (Why is no real title available?)
• scientific article; zbMATH DE number 3490283 (Why is no real title available?)
• scientific article; zbMATH DE number 1101474 (Why is no real title available?)
• scientific article; zbMATH DE number 1382769 (Why is no real title available?)
• Incomplete Split-Block Designs
• Mixed block designs
• Optimality and constructions of incomplete split-block designs
• Split orthogonal arrays and maximum independent resilient systems of functions

Cited in

(4)

• A construction for circulant type dropout designs
• Completely (quasi-)uniform nested Boolean Steiner quadruple systems
• GDD type spanning bipartite block designs
• Pairs in nested Steiner quadruple systems

This page was built for publication: Combinatorial designs for deep learning

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6146764)