Rethinking arithmetic for deep neural networks

DOI10.1098/RSTA.2019.0051zbMATH Open1462.68168arXiv1905.02438OpenAlexW2999134573WikidataQ92754316 ScholiaQ92754316MaRDI QIDQ4993501FDOQ4993501

Authors: George A. Constantinides 

Publication date: 15 June 2021

Published in: Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences (Search for Journal in Brave)

Abstract: We consider efficiency in the implementation of deep neural networks. Hardware accelerators are gaining interest as machine learning becomes one of the drivers of high-performance computing. In these accelerators, the directed graph describing a neural network can be implemented as a directed graph describing a Boolean circuit. We make this observation precise, leading naturally to an understanding of practical neural networks as discrete functions, and show that so-called binarised neural networks are functionally complete. In general, our results suggest that it is valuable to consider Boolean circuits as neural networks, leading to the question of which circuit topologies are promising. We argue that continuity is central to generalisation in learning, explore the interaction between data coding, network topology, and node functionality for continuity, and pose some open questions for future research. As a first step to bridging the gap between continuous and Boolean views of neural network accelerators, we present some recent results from our work on LUTNet, a novel Field-Programmable Gate Array inference approach. Finally, we conclude with additional possible fruitful avenues for research bridging the continuous and discrete views of neural networks.

Full work available at URL: https://arxiv.org/abs/1905.02438

Recommendations

• GXNOR-Net: training deep neural networks with ternary weights and activations without full-precision memory under a unified discretization framework
• Computing with discrete multi-valued neurons
• Arithmetic unit design for neural accelerators: cost performance issues
• scientific article; zbMATH DE number 6982943
• 2-1 addition and related arithmetic operations with threshold logic

zbMATH Keywords

artificial intelligenceacceleratorapplied mathematicscomputational mathematicscomputingdeep neural networkfield-programmable gate array

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Artificial neural networks and deep learning (68T07) Graph theory (including graph drawing) in computer science (68R10) Mathematical problems of computer architecture (68M07) Networks and circuits as models of computation; circuit complexity (68Q06)

Cites Work

• Machine learning. A probabilistic perspective
• Deep learning
• On a theory of computation and complexity over the real numbers: 𝑁𝑃- completeness, recursive functions and universal machines
• Modeling by shortest data description
• Graph theory with applications
• Accuracy and Stability of Numerical Algorithms
• The Problem of Simplifying Truth Functions
• Title not available (Why is that?)
• Title not available (Why is that?)
• Reconfigurable computing. The theory and practice of FPGA-based computation.
• Title not available (Why is that?)
• A Survey of Combinatorial Gray Codes
• Title not available (Why is that?)
• Computer arithmetic algorithms.
• Title not available (Why is that?)
• A continuous characterization of the maximum-edge biclique problem
• On-Line Algorithms for Division and Multiplication
• Stochastic rounding and reduced-precision fixed-point arithmetic for solving neural ordinary differential equations

Cited In (2)

• GXNOR-Net: training deep neural networks with ternary weights and activations without full-precision memory under a unified discretization framework
• Arithmetic circuits, structured matrices and (not so) deep learning

This page was built for publication: Rethinking arithmetic for deep neural networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4993501)