Accurate Computation of the Log-Sum-Exp and Softmax Functions

DOI10.1093/IMANUM/DRAA038MaRDI QIDQ6324880arXivFDO

Authors Pierre Blanchard, Desmond J. Higham, Nicholas J. Higham

Publication date 8 September 2019

Published in arXiv

Full work available at URL https://doi.org/10.1093/IMANUM/DRAA038

Mathematics Subject Classification ID

Computation of special functions and constants, construction of tables (65D20)

arXiv classification

cs.NAmath.NA

Abstract: Evaluating the log-sum-exp function or the softmax function is a key step in many modern data science algorithms, notably in inference and classification. Because of the exponentials that these functions contain, the evaluation is prone to overflow and underflow, especially in low precision arithmetic. Software implementations commonly use alternative formulas that avoid overflow and reduce the chance of harmful underflow, employing a shift or another rewriting. Although mathematically equivalent, these variants behave differently in floating-point arithmetic. We give rounding error analyses of different evaluation algorithms and interpret the error bounds using condition numbers for the functions. We conclude, based on the analysis and numerical experiments, that the shifted formulas are of similar accuracy to the unshifted ones and that the shifted softmax formula is typically more accurate than a division-free variant.

Cited in

(3)

• Implementation of Hyperelastic Physics-Augmented Neural Networks in the Explicit Finite Element Codes Simcenter Radioss and OpenRadioss with Applications to Impact Events
• Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training
• Deterministic and stochastic particle methods for the Fokker-Planck equation in S--formulation with application to point set registration

This page was built for publication: Accurate Computation of the Log-Sum-Exp and Softmax Functions

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6324880)