Barcodes as summary of loss function topology

DOI10.1134/S1064562423701570arXiv1912.00043OpenAlexW4393172380MaRDI QIDQ6124396FDOQ6124396

Authors: Serguei Barannikov, A. A. Korotin, D. A. Oganesyan, D. I. Emtsev, E. V. Burnaev 

Publication date: 27 March 2024

Published in: Doklady Mathematics (Search for Journal in Brave)

Abstract: We apply the canonical forms (barcodes) of gradient Morse complexes to explore topology of loss surfaces. We present a novel algorithm for calculations of the objective function's barcodes of local minima. We have conducted experiments for calculating barcodes of local minima for benchmark functions and for loss surfaces of neural networks. Our experiments confirm two principal observations for loss surfaces of neural networks. First, the barcodes of local minima are located in a small lower part of the range of values of loss function of neural networks. Second, increase of the neural network's depth brings down the barcodes of local minima. This has natural implications for the neural network learning and the generalization ability.

Full work available at URL: https://arxiv.org/abs/1912.00043

Recommendations

• Learning representations of persistence barcodes
• 1-dimensional topological invariants to estimate loss surface non-convexity
• A framework for differential calculus on persistence barcodes
• The loss surfaces of neural networks with general activation functions
• Archetypal landscapes for deep neural networks

zbMATH Keywords

persistent homologyneural networksMorse theorypersistence barcodesloss surface

Mathematics Subject Classification ID

Persistent homology and applications, topological data analysis (55N31)

Cites Work

• Title not available (Why is that?)
• Stability of persistence diagrams
• Title not available (Why is that?)
• Differentiable dynamical systems
• Title not available (Why is that?)
• Lectures on Morse theory, old and new
• A literature survey of benchmark functions for global optimisation problems
• Geometry helps in bottleneck matching and related problems
• Precise Arrhenius law for \(p\)-forms: the Witten Laplacian and Morse-Barannikov complex
• Scalar field analysis over point cloud data
• Entropy-SGD: biasing gradient descent into wide valleys
• Barcodes and area-preserving homeomorphisms

Cited In (2)

• Functional loops: monitoring functional organization of deep neural networks using algebraic topology
• 1-dimensional topological invariants to estimate loss surface non-convexity

This page was built for publication: Barcodes as summary of loss function topology

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6124396)