Optimization in machine learning: a distribution-space approach (Q6575304)

This paper presents a novel perspective on optimization problems in machine learning by framing them as convex functional minimization over a function space, subject to non-convex constraints introduced by model parameterization. The authors propose a distribution-space approach that transforms the original non-convex optimization problem into a convex one by considering the space of probability distributions over the training parameters. This reformulation allows for the development of numerical algorithms based on mixture distributions, which can perform approximate optimization directly in the distribution space. The paper not only establishes the consistency of this approximation but also demonstrates the numerical efficacy of the proposed algorithm through simple examples, highlighting its potential as an alternative approach to large-scale optimization in machine learning.