Daniel Csillag, Rodrigo Schuller, Pedro Dall'Antonia, Leonidas Guibas, Luiz Velho, Tiago Novello
A new functional gradient descent algorithm adaptively learns the representation of functional gradients during optimization, providing theoretical convergence guarantees and superior empirical performance.
Functional optimization problems are typically solved by optimizing parameters of a fixed representation (e.g., neural networks), leading to nonconvex losses. Functional gradient descent (FGD) offers strong convergence theory but is impractical because functional gradients are infinite-dimensional. Existing implementations use fixed approximations, introducing error.
The proposed algorithm adaptively learns the representation of functional gradients during optimization. At each iteration, it approximates the functional gradient as a linear combination of basis functions that are dynamically updated based on data and current model. The approximation error is explicitly incorporated into the analysis, proving convergence to a stationary point for smooth losses and to a global minimizer under smoothness plus a Polyak-Lojasiewicz condition.
Experiments on regression, PDE solving, and computer vision show consistent improvements over fixed-approximation FGD and neural network baselines in efficiency and accuracy. This is the first implementable FGD method with convergence guarantees in a general setting.