TL;DR
To strengthen the theoretical foundation for the intuition that SGD favors flat minima, this paper proposes a geometric sharpness measure that is invariant to reparametrizations and proves its link to generalization performance.
Problem
A widely held intuition in deep learning is that SGD implicitly favors flat minima for better generalization, but standard Euclidean flatness measures (e.g., trace or max eigenvalue of the loss Hessian) are not invariant under reparametrizations that preserve the network function, undermining the theoretical basis of this narrative.
Approach
Define 'Riemannian sharpness' mathematically based on the Riemannian geometry of the statistical manifold induced by the Fisher Information Matrix (FIM) and prove its invariance under smooth, function-preserving reparametrizations.
Formalize the gradient noise of mini-batch SGD as having a covariance structure proportional to the FIM and derive the stationary distribution of the resulting stochastic differential equation.
Show that the probability mass is exponentially concentrated at Riemannian-flat minima and derive a PAC-Bayes generalization bound explicitly controlled by SR, linking this geometric bias to test performance.
Results & Contribution
Riemannian sharpness (SR) is invariant to reparametrizations, unlike Euclidean sharpness, directly addressing the critique of Dinh et al.
Experiments on MNIST and CIFAR-10 confirm that SR reliably tracks generalization in ways that Euclidean sharpness does not.