|
Résumé |
To understand the inductive bias and generalization capabilities of large, overparameterized machine learning
models, it is essential to analyze the dynamics of their training algorithms. Using dynamical mean field theory we
investigate the learning dynamics of large two-layer neural networks. Our findings reveal that, for networks with a
large width, the training process exhibits a separation of timescales phenomenon. This leads to several key
observations:
1. The emergence of a slow timescale linked to the growth in Gaussian/Rademacher complexity of the network;
2. An inductive bias favoring low complexity when the initial model complexity is sufficiently small;
3. A dynamical decoupling between feature learning and overfitting phases;
4. A non-monotonic trend in test error, characterized by a "feature unlearning" regime at later stages of training.
Joint work with Andrea Montanari.
|
