|
Statut |
Confirmé |
|
Série |
SEM-LPTMC |
|
Domaines |
cond-mat |
|
Date |
Mardi 7 Avril 2026 |
|
Heure |
10:45 |
|
Institut |
LPTMC |
|
Salle |
campus Jussieu, couloir 12-13, 5ème étage, salle 5-23 |
|
Nom de l'orateur |
Urbani |
|
Prenom de l'orateur |
Pierfrancesco |
|
Addresse email de l'orateur |
|
|
Institution de l'orateur |
IPhT |
|
Titre |
Separation of timescales controls feature learning and overfitting in large neural networks |
|
Résumé |
To understand the inductive bias and generalization capabilities of large, overparameterized machine learning models, it is essential to analyze the out-of-equilibrium dynamics of their training algorithms. Using dynamical mean field theory we investigate the learning dynamics of large two-layer neural networks. Our findings reveal that, for networks with a large width, the training process exhibits a separation of timescales phenomenon. This leads to several key observations: 1. The emergence of a slow timescale linked to the growth of a carefully defined complexity measure of the network; 2. An inductive bias favoring low complexity when the initial model complexity is sufficiently small; 3. A dynamical decoupling between feature learning and overfitting phases; 4. A non-monotonic trend in test error, characterized by a "feature unlearning" regime at later stages of training. |
|
Numéro de preprint arXiv |
|
|
Commentaires |
|
|
Fichiers attachés |
|
