Seminar: Memorization and Learning Dynamics of Neural Networks
- Date
- September 24, 2025
- Time
- 11:10 AM EDT - 12:00 PM EDT
- Location
- ENG-210 and virtually via zoom
- Open To
- All faculty, staff, students and guests are welcome to attend
- Contact
- Pawel Pralat (pralat@torontomu.ca)
Speaker: Daniel Willhalm, TMU
Title: Memorization and Learning Dynamics of Neural Networks
Abstract: We examine the role of memorization in the learning dynamics of neural networks, focusing on how it relates to generalization. The talk will outline key ideas in neural network classification, including the risk of overfitting when minimizing empirical loss. It will explore the observation that neural networks tend to learn shared features before they begin to memorize individual data points. However, memorization is not always a negative outcome; for realistic datasets with a ``long tail'' of rare examples, models must memorize these points to achieve optimal generalization. Finally, we will introduce methods for measuring memorization, like prediction depth, second split forgetting, and area under the margin and talk about some interesting research papers.