Library · 16-ml-optimization
ML & optimization
| Title | Peer | Link |
|---|---|---|
| Decoupled Weight Decay Regularization (AdamW) | ◦ preprint | arxiv.org/abs/1711.05101 |
| Sharpness-Aware Minimization (SAM) | ✓ peer | arxiv.org/abs/2010.01412 |
| Symbolic Discovery of Optimization Algorithms (Lion) | ◦ preprint | arxiv.org/abs/2302.06675 |
| Shampoo: Preconditioned Stochastic Tensor Optimization | ◦ preprint | arxiv.org/abs/1802.09568 |
| Large Batch Optimization for Deep Learning (LAMB) | ◦ preprint | arxiv.org/abs/1904.00962 |
| Optimizing Neural Networks with Kronecker-factored Approximate Curvature (K-FAC) | ◦ preprint | arxiv.org/abs/1503.05671 |
| The Lottery Ticket Hypothesis | ◦ preprint | arxiv.org/abs/1803.03635 |
| Neural Tangent Kernel | ◦ preprint | arxiv.org/abs/1806.07572 |
| Deep Double Descent | — unref | arxiv.org/abs/1912.02292 |
| Grokking: Generalization Beyond Overfitting | ◦ preprint | arxiv.org/abs/2201.02177 |
| Representation Learning: A Review and New Perspectives | ◦ preprint | arxiv.org/abs/1206.5538 |
| A Simple Framework for Contrastive Learning (SimCLR) | ◦ preprint | arxiv.org/abs/2002.05709 |
| Momentum Contrast (MoCo) | ◦ preprint | arxiv.org/abs/1911.05722 |
| XGBoost: A Scalable Tree Boosting System | ◦ preprint | arxiv.org/abs/1603.02754 |
| Learning representations by back-propagating errors | ✓ peer | doi.org/10.1038/323533a0 |
| Support-Vector Networks (SVM) | ✓ peer | doi.org/10.1007/BF00994018 |
| Adam: A Method for Stochastic Optimization | ◦ preprint | arxiv.org/abs/1412.6980 |
| Deep learning (review) | ✓ peer | doi.org/10.1038/nature14539 |
| Improving Neural Networks by Preventing Co-adaptation (Dropout) | ◦ preprint | arxiv.org/abs/1207.0580 |
| Generative Adversarial Networks (GAN) | ◦ preprint | arxiv.org/abs/1406.2661 |
| Auto-Encoding Variational Bayes (VAE) | ◦ preprint | arxiv.org/abs/1312.6114 |
| Denoising Diffusion Probabilistic Models (DDPM) | ◦ preprint | arxiv.org/abs/2006.11239 |
| Extracting and Composing Robust Features with Denoising Autoencoders | — unref | doi.org/10.1145/1390156.1390294 |
| Neural Discrete Representation Learning (VQ-VAE) | ◦ preprint | arxiv.org/abs/1711.00937 |
| NVAE: A Deep Hierarchical Variational Autoencoder | ◦ preprint | arxiv.org/abs/2007.03898 |
| Maximum Likelihood from Incomplete Data via the EM Algorithm (GMM / EM) | ✓ peer | doi.org/10.1111/j.2517-6161.1977.tb01600.x |
| Convex Optimization | — | |
| Numerical Optimization (2e) | — | |
| The Elements of Statistical Learning (2e) | — | |
| Probabilistic Machine Learning: An Introduction | — | |
| VICReg: Variance-Invariance-Covariance Regularization for SSL | ◦ preprint | arxiv.org/abs/2105.04906 |
| Bootstrap Your Own Latent (BYOL) | ◦ preprint | arxiv.org/abs/2006.07733 |
| Masked Autoencoders Are Scalable Vision Learners (MAE) | ◦ preprint | arxiv.org/abs/2111.06377 |
| Muon is Scalable for LLM Training | ◦ preprint | arxiv.org/abs/2502.16982 |
| Sophia: A Scalable Stochastic Second-order Optimizer for LM Pre-training | ◦ preprint | arxiv.org/abs/2305.14342 |
| Lookahead Optimizer: k steps forward, 1 step back | ◦ preprint | arxiv.org/abs/1907.08610 |
| The Road Less Scheduled (Schedule-Free) | ◦ preprint | arxiv.org/abs/2405.15682 |
| Adafactor: Adaptive Learning Rates with Sublinear Memory Cost | ✓ peer | arxiv.org/abs/1804.04235 |
| ADADELTA: An Adaptive Learning Rate Method | ◦ preprint | arxiv.org/abs/1212.5701 |
| Adaptive Subgradient Methods (Adagrad) | jmlr.org/papers/v12/duchi11a.html | |
| Opening the Black Box of Deep Neural Networks via Information | ◦ preprint | arxiv.org/abs/1703.00810 |
| Prevalence of Neural Collapse during the Terminal Phase of Training | ◦ preprint | arxiv.org/abs/2008.08186 |
| On Large-Batch Training: Generalization Gap and Sharp Minima | ◦ preprint | arxiv.org/abs/1609.04836 |
| Gradient Descent Typically Occurs at the Edge of Stability | ◦ preprint | arxiv.org/abs/2103.00065 |
| Artificial Intelligence: A Modern Approach (4e) | — | |
| Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning | ◦ preprint | arxiv.org/abs/2608.11690 |
| Weaves, Wires, and Morphisms: Formalizing and Implementing the Algebra of Deep Learning | arxiv.org/abs/2604.07242 |