Suggest edit — Limited Samples and Infinite Compute

Title

Name

Note

---
title: "Limited Samples and Infinite Compute"
type: concept
source: https://www.jemoka.com/posts/kbhlots_of_samples_and_infinite_compute/
confidence: high
status: active
---

One-Liner &ldquo;If you have infinite compute but limited samples, how do you pretrain?&rdquo;
Novelty closed-form best approach which reduces loss given the current data budget Notable Methods Outline Takes 200 million tokens, 300 million parameters, from a corpus Measure validation loss Vary training recipes regularized parameter scaling Stick in some more weight decay: more parameters, more weight decay
Model WDK 150M 0.8 300M 1.6 600M 3.2 1.4B 3.2 ensembling Train a bunch of seperate models (i.e. with random shuffles?) and then parameter merge; different initialization, etc.
Key Results Various training recipes gets various results:
epoching: overfitting eventually, and larger models does so quickly regularized parameter scaling: faster improvements in loss at increased parameter scales ensemble: lower asymptote Takeaways training recipes current aprpocah overfit regularization can scaling law esembling decreases loss at asymptote inference time efficiency This is an ad for ensembling, and that&rsquo;s expensive! You can distill the model down to a single dense size and a 4-ensemble distilled down densely can even outperform an optimal 4 ensemble.
Self distillation can be good as well!
continual pre-training Training with ensembling gives efficiency gains even at large data scale regimes (4B tokens, etc.)
New Concepts Notes