Suggest edit — Pretraining Data

Title

Name

Note

---
title: "Pretraining Data"
source: https://www.jemoka.com/posts/kbhpretraining_data/
---

Small scale: DCLM Baseline Data Legally friendly data: CommonPile Web scraped data with quality group: NemoTron people measure isoFLOPS
Problems of pre-training data pre-training influence downstream capabilities &hellip;and therefore can escape into model generation real world users expect novelty Changes in Distribution Big Pretraining Data GPT2 deduplicated data Removed Wikipedia (to prevent data leak) Heuristic based cleaning GPT3 Deduplicated based on leaked data Llama the usual spheal
removed high perplexity data using wiki n-gram model removed non-English deduplicated Llama 2 removed high volue of PII Removed non-english Pretraining Curation Decisions what to include what is the timestamp being scraped heuristic based cleaning? data cleaning? etc. language filtering (only take English?) PII removal dedup Toxicity + SafeURL filtering &ldquo;quality filtering&rdquo; sampling distributions Change in Model Age Good alignment shown between validation year and pre-training year, even mixing in older data.
Implication: &ldquo;fine-tuned T5 may still be worse than fine-tuned llama, because T5 was pretrained using older data&mdash;despite even if FTing is newer&rdquo;
Change in Toxicity Filtering toxicity made the model worst at spotting toxicity.
Change in Data Distribution out of domain answers do worse on out of domain results
Reduce Memorization de-duplication using approximate matching think carefully for multiple-epoch training (what is ok to memorize?) remove sensitive memorization from pre-training data Two iffy strategies:
Check for memorization Trivial style transfers can get around safety checks &ldquo;do the [copyrighted thing] in French&rdquo;; &ldquo;do the [copyrighted thing] with double the spaces&rdquo;.
Use RLHF or something &ldquo;hide flaws, and not eliminate them&rdquo;&mdash;edge case problems doesn&rsquo;t eliminate the underlying vulnerability.