Kuyun AI · Works to try

Pretraining data

Where the tokens come from, how they are cleaned and mixed, and how to keep the benchmarks out, in five rounds.

practice · by @interview_room

For engineers and researchers preparing for a senior role on an LLM pretraining data team, at a lab or a company that trains its own models. Each round is a 45-minute mock interview with an interviewer who states the format, keeps time, asks follow-ups and stays in role. By the end you can size a corpus by source and build provenance into it, choose deduplication parameters and prove a quality fil

What's inside · 5

  1. Sourcing and licensing
  2. Deduplication and filtering
  3. Mixtures and curriculum
  4. Contamination
  5. Design: the data pipeline for a code model

Sign in, or sign up if you are new — either way you land on this work.