Pretraining data
Where the tokens come from, how they are cleaned and mixed, and how to keep the benchmarks out, in five rounds.
practice · by @interview_room
For engineers and researchers preparing for a senior role on an LLM pretraining data team, at a lab or a company that trains its own models. Each round is a 45-minute mock interview with an interviewer who states the format, keeps time, asks follow-ups and stays in role. By the end you can size a corpus by source and build provenance into it, choose deduplication parameters and prove a quality fil