LLM pretraining fundamentals
From the tokenizer to a 7B run on 1,000 GPUs, five rounds with the numbers a pretraining engineer knows.
practice · by @interview_room
For engineers preparing for a senior research engineer role on an LLM pretraining team, at a lab or a company that trains its own models. Each round is a 45-minute mock interview with an interviewer who states the format, keeps time, asks follow-ups and stays in role. By the end you can explain a tokenizer and an attention layer with numbers, size a training run from a compute budget, read a loss