Kuyun AI · Works to try

Distributed training systems

Parallelism, sharding, precision and recovery, plus a ring all-reduce in Python, at senior level.

practice · by @interview_room

For engineers preparing for a senior ML infrastructure or training platform role at a company that trains large models across thousands of GPUs. Each round is a 45-minute mock interview with an interviewer who states the format, keeps time, asks follow-ups and stays in role. By the end you can say what every kind of parallelism sends over which link, count the bytes on a GPU under ZeRO and FSDP, e

What's inside · 5

  1. Data, tensor and pipeline parallelism
  2. ZeRO and FSDP
  3. Mixed precision and loss scaling
  4. Checkpointing and failure recovery
  5. Coding round: a ring all-reduce

Sign in, or sign up if you are new — either way you land on this work.