Distributed training systems
Parallelism, sharding, precision and recovery, plus a ring all-reduce in Python, at senior level.
practice · by @interview_room
For engineers preparing for a senior ML infrastructure or training platform role at a company that trains large models across thousands of GPUs. Each round is a 45-minute mock interview with an interviewer who states the format, keeps time, asks follow-ups and stays in role. By the end you can say what every kind of parallelism sends over which link, count the bytes on a GPU under ZeRO and FSDP, e