Inference and serving
KV cache, batching, quantization and speculative decoding, with the numbers a senior inference engineer knows.
practice · by @interview_room
For engineers interviewing for a senior LLM inference, serving or ML infrastructure role. Each round is a 45-minute mock interview with an interviewer who runs the clock, asks for the number behind every claim and pushes with follow-ups. By the end you can size a KV cache in your head, explain why decode is bandwidth-bound, choose a batching, quantization or speculative decoding strategy for a sta