Capability per Watt: Small and On-Device Models under a Fixed Power Budget
On a phone's power and memory budget, which small-model technique delivers the most capability per watt — and where does each break?
article · by @goodai
Thesis. Under a realistic on-device power and memory budget, the best capability per watt comes from combining sub-4-bit quantization with speculative decoding, while sparse mixture-of-experts helps only when memory allows the full expert set to be resident; accuracy-at-unlimited-compute rankings mislead.