NPU v12: M=32 batch decode at 97 tok/s
One session took decode from 244 to 10 ms per token, a 24x speedup, and put v12 at 97 tok/s with M=32 batch decode, 46% ahead of FLM Kraken Point. OpenMP attention plus an LM head, all C++23, zero Python.
One session
V12 did not arrive through a long campaign of tuning. In a single session decode went from 244 to 10 ms per token, a 24x speedup, by moving to M=32 batch decode and clearing the two biggest stalls in the path.
The numbers
| measure | value |
|---|---|
| batch width M | 32 |
| decode before | 244 ms/tok |
| decode after | 10 ms/tok |
| speedup | 24x |
| throughput | 97 tok/s |
| vs FLM Kraken Point | +46% |
The M=32 number matters: at M=1 the NPU spends most of a dispatch waiting. Batch decode spreads the fixed launch cost across 32 tokens instead of one.
What changed
Two changes did the work. Attention moved to an OpenMP parallel path so the batch width actually scales, and the LM head moved out of a separate round trip and into the same steady pipeline. Same hardware, less time standing still.
The stack
The whole engine is full C++23 with zero Python in the inference path. No interpreter, no glue process, no second runtime to break. The kernel is a single binary that talks to the NPU directly.
The session result
This is one session's before and after, written down while it was still fresh. A session result is not a guarantee, but it is the kind of record a build log is for.