I reverse-engineered AMD's NPU stack in 4 days
A laptop, a disassembler, and no docs. AMD shipped a 50 TOPS XDNA 2 NPU locked behind a closed-source runtime: 22 proprietary libraries, 209 xclbin bitstreams, zero documentation. We took it apart and replaced it with open C++. This is the story, and the unavoidable corrections.
The wall
The machine arrives with a 50 TOPS NPU that is, for all practical purposes, a paperweight unless you have the keys. The only way to drive it is FastFlowLM, a closed-source runtime. That runtime is 22 .so files. Behind it sit 209 xclbin bitstreams, pre-compiled kernels for architectures we are not allowed to see. And there is zero documentation. No ISA. No register map. No example. Nothing.
So the question is not "is it fast," but "can anyone other than AMD make it go at all." We decided the honest answer should be yes.
The dismantling
The whole thing came apart in four days: each proprietary layer disassembled, each dispatch traced, each xclbin decoded until the operation it performed was obvious. Point the disassembler at the runtime, then point a C++ compiler at the result. Around ~600 hours of engineering, all of it open source and MIT.
The thing that came out the other side is a single small binary that drives the NPU without FastFlowLM. It is not just a re-implementation but the better half of an engine, and it is the reason the rest of this site exists.
The sprint: 244 ms/tok to 3.4 ms/tok
The first usable version was slow, and the path down was a series of measured steps, not one lucky leap. Each number below is on-device.
| Version | ms/tok | what changed |
|---|---|---|
| v3 | 244 | baseline |
| v6 | 50 | batch-4 decode + OpenMP LM head |
| v8 | 27 | M=8 batch decode |
| v9 | 16 | M=16 batch decode |
| v12 | 10 | 97 tok/s, 24x |
| fused engine | 3.4 | 291 tok/s, 38 KB binary |
The breakthrough is in the footnote, not the headline. A microsecond probe showed that 1363 µs of every 1363 µs GEMM dispatch is spent in launch-and-wait; only 0.5-5 µs is the actual multiply. The NPU is 99% idle in the batch dimension at M=1, and every single token pays it. So we stopped dispatching one token at a time.
Generate the top-16 candidates from the LM head logits, then run them through one batched forward pass: same 1334 µs compute, sixteen times the work. That single idea is the whole 15x.
One dispatch per layer
Summing, attention, and feed-forward were still separate round-trips. The fused layer engine chains QKV, attention, and FFN into a single xclbin dispatch: 1 call per transformer layer instead of 5/6. That is the difference between a working engine and a fast one, and it is the architecture this site ships today.
The corrections
This is the part most projects quietly skip, and it is the reason to trust the rest of these numbers. A lot of the early figures were real process and wrong evidence. We quarantined them rather than pretend.
Several historical throughput and speedup figures were later found unsourced and are quarantined in benchmarks/latest.json _unverified. The "NPU is 132x more efficient than the GPU" claim was retracted. The projected DSpark 572 tok/s was disproven end-to-end: it measured 0.1-0.2 tok/s at 0% draft acceptance. And the celebrated 97 tok/s was a real number with garbage behind it: three silent bugs (LM head substitution, weight transpose, activation clipping) were producing nonsense output.
That is not a disclaimer tacked on to be safe. It is the record: what we measured, what it actually meant, and when we got it wrong. An engine you can trust is worth more than a story that always wins.
The point
Four days, a disassembler, and a C++ compiler turned a locked-down NPU into an open-source engine. The number that matters is not the tok/s but the fact that the keys exist now, and nobody is holding them.