Three bugs that broke 97 tok/s
The 97 tok/s number was real. The output behind it never was. Three silent bugs, LM head substitution, weight transpose, and activation clipping, were producing garbage at full speed. This is the classic story of a fast number with wrong output, and the discipline of not shipping it.
The number was real
97 tok/s was real. It was measured on-device, stable across runs, reproducible on demand. Any benchmark harness would have printed it and called it a good day. That was the trap. The number was true and the output it produced was garbage, and garbage at full speed is not a feature.
The engine ran, the profiler was happy, and every token was wrong. Finding out took longer than any of us would like to admit.
Bug one, the LM head
The first was LM head substitution. The token probabilities were coming from the wrong head, one attached to a different part of the model than the one that should have produced the next token. The output sampled from that distribution, so it looked like text, read like text, and was the wrong text. A swap at load time, silent at runtime, hidden inside the path that turned logits into tokens.
Bug two, the weight transpose
The second was a weight transpose in the load path. A matrix was applied in its transposed form, mathematically plausible, dimensionally consistent, and silently wrong. Nothing crashed, nothing overflowed, and every layer downstream built on a tensor that was oriented the wrong way.
Bug three, activation clipping
The third was activation clipping. A hard clip sat in the pipeline where the math behind the benchmark did not include it. Values that should have passed through were cut, and the error compounded through every layer below. It was a silent change to the model, not a measured one.
| Bug | What it was | Why it was silent |
|---|---|---|
| LM head substitution | wrong head selected at load | logits and shapes looked fine |
| Weight transpose | matrix applied in transposed form | dimensions still matched |
| Activation clipping | values cut outside the math | no error, no NaN |
Why it took three
None of these crashed. No NaN, no divide by zero, no out-of-bounds read. The token stream stayed grammatical, the timing stayed flat, and the throughput stayed at 97 tok/s. The only thing that caught any of them was comparing the output token by token against a reference implementation.
./1bit-engine --bench --verify ./ref
When the first mismatch appeared, we stopped celebrating the number and started finding the rest. It was not a hunt for a bug. It was a hunt for all of them.
The discipline
This is the classic story of a fast number with wrong output, and the discipline it teaches is the one the site runs on: do not ship it. A throughput figure is only a measurement if the output behind it is real, and ours was not. The 97 tok/s number is real, and it is quarantined rather than quoted until the same engine produces a verified stream at that speed.
A fast number you cannot trust is a bug, not a benchmark.