From model graph to matrix instructions.
An open software stack that runs Stories260K end to end on Proton NPU in RTL simulation, using bare-metal IREE and INT8 matrix kernels.
Language-model RTL milestone · October 2026. Verified in Verilator on the existing RTL. Batch one, bounded context, model-specific graph. These results do not establish silicon performance or general model support.
A small language model. A complete target execution.
Stories260K is a pretrained Llama-style decoder with roughly 260,000 parameters and five transformer layers. The verified run generates 16 greedy tokens, with the decoder, KV cache and token selection executing on the simulated target.
Actual target output: “Once upon a time, there was a little girl named Lily. She”
- 0116 — Tokens generated on target
- 0236 — Matrix-backed projections per token
- 038,192 — Logits checked against reference
Batch one, context bounded to 64 tokens, generation from the beginning-of-sequence token. This is a tiny-model bring-up milestone.
A compiled graph calls the kernels on the target.
Python validates the checkpoint, packs INT8 weights and emits the model-specific MLIR graph. IREE compiles it to VMFB and static RISC-V dispatch code, linked with the runtime, kernels and weights into one bare-metal executable.
The IREE VM and synchronous local HAL invoke 68 semantic dispatches per decoder step. The application drives the outer token loop. Linux is used for host tools, not as an operating system on the target.
The integration uses custom CPU dispatches. Standard IREE ukernel selection, a new matrix dialect and automatic lowering of arbitrary imported models are not implemented.
Key linear operations reach the matrix engine.
The matrix engine executes Q, K, V, attention output, FFN and vocabulary projections. Attention scores, softmax, value aggregation and nonlinear operations remain scalar FP32.
Proton kernels quantize activations, transfer tiles through CPU MMIO, and issue mzero and mmacc. The engine computes signed INT8 products with INT32 accumulation; kernels rescale the results to FP32. Submission is synchronous, with no matrix DMA.
- 0136 linear projections accelerated per token
- 0265,152 mmacc and 14,048 mzero instructions in the 16-token run
- 03All 79,200 commands reconciled across issue, completion, response and retirement
The model currently uses scalar and matrix execution. Compiler-generated RVV remains disabled pending a separately documented verifier stall investigation.
Check the numbers, then inspect the execution.
All 8,192 target logits were checked against an independent quantized reference, with a maximum absolute difference of 0.00000763. Kernel arithmetic, partial tiles, a representative waveform and deliberate failure cases were checked separately.
The first decoder invocation used 2,327,901 cycles with matrix kernels versus 6,713,729 with scalar kernels, using identical W8A8 arithmetic and bit-identical logits. This 2.88× comparison covers that invocation only; it excludes runtime setup, application transfers, checking and console output.
Functional RTL cycle evidence is not silicon throughput or a full-generation speedup. Quantization error relative to FP32 is reported separately.
A proven path with an explicit supported scope.
The input is a pinned llama2.c Stories260K checkpoint and tokenizer. The graph generator is specific to that architecture; the SDK does not yet accept arbitrary ONNX, PyTorch or GGUF models.
The existing target has 16 MiB of simulated SRAM, with no DDR controller. Weights are linked into the executable. The kernel currently bounds input and output dimensions to 512; total capacity must also cover runtime, activations, KV cache, heap and stack.
This deployment required no RTL or memory-size changes. Larger models, CNNs and ViTs need additional graph import, operator coverage, shape support and validation.
Start from the evidence. Extend a clear boundary.
The public repository separates compiler integration, runtime, kernels, model preparation, target manifests and verification. Hardware revisions are pinned, so a new matrix layout or memory map can be qualified through a new adapter and target contract.
Next compiler work connects standard framework import and general operator lowering to the verified kernel path. Standard ukernel integration and broader model support can build on this foundation; llama.cpp remains a separate possible integration.