TL;DR: The Triton kernels that ship in rotalabs-accel have an educational companion repository, triton-kernels (MIT). Every kernel comes with a benchmark script and a roofline plot explaining why it is fast and, just as important, where it stops being fast. Two of them, a fused MoE dispatch and a 4-bit W4A16 GEMM, are on the Hugging Face Kernel Hub and run on NVIDIA and AMD with no CUDA. The W4A16 kernel beats cuBLAS FP16 by 1.2 to 1.3x in the decode regime on current models. This post is about the roofline part.

Why a trust lab ships kernels

Everything we publish about evaluation, from quality-diversity red-teaming to statistical rigor in LLM evals, turns into an inference bill. Confidence intervals need repeated runs. Adversarial search needs thousands of them. And the models we most want to evaluate in 2026 are mixture-of-experts and 4-bit quantized, which is exactly where the fast kernels are CUDA-only and vendor-locked. When your GPUs are whatever you can get that month, portability is not a nicety.

So we wrote the kernels in OpenAI Triton. The packaged, drop-in form is rotalabs-accel. The design notes, benchmarks, and roofline plots live in triton-kernels, which sits under a personal GitHub account rather than the rotalabs org because it is deliberately educational, MIT-licensed, and predates the package.

This is engineering, not trust research. But it follows the same rule we apply to everything else: publish the measurement, publish the script that produced it, and say plainly what the number does not cover.

The roofline habit

LLM inference is memory-bandwidth bound. A 7B model in FP16 has to stream 14 GB of weights through the GPU for every forward pass. On an A100 that takes about 7 ms, while the arithmetic takes well under a tenth of that. Most transformer ops have an arithmetic intensity below 10 FLOPs per byte.

That makes a bare speedup number almost meaningless. “8x faster than PyTorch” could mean a brilliant kernel or a terrible baseline. A roofline plot fixes this by placing each kernel at its arithmetic intensity and achieved throughput against the machine’s two ceilings: memory bandwidth on the left, peak compute on the right. You can read two things off it directly. How far the kernel is from the ceiling, and which ceiling it is under.

Take RMSNorm. The Triton version is 8.1x faster than PyTorch’s, which sounds like sorcery until you see the bandwidth column. PyTorch launches several kernels and writes intermediates, achieving 168 GB/s. The fused kernel does one pass at 1365 GB/s, or 88% of the A100-40GB’s peak. The speedup is not clever math. It is the removal of wasted memory traffic, and the roofline tells you there is roughly 12% left to find before you need a different algorithm.

Kernel	Speedup vs PyTorch	What the roofline says
rmsnorm	8.1x	88% of peak bandwidth. Near optimal.
rmsnorm_residual_fused	6.0x	83% of peak bandwidth. One intermediate tensor eliminated.
swiglu_fused	1.6x	79% of peak bandwidth. PyTorch was already decent here.
int8_gemm	~1.0x	31% of peak bandwidth. Limited by dequant overhead, not memory. Buys 2x weight memory, not speed.
fused_moe_forward	up to 9.1x	Expert FFN at 45% bandwidth and 37% compute simultaneously.
w4a16_gemm	1.2 to 1.3x vs cuBLAS FP16 (decode)	Wins below the ridge point, loses above it. See below.

RMSNorm, SwiGLU and INT8 numbers are from an A100-40GB. MoE and W4A16 are from an A100-80GB. Every row has a script in the repo’s benchmarks/ directory that regenerates it.

W4A16: winning in decode, losing in prefill

W4A16 is 4-bit weight-only quantization with FP16 activations, the GPTQ and AWQ format most deployed models ship in. The fast kernels for it (Marlin, exllama, the AWQ kernels) are hand-written CUDA. This is a pure Triton implementation that dequantizes inside the K-loop, so weights cross the memory bus at 4 bits and are expanded to FP16 only in registers.

The plot is the whole story. Because it moves 4x less weight data, W4A16 sits at roughly 4x the arithmetic intensity of FP16 at any given batch size. Three regimes fall out of that:

• Decode (M = 1 to 8). Both kernels sit far below the roofline, because a skinny GEMM launches too few programs to saturate HBM. Neither is bandwidth-bound; both are parallelism-bound. The plain W4A16 kernel actually loses to cuBLAS here despite moving less data. Dispatching to a split-K variant for small M, which splits the K reduction across programs, restores the parallelism and gives the 1.22 to 1.29x win.
• Small prefill (M = 32). Roughly a draw. W4A16’s operating point is approaching the ridge.
• Prefill (M = 128 and up). W4A16 crosses the ridge point (arithmetic intensity 153) into compute-bound territory, where cuBLAS’s tensor-core feeding wins outright. At M = 128 it is 78 TFLOPS against cuBLAS’s 165.

Measured on current-generation shapes, group size 128, split-K dispatch autotuned per shape:

Shape (K, N)	Model	M = 1	M = 8
(7168, 18432)	DeepSeek-V3.2 FFN	1.28x	1.19x
(5120, 25600)	Qwen3-32B FFN	1.29x	1.16x
(8192, 28672)	Llama 3.3 70B FFN	1.28x	1.21x
(7168, 2048)	DeepSeek-V3.2 MoE expert	1.22x	1.14x

The honest part: peak achieved weight-stream bandwidth is about 24% of the A100’s HBM peak. The theoretical ceiling for a 4x traffic reduction is a 4x speedup, and this kernel gets nowhere near it. The gap is the INT4 unpack, dequant, and reshape overhead, plus feeding the tensor cores from a simple weight layout. Marlin closes that gap with a bespoke permuted layout and async pipelining, and a simple-layout Triton kernel is not going to match it. What you get instead is a kernel that runs on AMD unchanged, beats FP16 where decode actually happens, and carries a 4x weight-memory reduction at every batch size.

MoE dispatch: five launches instead of twenty-four

The fused MoE kernel does the full forward pass, router scoring through top-k gating, permutation, grouped expert GEMMs, and weighted un-permutation, in five kernel launches. The PyTorch reference for Mixtral-8x7B is 24 cuBLAS calls in a Python loop.

Tokens	PyTorch reference	Triton fused	Speedup
1	9.32 ms	1.02 ms	9.1x
128	13.14 ms	2.27 ms	5.8x
512	25.92 ms	7.93 ms	3.3x
4096	122.82 ms	32.31 ms	3.8x

The 9.1x at one token is mostly launch overhead disappearing, which the roofline makes obvious: router and permute are under 5% of the time at any batch size, and the expert FFN is over 95%. The gain that matters at scale is fusing the gate and up projections into one kernel that shares its A-tile loads through L2 and applies SiLU in FP32 registers. That deletes two intermediate buffers and about 35% of global memory traffic, and the benefit grows with batch size because those buffers scale with token count.

Against Megablocks, the CUDA block-sparse state of the art, the Triton kernel is faster below 128 tokens, at 89 to 93% of its throughput at 512, and at 56 to 61% at 2048 where the hand-tuned CUDA extracts more from the tensor cores. DeepSeek-V3’s 256-expert configuration is the hard case: at 512 tokens each expert sees about two tokens, the grouped GEMM cannot fill the tensor cores, and the roofline shows the expert stage flipping from compute-bound to memory-bound. Knowing that is what tells you the next optimization is a different schedule, not more autotuning.

All 162 tests pass on an AMD MI300X under ROCm with zero code changes. AMD performance numbers are still future work, and we say so in the repo.

Using them

The two hub kernels load directly, no install:

from kernels import get_kernel

# trust_remote_code=True is required until the publisher is on the trusted list
moe = get_kernel("bassrehab/moe-dispatch", version=1, trust_remote_code=True)
w4a16 = get_kernel("bassrehab/w4a16", version=1, trust_remote_code=True)

For the drop-in nn.Module replacements with automatic PyTorch fallback when Triton or a GPU is missing, use the package:

pip install rotalabs-accel[triton]

And to regenerate any number in this post, clone triton-kernels and run the matching script under benchmarks/. The roofline plots come from benchmarks/roofline/.

What is not there

No Marlin-class weight layout, so the 4-bit kernel leaves most of its theoretical bandwidth win on the table. No down-projection and scatter fusion in the MoE path, because Triton cannot index a 2D accumulator by scalar row. No AMD performance characterization yet. Each of these is listed in the repo’s docs with the reason, which we think is the more useful half of a benchmark writeup.

Questions, or a shape you would like benchmarked? [email protected].