KingJones

Quantizing large models for hardware most people don't benchmark on — and publishing the results that didn't work.

Ryzen AI Max+ 395 Strix Halo · gfx1151 Radeon 8060S ROCm 7.2.4 128 GB unified NVFP4 · NVIDIA

18repositories

846GiB published

14base models

13ROCmFP4 builds

What I actually found

ROCmFP4 does not universally speed up decode. That's the most useful thing I can tell you, and it took four builds and one discarded model to learn it.

Model	Architecture	Active	Decode, short ctx	Decode, long ctx	Verdict
Laguna-S-2.1 118B-A8B	`laguna`	~8B	+62.6%	+43.6%	ship it
Step-3.7-Flash 198B MoE	`step35`	~11B	+18%	+20%	ship it
Leanstral-1.5 119B-A6B	`deepseek2` · MLA	~6.5B	+1.5%	−8.2%	size only
KAT-Coder-V2.5-Dev 35B-A3B	hybrid linear	~3B	+12%	−37%	discarded

The builds that gained were the ones pushing more active parameters through the FP4 FFN kernels. MLA and hybrid-linear attention shift work away from that path and gained nothing — one regressed badly as context grew.

Benchmarking at one context length will lie to you. KAT-Coder looks like a 12% win at short context. At long context it's 37% slower. I nearly shipped it.

Attention type and active-param count are confounded in this sample. Four data points is a hypothesis, not a proof. I'd rather it be tested than believed.

How I measure

Numbers are worth exactly as much as the discipline behind them.

Equal generation length — every run emits exactly 256 tokens with ignore_eos. Comparing tok/s across runs with different token counts is meaningless; one early run of mine was soft because a baseline stopped after 4 tokens.
Nonce-prefixed prompts — every prompt opens with a fresh UUID so the prefix cache can't serve a seen prefix and inflate prefill. cached_tokens is recorded each run so you can check it was 0. I caught a contaminated campaign this way.
Two context lengths minimum — see KAT-Coder.
One model resident, cold-loaded, three runs, median — on 128 GB unified memory two large models won't coexist, and a half-swapped model produces noise.
Failures published — OOM, crash and regression get a row in the table, not a deletion. Where a run shared the machine with other traffic, the card says the ordering is reliable but the exact ratios aren't.

The builds

ROCmFP4 — AMD Strix Halo

Requires the ROCmFPX fork. These will not load in stock llama.cpp, Ollama or LM Studio.

Model	Base	Repo size	Contents
Muse-Glimmer-30B	meta-models	63.0 GiB	4 variants · drafter · vision
Laguna-S-2.1	poolside	58.3 GiB	STRIX_LEAN
Leanstral-1.5-119B-A6B	mistralai	59.0 GiB	STRIX_LEAN
Step-3.7-Flash	stepfun-ai	101.4 GiB	STRIX_LEAN · vision
DeepSeek-V4-Flash-180B	deepseek-ai	181.7 GiB	2 variants
DeepSeek-V4-Flash-0731	deepseek-ai	100.4 GiB	ROCmFP4
Qwen3-Next-80B-A3B	Qwen	39.7 GiB	STRIX
BTL-4	badtheorylabs	18.2 GiB	STRIX · vision
North-Mini-Code-1.0	CohereLabs	15.3 GiB	STRIX
GLM-4.7-Flash	zai-org	14.9 GiB	STRIX
Instella-ToolCall-16B-A3B	amd	8.0 GiB	STRIX
Instella-MoE-16B-A3B-Think	amd	7.9 GiB	STRIX

NVFP4 — NVIDIA

Model	Base	Repo size	Notes
Ling-3.0-flash	inclusionAI	75.8 GiB	SGLang · MTP working
Frontis-MA1-35B	FrontisAI	23.3 GiB	with MTP
Macaron-V1-Tall	mindlab-research	23.3 GiB	MTP · LoRA-compatible
Instella-ToolCall-16B-A3B	amd	9.0 GiB	tool calling

Things that cost me time, so they don't cost you any

A vision port can compile cleanly and be completely wrong. Muse Glimmer needs per-layer sparse-window attention masks. Drop them and the build works, runs, and describes images fluently and incorrectly. A four-quadrant colour test catches it — a broken mask names the colours right and puts them in the wrong corners. "Describe this image" does not catch it.
Per-token acceptance is the wrong metric for a block drafter. DFlash proposes 15 tokens per pass; ~21% acceptance sounds broken and actually means ~4.13 tokens landing per target pass, which is healthy. Read tokens-per-pass.
A higher acceptance rate can be slower. n-gram speculation hit 42.9% acceptance against DFlash's 21% and ran 45% slower, because it proposes far fewer tokens per pass.
Check what else is on the box. An identical request measured 40s on a quiet machine and 118s while another model was resident. Contention will invent a result and hand it to you with a straight face.

Credit

The ROCmFP4 / ROCmFPX tensor formats are not my work — they're charlie12345 / caf's ROCmFPX fork of llama.cpp, with contributions from ciru-ai, Tom Turney, PlunderStruck and Aydan S. Every ROCmFP4 file here was produced with their quantizer and runs on their runtime. If these builds are useful to you, star that repo.

Built on llama.cpp and AMD ROCm. All base model weights, licences and capabilities belong to their original authors — I contribute quantisation and measurement only.