Qwen3.8-Flash-Next-Heretic2-IQ4XS-NGQ4-GGUF

IQ4_XS quantization of Qwen3.8-Flash-Next abliterated with Heretic v1.3.0+custom (fork), with NGQ4-compressed n-gram prediction table. Built and benchmarked on AMD Strix Halo (Ryzen AI Max+ 395, gfx1151, 128 GB unified).

Files

File	Size	Description
Qwen3.8-Flash-Next-Heretic2-IQ4XS-NGQ4.gguf	92 GB	Main model (IQ4_XS, NGQ4-compressed PLE table)
mmproj-Heretic2-BF16.gguf	866 MB	Vision projector (multimodal/image input)

Quantization

Quantized from the Heretic2 BF16 safetensors → Q8_0 intermediate → IQ4_XS using the llama-flash-next fork of llama.cpp (build 10656, commit 035e22731). The NGQ4 step compresses the 54 GB per-layer n-gram prediction table to ~27 GB by lowering precision on the indexer's token embedding, reducing the model from 116 GB to 92 GB with no measurable accuracy impact through 32K context.

Byte-verified against the Heretic2 BF16 checkpoint.

Tensor types

Tensor class	Type	Notes
Expert weights (ffn_down/up/gate_exps, shared experts, hc layers)	IQ4_XS	Bulk of parameters
Dense/trunk (non-expert weights)	Q8_0	Attention projections, etc.
Output projection (output.weight)	Q6_K	Higher precision for output
Token embeddings (token_embd.weight)	BF16	Small separate embedding [2560, 248320]
Per-layer token embeddings	IQ4_NL	NGQ4 compression (was BF16 in uncompressed IQ4_XS)
Norms (1D tensors)	F32
Indexer projections (indexer.k_proj, q_proj)	Q8_0

How to run

Requires the llama-flash-next fork (qwen4exp branch) — standard llama.cpp cannot load the qwen4exp architecture.

# Text only
./llama-server \
  -m Qwen3.8-Flash-Next-Heretic2-IQ4XS-NGQ4.gguf \
  --alias qwen38-flash-next-heretic2 \
  -ngl 99 -fa on -c 65536 -np 1 --jinja \
  --host 127.0.0.1 --port 8096 \
  -dev Vulkan0

# With vision
./llama-server \
  -m Qwen3.8-Flash-Next-Heretic2-IQ4XS-NGQ4.gguf \
  --mmproj mmproj-Heretic2-BF16.gguf \
  --alias qwen38-flash-next-heretic2 \
  -ngl 99 -fa on -c 65536 -np 1 --jinja \
  --host 127.0.0.1 --port 8096 \
  -dev Vulkan0

Thinking mode is optional — disable it with "chat_template_kwargs": {"enable_thinking": false} in API requests to use the full token budget for output.

Benchmarks — HumanEval+ (EvalPlus)

164 problems, enable_thinking: false, temperature: 0.0, max_tokens: 1024.

Variant	Engine	Base pass@1	Plus pass@1	Mean s/prob	Median s/prob
This model (Heretic2 IQ4_XS NGQ4)	llama-server	84.1%	79.3%	4.16	3.31
Q6_K_v7_v7	llama-server	88.4%	82.9%	11.26	10.02
Q6_K	llama-server	88.4%	81.7%	6.43	5.63
Base flash-next IQ4_XS NGQ4	llama-server	82.3%	78.0%	4.82	3.21
Halogen Official w4b (native + overlay)	Halogen	82.3%	78.0%	13.82	10.66
Uncensored IQ4_XS NGQ4	llama-server	81.7%	78.7%	4.91	4.31
Halogen Heretic2 IQ4_XS (BYO, no overlay)	Halogen	65.2%	61.0%	4.32	3.62

This model offers the best accuracy-to-speed ratio in the IQ4_XS tier: 84.1% base at 4.16s per problem. Only the Q6_K variants score higher, but they require 129 GB (won't coexist with other workloads on 128 GB) and take 50–170% longer per response.

Origin story: Why this model exists

This GGUF was originally built as part of an apples-to-apples comparison between llama-server and Halogen Flash Server on AMD Strix Halo. Halogen's prefill throughput is dramatically faster — 2–4x at short contexts, approaching 4x at 16K+ tokens — so the question was whether it could replace llama-server as the default inference engine.

The answer turned out to be no, because of a critical limitation in Halogen's BYO (bring-your-own) GGUF workflow:

The overlay problem

Halogen's official checkpoint ships with a quality overlay sidecar (qwen38-flash-next-w4b.overlay.hgn, 2.4 GiB, 741 activation-aware re-quantized tensors). The official model with overlay scores 82.3% / 78.0% on HumanEval+ — competitive with llama-server. But the overlay is built through an internal activation-aware quantization pipeline that isn't publicly available. There's no halogen-quantize --build-overlay command.

When you bring your own GGUF — any fine-tune, any abliteration, any custom variant — you run without the overlay. The result: 65.2% / 61.0%, a 17-point collapse from the official model, using the exact same Heretic2 weights that score 84.1% on llama-server.

The thinking mode problem

Halogen forces thinking mode on Qwen3.8-Flash-Next with no way to disable it. Every request burns ~1,049 hidden reasoning tokens before producing visible output. With max_tokens=2048, that leaves ~999 tokens for code. The official model with overlay compensates by producing better code in fewer tokens. Without the overlay, the BYO model truncates — 68% of HumanEval+ failures were SyntaxErrors from incomplete code that the model reported as finish_reason=stop.

llama-server doesn't have this problem: thinking mode is optional via enable_thinking: false, and the full token budget goes to output.

The outcome

This Heretic2 IQ4_XS GGUF — originally built just for comparison — turned out to be a better model for production deployment than either the Halogen BYO setup (fast but 65% accuracy) or the Halogen official (82% accuracy but 3x slower per problem due to forced thinking). It runs on llama-server with full control over thinking mode, optional vision, and 84.1% HumanEval+ accuracy at 4.16s per problem.

For the full technical assessment, see the companion write-up: Halogen on Strix Halo: Fast but Not Ready

Abliteration details

The Heretic2 abliteration was performed using Heretic v1.3.0+custom (fork by timrohrbaugh) on the base Qwen/Qwen3.8-Flash-Next weights. Direction-removal method with per-layer direction index.

Metric	Heretic2	Original
KL divergence	0.0818	0
Refusals (100 prompts)	0/100	99/100

The abliteration has no measurable impact on code generation accuracy — Heretic2 IQ4_XS scores 84.1% vs base IQ4_XS at 82.3%, and the plus tests are within noise (79.3% vs 78.0%).

Hardware tested

• CPU: AMD Ryzen AI Max+ 395
• GPU: AMD Radeon 8060S (gfx1151, RDNA 4 integrated)
• Memory: 128 GB unified (DDR5)
• Backend: Vulkan (ROCm also works but is slower for this architecture)
• OS: Ubuntu

Throughput on Strix Halo (llama-server, Vulkan)

Prefill (IQ4_XS standard, same architecture)

Context	tok/s
512	312
2K	382
8K	363
16K	327

Decode

~24 t/s serial (memory-bandwidth limited on unified memory).

License

Qwen Community License 1.0 (inherited from the base model).

Credits

• Base model: Qwen/Qwen3.8-Flash-Next by Alibaba Qwen team
• Abliteration: Heretic by p-e-w, custom fork by timrohrbaugh
• Quantization fork: llama-flash-next (qwen4exp branch) by nicoboss
• Quantization & benchmarking: cygnal