Instructions to use kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF:Q4_0 # Run inference directly in the terminal: llama cli -hf kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF:Q4_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF:Q4_0 # Run inference directly in the terminal: llama cli -hf kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF:Q4_0
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF:Q4_0 # Run inference directly in the terminal: ./build/bin/llama cli -hf kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF:Q4_0
Use Docker
docker model run hf.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF:Q4_0
- Llama new
- LM Studio
- Jan
- vLLM
How to use kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF:Q4_0
- Ollama
How to use kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF with Ollama:
ollama run hf.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF:Q4_0
- Unsloth Desktop
- Pi
How to use kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF:Q4_0
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF:Q4_0" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF with Docker Model Runner:
docker model run hf.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF:Q4_0
- Lemonade
How to use kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF:Q4_0
Run and chat with the model
lemonade run user.Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF-Q4_0
List all available models
lemonade list
- Hermes Agent
How to use kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF:Q4_0
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF:Q4_0
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF:Q4_0
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF:Q4_0" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Thanks & MTP Query
Have just come across your profile, and as a Strix user it's an absolute gold mine. Very excited to see what your quants and build will do for my boxes!
I'm curious what the blocker(s) are for MTP on this model please - I understand the Qwen uploads include the heads, but they seem to be dropped in many other builds. Is there something holding these back?
Otherwise if you need some extra runs or data on a model let me know - I have halobench.com mainly for my own reference, but I feel like it's going to have more of your models showing up now haha
Thanks, wanted to say this but found your comment . Interested in knowing about MTP
And thanks Kingjones, found this repo today
confirmed. 21.69 t/s generation at 72,425 ctx tokens (214 t/s prompt eval)
Thanks β glad it's useful, and yes, please do put them on halobench.
Straight answer on MTP: it's our converter dropping them, not Qwen and not the runtime. I just checked rather than guess.
Upstream ships the heads β 31 mtp.* tensors out of 1658, with text_config.mtp (1 hidden layer, hybrid: true, full_attention, rope_theta 1e7). Our published GGUFs contain exactly zero of them: 0 mtp/nextn keys and 0 tensors across all three shards.
The reason is a flag in our converter, conversion/qwen4exp.py:
the MTP block is a separate draft head; vLLM drops it too
supports_mtp_export = False
no_mtp = True
That was a parity call with vLLM, but it isn't just a flag to flip. This MTP block isn't a plain nextn layer the way Ling's or Qwen3.8-27B's is β it carries mtp.fc_embedding, mtp.fc_hidden, and a hyper_connection_mixer / attn_hyper_connection stack. So making it usable needs two things that don't exist yet on our path: GGUF tensor mapping for that sub-module, and graph support to actually run it. The loader side is partway there β qwen4exp.cpp already tracks nextn_predict_layers and excludes them from the main-layer count.
Worth being clear about the consequence: every speed number on our Flash-Next cards is non-MTP. On our other builds MTP has been worth +13β17% on Ling-3.0-flash-base and 2.83Γ on Qwen3.8-27B, so this is a real gap, not a rounding error. It's on the list.
One correction to something in our earlier thread while I'm here β the runtime claim now resolves to a real, public tree: https://github.com/kingjones30/ROCmFPX, a fork of charlie12345/ROCmFPX on upstream c49ebdb. Clean clone β cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 β¦ β 0 errors, and it loads these files.
On extra runs β yes, and here's the one I'd most like. QSA has a 512-block / 2048-token budget and nobody has characterised it at depth. What we have so far on a 395 / gfx1151 / 128 GB:
262,144 ctx granted, zero CPU-offload lines
GTT 64.6 GiB at 32k β 73.7 GiB at 262k (from a 99.3 GiB file β the ~35 GiB PLE never enters GPU memory)
only 0.1 GiB of the PLE actually faulted in across four generations, so the working set tracks prompt diversity, not model size
24β27 tok/s generation warm, short prompts
What's missing is pp/gen at real prompt depth β 8k / 32k / 64k / 131k / 262k with genuinely that many tokens in the prompt, plus whether GTT stays near 74 GiB or climbs. That's the number that decides daily-driver status, and I'd rather have it measured on someone else's box than only mine.
Two measurement traps, since they each cost me runs: use one fixed prompt with cache_prompt: false rather than varying prompts (varying them injects slice-to-slice variance β it made our published pp read ~45% low), and check prompt_n matches what you actually sent.
Here's my results from overnight, need to troubleshoot to return the gaps haha
I ran the requested fixed-prompt depth ladder on the full-STRIX ROCmFP4 model (gfx1151, 128 GB, HIP 7.1), using raw /completion, cache_prompt:false, thinking off, and the observed 512 checkpoints / 2048-token cadence. Every accepted row matched the requested prompt_n.
| Real prompt depth | PP tok/s | Gen tok/s | Output tokens | GTT after request |
|---|---|---|---|---|
| 8,192 | 405.75 | β | 1 / zero bytes | 72.34 GiB |
| 32,768 | 336.15 | 20.91 | 2,048 | 73.85 GiB |
| 65,536 | 266.79 | 17.82 | 38 | 75.84 GiB |
| 131,072 | 194.09 | 14.94 | 58 | 79.78 GiB |
| 260,096 | 121.84 | β | 1 / zero bytes | 87.48 GiB |
| 262,144 | β | β | HTTP 400; no timing payload | 87.48 GiB |
8k/32k are medians of three settled runs; deeper points are single settled runs. The 8k and 260,096 βgenerationβ results are one-token/zero-byte stops, so Iβm not treating their reported sentinel rate as throughput.
The useful answer is that GTT does not remain near 74 GiB: it rises progressively to 87.48 GiB near full context. Decode also falls from 20.91 tok/s at 32k to 14.94 tok/s at 131k. The per-PLE mincore adapter was incomplete, so I canβt make a defensible PLE-fault/working-set claim from this run.
Good run. Four separate causes in there, not one β and one of them means you were chasing something that isn't in that model.
262,144 β HTTP 400 is correct. n_ctx_train is 262,144, so a prompt of exactly that leaves zero room to generate. llama.cpp rejects when prompt_n + n_predict > n_ctx, before the graph runs β that's why there's no timing payload. Your -c has to exceed prompt + n_predict.
260,096 β 1 token is the same wall one step back. 260,096 = 262,144 β 2,048. If n_predict was 2,048 you were sitting exactly on the boundary. Headroom failure, not a generation failure.
8,192 β 1 token is different: raw /completion applies no chat template. --jinja only affects /v1/chat/completions. On raw /completion the prompt goes in verbatim, the model sees an unterminated fragment, and EOS as the first token is a legitimate continuation. That's also your inconsistent output lengths β 2,048 / 38 / 58 / 1 / 1 is the model choosing where to stop from raw text, not following an instruction. Either move to /v1/chat/completions, or set "ignore_eos": true with an explicit n_predict if you want a clean throughput number.
And you were right not to treat those sentinel rates as throughput. predicted_per_second = predicted_n / (predicted_ms/1000) β 1 token in ~0 ms gives you nonsense. Any tok/s in the thousands is a failed generation, not a fast one.
The PLE adapter was incomplete because full-STRIX 27B has no PLE table. Straight from the file:
Qwen3.8-27B-Q4_0_ROCMFP4_STRIX : arch qwen35 β per_layer/PLE tensors: 0
Qwen3.8-Flash-Next : arch qwen4exp β per_layer_token_embd.weight = [160, 320001536]
PLE is a qwen4exp feature. On qwen35 there's nothing to fault on and no working set to measure. Run that one against Flash-Next.
Your two real findings both hold, and they match what I measure independently.
GTT 72.34 β 87.48 GiB is the KV cache doing exactly what it should β ~15.1 GiB over ~252k tokens, about 0.062 MiB/token. Never estimate KV from file size; on Strix that ratio swings 2%β105%. Read it off the server's own prompt_save β¦ total state size line.
Decode 20.91 β 14.94 reproduces here (22.2 @40k β 13.3 @60k ). It fits a two-component model:
cycle β weights_read/100 GB/s + KV_bytes/19 GB/s
The KV scan runs about 5Γ slower than weight streaming, so the KV term grows until it dominates. Two consequences worth your time: quantized KV (-ctk q8_0 -ctv q8_0, or turbo4 on V) attacks the term that's actually growing β and if you're running speculative decoding, measure it spec-on and spec-off separately. Here the base forward path only degrades ~13% from 40kβ60k while the speculative path degrades ~40%. A lot of "long-context collapse" is a speculation pathology, not a hardware limit.