Instructions to use unsloth/GLM-5.3-Flash-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use unsloth/GLM-5.3-Flash-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./llama-cli -hf unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./build/bin/llama-cli -hf unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL

Use Docker

docker model run hf.co/unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL

LM Studio
Jan

vLLM

How to use unsloth/GLM-5.3-Flash-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "unsloth/GLM-5.3-Flash-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/GLM-5.3-Flash-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL

Ollama
How to use unsloth/GLM-5.3-Flash-GGUF with Ollama:
```
ollama run hf.co/unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL
```
Unsloth Desktop

How to use unsloth/GLM-5.3-Flash-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use unsloth/GLM-5.3-Flash-GGUF with Docker Model Runner:
```
docker model run hf.co/unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL
```

Lemonade

How to use unsloth/GLM-5.3-Flash-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL

Run and chat with the model

lemonade run user.GLM-5.3-Flash-GGUF-UD-Q4_K_XL

List all available models

lemonade list

Hermes Agent

How to use unsloth/GLM-5.3-Flash-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use unsloth/GLM-5.3-Flash-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

You can now Run GLM-5.3-Flash Locally! ✨

pinned

by danielhanchen - opened 9 days ago

Discussion

danielhanchen

Unsloth AI org 9 days ago

•

edited 9 days ago

Hey guys, GLM-5.3-Flash can now be run locally in Unsloth Desktop! ✨

Run 3-bit on 128GB RAM or 1-bit on 100GB. The bigger ones are still uploading.
GLM-5.3-Flash (ox-alpha) rivals Claude Opus 4.8 on DeepSWE, coding & agentic benchmarks.

Unsloth GitHub: https://github.com/unslothai/unsloth
Guide: https://unsloth.ai/docs/models/glm-5.3-flash

danielhanchen pinned discussion 9 days ago

Banaxi-Tech

8 days ago

Please Unsloth make TQ1_0 I need fit on 96G

dnnlee

8 days ago

Does the MTP layer work? I tried loading them but it results in an error message saying nextm isn't implemented

ahmadriaz9999

7 days ago

Please Unsloth make TQ1_0 I need fit on 96G

Yes agreed please and thx

MilestoneCap

7 days ago

What type of speed are people getting with 128GB Macs?

BingoBird

7 days ago

•

edited 6 days ago

Unsloth fork llama.cpp, 08-2026 pull
Default thinking runs ridiculously long, but 'low' seems okay so far
-mmap, -fit on
-b 4096 -ub 1024
full 1m ctx seems to be runnable

Major bottlneck 2ch DDR5, 4800mt/s.
At 108k context: PP 60-70, TG 6-7, rtx3090 gpus @100w /gpu, cpu @6threads 22%, DRAM use 118GB.
At 240k context: PP 8-11, TG 6

Increasing to 1M context drops it to a slog (why? until the ctx is actually filled, it should be fast... there must be a patch...)

Performance for this 176GB system is in same league as Minimax-M2.7, MiMo-2.5 and DeepSeek4-flash. All are so smart that it's hard to differentiate between them. GLM5.3 definitely up there - at these sizes the main productivity differences come from my degree of resonance and shared assumptions, the communication, the alignment with language.

MilestoneCap

7 days ago

This comment has been hidden (marked as Off-Topic)

Deniz-Eren

3 days ago

•

edited 3 days ago

In the guide page "https://unsloth.ai/docs/models/glm-5.3-flash" it says use this PR for llama.cpp "https://github.com/unslothai/llama.cpp/pull/61" which is branch "iq1-narrow" - I suspect this is wrong because I get the failure error:

0.00.128.529 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.336.555 I srv load_model: loading model '/mnt/nfs_storage/models/GLM-5.3-Flash-GGUF/UD-Q4_K_XL/GLM-5.3-Flash-UD-Q4_K_XL-00001-of-00006.gguf'
0.00.536.904 E llama_model_load: error loading model: unknown model architecture: 'glm5next'
0.00.536.915 E llama_model_load_from_file_impl: failed to load model
0.00.536.922 E cmn common_init_: failed to load model '/mnt/nfs_storage/models/GLM-5.3-Flash-GGUF/UD-Q4_K_XL/GLM-5.3-Flash-UD-Q4_K_XL-00001-of-00006.gguf'
0.00.536.927 E srv load_model: failed to load model, '/mnt/nfs_storage/models/GLM-5.3-Flash-GGUF/UD-Q4_K_XL/GLM-5.3-Flash-UD-Q4_K_XL-00001-of-00006.gguf'
0.00.536.930 I srv operator(): operator(): cleaning up before exit...
0.00.538.612 E srv llama_server: exiting due to model loading error

Instead I used "glm5next/upstream" branch which works for a while but randomly crashes when left over night with some out of memory errors:

/home/.../unslothai-llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:107: CUDA error
89.11.249.309 E CUDA error: out of memory
89.11.249.315 E current device: 0, in function alloc at /home/.../unslothai-llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:590
89.11.249.316 E cuMemCreate(&handle, reserve_size, &prop, 0)
/home/.../unslothai-llama.cpp/build/bin/libggml-base.so.0(+0x1b236) [0x75eafa343236]
/home/.../unslothai-llama.cpp/build/bin/libggml-base.so.0(ggml_print_backtrace+0x20d) [0x75eafa3436bd]
/home/.../unslothai-llama.cpp/build/bin/libggml-base.so.0(ggml_abort+0x166) [0x75eafa3438a6]
/home/.../unslothai-llama.cpp/build/bin/libggml-cuda.so.0(Z15ggml_cuda_errorPKcS0_S0_iS0+0xb7) [0x75eaee68a617]

Can you please update the guide with the correct branch to use?

I will try branch "glm5next/public" overnight and see.

Deniz-Eren

3 days ago

public branch also crashes:

476.16.519.389 E ggml_cuda_compute_forward: SOFT_MAX failed
476.16.519.401 E CUDA error: invalid configuration argument
476.16.519.404 E /home/.../unslothai-llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:107: CUDA error
current device: 0, in function ggml_cuda_compute_forward at /home/.../unslothai-llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:2412
476.16.519.406 E err
/home/.../unslothai-llama.cpp/build/bin/libggml-base.so.0(+0x1b0f6) [0x72b0ca1440f6]
/home/.../unslothai-llama.cpp/build/bin/libggml-base.so.0(ggml_print_backtrace+0x20d) [0x72b0ca14457d]
/home/.../unslothai-llama.cpp/build/bin/libggml-base.so.0(ggml_abort+0x166) [0x72b0ca144766]
/home/.../unslothai-llama.cpp/build/bin/libggml-cuda.so.0(Z15ggml_cuda_errorPKcS0_S0_iS0+0xb7) [0x72b0be48a567]

BingoBird

3 days ago

•

edited 3 days ago

error loading model: unknown model architecture: 'glm5next': shows wrong llama.cpp version.

second error on public branch: use 'ldd' to confirm executeable is pulling the libggml-base and libggml-cuda from the correct location

also share build command and CUDA version.

Deniz-Eren

3 days ago

Current branch

$ git status
On branch glm5next/upstream
Your branch is up to date with 'origin/glm5next/upstream'.

nothing to commit, working tree clean

$ git log -n1
commit 949f7efb097eb20ef36fecdb1afaebff9a4ae7ed (HEAD -> glm5next/upstream, origin/glm5next/upstream)
Author: Daniel Han danielhanchen@gmail.com
Date: Mon Aug 31 10:39:19 2026 +0000

glm5next: cut comments

Second reduction pass over the arch's comments: 405 comment lines on the
branch's own added lines down to 234, no code changes.

Deletes rather than reshortens. What stayed is limited to things whose absence
would let a reader make a specific mistake: reference constants and sign
conventions, the ordering and precision constraints the graph relies on, and
the shapes of ggml tensors, whose type carries none.

Full crash output:

/home/.../unslothai-llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:107: CUDA error
100.52.827.266 E CUDA error: out of memory
100.52.827.273 E current device: 0, in function alloc at /home/.../unslothai-llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:590
100.52.827.274 E cuMemCreate(&handle, reserve_size, &prop, 0)
/home/.../unslothai-llama.cpp/build/bin/libggml-base.so.0(+0x1b236) [0x7399f9aea236]
/home/.../unslothai-llama.cpp/build/bin/libggml-base.so.0(ggml_print_backtrace+0x20d) [0x7399f9aea6bd]
/home/.../unslothai-llama.cpp/build/bin/libggml-base.so.0(ggml_abort+0x166) [0x7399f9aea8a6]
/home/.../unslothai-llama.cpp/build/bin/libggml-cuda.so.0(Z15ggml_cuda_errorPKcS0_S0_iS0+0xb7) [0x7399ed88a617]
/home/.../unslothai-llama.cpp/build/bin/libggml-cuda.so.0(_ZN18ggml_cuda_pool_vmm5allocEmPm+0x48c) [0x7399ed8a4ecc]
/home/.../unslothai-llama.cpp/build/bin/libggml-cuda.so.0(_Z24argsort_f32_i32_cuda_cubR14ggml_cuda_poolPKfPiii15ggml_sort_orderP11CUstream_st+0x408) [0x7399ed6c99c8]
/home/.../unslothai-llama.cpp/build/bin/libggml-cuda.so.0(_Z18ggml_cuda_op_top_kR25ggml_backend_cuda_contextP11ggml_tensor+0x20f) [0x7399eda4e97f]
/home/.../unslothai-llama.cpp/build/bin/libggml-cuda.so.0(+0x2a11b9) [0x7399ed8a11b9]
/home/.../unslothai-llama.cpp/build/bin/libggml-base.so.0(ggml_backend_sched_graph_compute_async+0x89f) [0x7399f9b0985f]
/home/.../unslothai-llama.cpp/build/bin/libllama.so.0(_ZN13llama_context13graph_computeEP11ggml_cgraphb+0xa0) [0x7399f83032c0]
/home/.../unslothai-llama.cpp/build/bin/libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0x119) [0x7399f83073a9]
/home/.../unslothai-llama.cpp/build/bin/libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x476) [0x7399f830e606]
/home/.../unslothai-llama.cpp/build/bin/libllama.so.0(llama_decode+0x12) [0x7399f830fcd2]
/home/.../unslothai-llama.cpp/build/bin/libllama-server-impl.so(_ZNSt17_Function_handlerIFvvEZN19server_context_impl6decodeERiiR11llama_batchEUlvE_E9_M_invokeERKSt9_Any_data+0x49) [0x7399f9569629]
/home/.../unslothai-llama.cpp/build/bin/libllama-server-impl.so(_ZN12server_queue14yield_to_queueEOSt8functionIFvvEE+0xa9) [0x7399f951de99]
/home/.../unslothai-llama.cpp/build/bin/libllama-server-impl.so(_ZN19server_context_impl6decodeERiiR11llama_batch+0x199) [0x7399f957a999]
/home/.../unslothai-llama.cpp/build/bin/libllama-server-impl.so(_ZN19server_context_impl12update_slotsEv+0x53c) [0x7399f957c5fc]
/home/.../unslothai-llama.cpp/build/bin/libllama-server-impl.so(_ZN12server_queue10start_loopEl+0x138) [0x7399f9521cd8]
/home/.../unslothai-llama.cpp/build/bin/libllama-server-impl.so(_Z12llama_serverR13common_paramsiPPc+0x3ed1) [0x7399f94d5611]
/home/.../unslothai-llama.cpp/build/bin/libllama-server-impl.so(_Z12llama_serveriPPc+0x115e) [0x7399f94d763e]
/usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a601) [0x7399f8c2a601]
/usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x88) [0x7399f8c2a718]
/home/.../unslothai-llama.cpp/build/bin/llama-server(+0x1325) [0x5ed079fda325]
./run_GLM-5.3-Flash-UD-Q4_K_XL.sh: line 30: 3626398 Aborted numactl --interleave=0,1 ~/Repos/unslothai-llama.cpp/build/bin/llama-server -m /mnt/nfs_storage/models/GLM-5.3-Flash-GGUF/UD-Q4_K_XL/GLM-5.3-Flash-UD-Q4_K_XL-00001-of-00006.gguf --alias GLM-5.3-Flash-UD-Q4_K_XL --numa numactl --n-gpu-layers all --n-cpu-moe 45 -ctk q4_0 -ctv q4_0 -c 322144 -fa on --fit off -np 1 --host 0.0.0.0 --port 8090 --threads 28 --threads-batch 28 --temp 1.0 --top-p 0.95 --chat-template-kwargs '{"reasoning_effort":"max"}' --cache-reuse 256 -b 512 -ub 512 --load-mode mmap+mlock --api-key-file /mnt/nfs_storage/models/api_keys.txt --metrics --slots --props --reasoning-budget 8000

ldd output:

$ ldd ~/Repos/unslothai-llama.cpp/build/bin/llama-server
linux-vdso.so.1 (0x0000707a3d2d2000)
libllama-server-impl.so => /home/.../unslothai-llama.cpp/build/bin/libllama-server-impl.so (0x0000707a3cc00000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x0000707a3c800000)
libgcc_s.so.1 => /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 (0x0000707a3d28b000)
libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x0000707a3c400000)
libllama-common.so.0 => /home/.../unslothai-llama.cpp/build/bin/libllama-common.so.0 (0x0000707a3be00000)
libmtmd.so.0 => /home/.../unslothai-llama.cpp/build/bin/libmtmd.so.0 (0x0000707a3c642000)
libllama.so.0 => /home/.../unslothai-llama.cpp/build/bin/libllama.so.0 (0x0000707a3ba00000)
libggml.so.0 => /home/.../unslothai-llama.cpp/build/bin/libggml.so.0 (0x0000707a3d27c000)
libggml-base.so.0 => /home/.../unslothai-llama.cpp/build/bin/libggml-base.so.0 (0x0000707a3cb28000)
libssl.so.3 => /usr/lib/x86_64-linux-gnu/libssl.so.3 (0x0000707a3c2f2000)
libcrypto.so.3 => /usr/lib/x86_64-linux-gnu/libcrypto.so.3 (0x0000707a3b200000)
libm.so.6 => /usr/lib/x86_64-linux-gnu/libm.so.6 (0x0000707a3b8da000)
/lib64/ld-linux-x86-64.so.2 (0x0000707a3d2d4000)
libggml-cpu.so.0 => /home/.../unslothai-llama.cpp/build/bin/libggml-cpu.so.0 (0x0000707a3b040000)
libggml-blas.so.0 => /home/.../unslothai-llama.cpp/build/bin/libggml-blas.so.0 (0x0000707a3d26e000)
libggml-cuda.so.0 => /home/.../unslothai-llama.cpp/build/bin/libggml-cuda.so.0 (0x0000707a30c00000)
libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x0000707a3cac9000)
libz.so.1 => /usr/lib/x86_64-linux-gnu/libz.so.1 (0x0000707a3d24e000)
libzstd.so.1 => /usr/lib/x86_64-linux-gnu/libzstd.so.1 (0x0000707a3af7d000)
libblas.so.3 => /usr/lib/x86_64-linux-gnu/libblas.so.3 (0x0000707a3b853000)
libcudart.so.12 => /usr/lib/x86_64-linux-gnu/libcudart.so.12 (0x0000707a30800000)
libcublas.so.12 => /usr/lib/x86_64-linux-gnu/libcublas.so.12 (0x0000707a29c00000)
libcuda.so.1 => /usr/lib/x86_64-linux-gnu/libcuda.so.1 (0x0000707a24400000)
libnccl.so.2 => /usr/lib/x86_64-linux-gnu/libnccl.so.2 (0x0000707a15e00000)
libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x0000707a13920000)
libdl.so.2 => /usr/lib/x86_64-linux-gnu/libdl.so.2 (0x0000707a3d247000)
libpthread.so.0 => /usr/lib/x86_64-linux-gnu/libpthread.so.0 (0x0000707a3d240000)
librt.so.1 => /usr/lib/x86_64-linux-gnu/librt.so.1 (0x0000707a3d23b000)
libcublasLt.so.12 => /usr/lib/x86_64-linux-gnu/libcublasLt.so.12 (0x00007079f5800000)
libmvec.so.1 => /usr/lib/x86_64-linux-gnu/libmvec.so.1 (0x0000707a30b05000)
libgfortran.so.5 => /usr/lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007079f5400000)

My build command:

cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON -DGGML_BLAS=ON -DGGML_CUDA=ON -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build build -j28

CUDA version

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0

BingoBird

3 days ago

libs seem to be pulling the right ones. that's sometimes a problem.

CUDA 12.x is said to be unproblematic.

maybe that ALL_QUANTS or BLAS is tripping things up? I built without them. GL!

Deniz-Eren

3 days ago

Yes that's exactly what I did just now. I will leave it overnight and see what I find in the morning.

Deniz-Eren

2 days ago

•

edited 2 days ago

Crash again with the new build command cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON -DGGML_CUDA=ON -DBUILD_SHARED_LIBS=OFF

Basically whenever I put "-c" option more than 262144 it crashes, even though it loads initially and everything fits into the VRAM. It also only crashes during token generation and not prefill.

Deniz-Eren

2 days ago

Perhaps --load-mode mmap+mlock causes the issue, I'm testing now with --load-mode none

Deniz-Eren

1 day ago

Stable now with --load-mode none

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment