darioooooo0o/drama-3-w4a16

Drama-3 W4A16 (int4 experts)

Built with Fish Audio. This is an unofficial int4 quantization of fishaudio/drama-3, Fish Audio's 35B-parameter (3B active) multi-speaker text-to-speech model. It is about 38% smaller (41.5 → 25.9 GB). Quantized and tested by @imdariotoo. It is not affiliated with or endorsed by Fish Audio.

32 GB card (RTX 5090)? Use drama-3-w4a16-32gb (int8 backbone too, 30.7 GiB peak with its launch settings).

Listen first

Every clip below was generated with this int4 checkpoint, with no cherry-picking except where noted. Transcripts are from Whisper large-v3.

prompt	audio	Whisper transcript
Hello, how are you? The weather is lovely today, so let's take a long walk by the river.		exact
Voice clone of the cookbook reference clip: Get the trust fund to the bank early.		exact
Voice clone with tags: [sighs] Fine. If we miss it, it's on you. [laughs] Well, maybe that's fair.		exact (3 of 4 tries; 1 came out short)
The quarterly report is due on Thursday, so please finish it by noon. After that, we will review the numbers together and decide whether the new pricing plan should launch next month or wait until spring.		exact
今天天气很好，我们一起去公园散步吧。		exact

To compare against the original FP8 model, Fish Audio's own clip for the tags prompt is in the Drama 3 cookbook. Whisper transcribes it with the same words as our sample.

The clips are also in samples/ (44.1 kHz WAV).

At a glance

	original fishaudio/drama-3	this repo
download size	41.5 GB	25.9 GB
routed MoE experts (30,720 matrices)	block-FP8 (128×128 scales)	int4, symmetric, group 128, GPTQ packing
attention, shared experts, embeddings, lm_head, audio decoder	bf16	bf16 (byte-identical copies)
tokenizer, codec.pth	—	identical copies
GPU memory when serving (tested)	needs > 48 GB with default settings	37.5 GB on a 48 GB card
runtime	SGLang-Omni	SGLang-Omni + apply_patch.py (included)

Requirements

• GPU: an NVIDIA GPU with compute capability ≥ 8.0 (Ampere or newer) and ≥ 40 GB of VRAM, e.g. A6000, L40S, RTX 6000 Ada, 48 GB RTX 4090 mods, A100, H100. Tested: RTX 6000 Ada 48 GB.
• 24 GB cards won't fit: the int4 weights alone are 22.5 GiB on the GPU.
• Linux, CUDA 12.8+, Python 3.12, SGLang-Omni (installed below).
• ~30 GB of free disk.

Quick start

# 1. install SGLang-Omni (Fish Audio's officially supported runtime for Drama 3)
git clone https://github.com/sgl-project/sglang-omni.git && cd sglang-omni
uv venv -p 3.12 .venv && . .venv/bin/activate
uv pip install --prerelease=allow -e ".[fishaudio]"

# 2. download this checkpoint and patch the loader (idempotent; adds one quantization config)
hf download darioooooo0o/drama-3-w4a16 --local-dir drama-3-w4a16
python drama-3-w4a16/apply_patch.py .

# 3. serve
sgl-omni serve --model-path drama-3-w4a16 --port 8000 \
  --tts_engine.gpu_memory_fraction 0.62 --vocoder.gpu_memory_fraction 0.2

About the memory flags: by default the engine takes 70% of the GPU, which leaves the vocoder too little room on a 48 GB card (the server refuses to start). The int4 weights free enough space to give the engine 62% (weights + KV cache) and the vocoder 20%. On an 80 GB card you can drop both flags.

Generate speech

# plain TTS
curl -X POST http://localhost:8000/v1/audio/speech -H "Content-Type: application/json" \
  -d '{"input": "Hello, how are you?"}' --output out.wav

# voice cloning (reference clip + its transcript), with inline tags
curl -X POST http://localhost:8000/v1/audio/speech -H "Content-Type: application/json" \
  -d '{"input": "[sighs] Fine. If we miss it, it is on you. [laughs] Well, maybe that is fair.",
       "references": [{"audio_path": "https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval-mini/resolve/main/en/prompt-wavs/common_voice_en_10119832.wav",
                        "text": "We asked over twenty different people, and they all said it was his."}]}' \
  --output clone.wav

Streaming, multi-speaker tags (<|speaker:0|> … <|speaker:15|>), editing with previous_takes and the other features work the same as with the original. See the Drama 3 cookbook.

What the patch does

SGLang-Omni's Drama 3 loader only recognises the official FP8 checkpoint (quant_method: "bsgemm-moe"). apply_patch.py adds ExpertW4A16Config (quant_method: "bsgemm-moe-w4a16") to sglang_omni/models/fishaudio_drama3/quantization.py and registers it in engine_builder.py:

• the routed experts (FusedMoE) run SGLang's existing moe_wna16 int4 Triton kernel;
• every linear layer stays unquantized bf16, the same as in the original.

It is pure Python and changes nothing else. The same patch also covers the 32 GB edition.

How it was made

• Script: quantize.py (included). Run python quantize.py fishaudio-drama-3-dir out-dir --device cuda; it takes about 2 minutes on one GPU.
• Method:
  • Each FP8 expert matrix is dequantized with its 128×128 block scales.
  • It is re-quantized to int4 with round-to-nearest, one scale per 128 input weights.
  • Each scale is chosen from 20 clipping ratios (1.0 → 0.81) by minimum squared error.
• Calibration data: none (no GPTQ/AWQ calibration).
• Accuracy: mean relative reconstruction error per expert matrix is 10.3% (max 16.2%, see quant_stats.json), about what int4 group-128 rounding gives on Gaussian weights.
• Checked: the packing was verified against a reimplementation of SGLang's moe_wna16 loader and kernel unpacking before upload.

Limitations

• Coverage: these are spot checks (the 5 prompts above), not a benchmark. A round-to-nearest int4 quant can lose some prosody, expressiveness or voice similarity compared with FP8, so listen before you rely on it, especially for voice cloning.
• Hardware: tested on one GPU type (RTX 6000 Ada) with SGLang-Omni at commit main on 2026-10-13. The patch may need updating if SGLang-Omni's loader changes.
• Sampling variance: like the original, outputs are sampled; an occasional take can come out truncated (1 of 4 on the tags prompt). Regenerate when that happens.

License and attribution

This is a Derivative Work of Fish Audio Materials under the Fish Audio Community License, Version 1.0 (LICENSE), the same license as the original. NOTICE lists the required attribution and exactly what was changed.

In short (the LICENSE is what counts):

• Research and non-commercial use are free.
• Commercial use is allowed only for entities under USD 500k annual revenue.
• No hosted TTS, voice-cloning API or similar service without a separate license from Fish Audio.
• No cloning of real people's voices without their consent; no impersonation or fraud.
• Don't remove watermarks.
• Don't use the outputs to train other speech or foundation models.

These Fish Audio Materials are licensed under the Fish Audio Community License, Version 1.0. Copyright © 39 AI, INC. All Rights Reserved.

Credits

• Model: Fish Audio, fishaudio/drama-3.
• Runtime: SGLang-Omni; int4 MoE kernel from SGLang.
• Quantization, patch and testing: @imdariotoo. Questions or issues: reply on X or open a discussion on this repo.