TaoMate-H3-FL2VA-ComfyUI

ComfyUI version of 3-step turbo LoRA for Minimax-H3-FL2VA

• Format: bf 16
• Rank :128

Original repo: https://huggingface.co/TaoLiveAIGC/TaoMate-H3-FL2VA

---

TaoMate-H3-FL2VA

🎥 Demo Video

https://github.com/user-attachments/assets/15d276f9-02c0-49c6-b98f-98656c36164a

TaoMate-H3 is a low-latency streaming audio-video generation runtime built on MiniMax H3 with a 3-step LoRA. It generates synchronized audio and video in small chunks and supports continuous long-form generation at 480p/768p/1080p resolutions.

Developed by the Alibaba TaoLive AIGC Team. Powered by MiniMax H3.

🔥 News

• 🚀 2026/10/15: We release TaoMate-H3-FL2VA, to our knowledge the first real-time streaming first/last-frame-to-audio-video (FL2VA) model. Three-step generation combines first/last-frame control with continuous audio-video KV caching.
• 🤩2026/09/08: We first migrate MiniMax H3 to TaoMate (T2AV) for streaming. The new models of complete streaming FL2AV and Ref2AV are on the way. Please stay tuned!❤️

🚀 Features

• Real-time streaming FL2VA — first/last-frame control with three-step blockwise generation, persistent clean AV-KV, and a global endpoint deadline.
• Three-step LoRA streaming generation — each small chunk uses three Stage3 denoising intervals.
• Audio-video joint generation — speech, sound, and video are generated on one synchronized timeline.
• Low chunk latency — each small chunk reaches its final latent state well before a full MiniMax H3 request completes.
• Long-form continuity — clean KV cache and integrated audio guidance preserve visual identity, voice, and motion across prompt boundaries.
• Multiple resolutions — portrait and landscape generation at 480p, 768p, and aligned 1080p.
• Single-node inference — supports 4 GPUs or 8 GPUs with TP2 and Ulysses sequence parallelism.

🧱 Model Downloads

Models	Download Link	Notes
MiniMax H3	🤗 Huggingface	Base model
TaoMate-H3 T2AV	🤗 Huggingface	Our 3-step LoRA weights for T2AV
TaoMate-H3 FL2AV	🤗 Hugging Face	Our 3-step LoRA weights for FL2AV
TaoMate-H3 Ref2AV	TBD	TBD

The FL2VA adapter follows the same three-file layout as the T2AV release: config.json, adapter_config.json, and adapter_model.safetensors. It preserves the online generator weights in float32 and includes the inference configuration; optimizer state, EMA, and training-only score models are excluded.

🛠️ T2AV Installation

Requirements

• Linux
• Python 3.10 or 3.11
• NVIDIA Hopper/SM90 GPUs; 8 × H20 96 GB is the validated configuration
• CUDA 12.8 and PyTorch 2.8
• FFmpeg with H.264 and AAC support

Create the environment:

conda create -n taomate-h3 python=3.10 -y
conda activate taomate-h3

pip install torch==2.8.0 torchvision==0.23.0 \
  --index-url https://download.pytorch.org/whl/cu128
pip install triton==3.4.0 vllm==0.11.1

git clone https://github.com/Dao-AILab/flash-attention.git
pip install --no-build-isolation ./flash-attention/hopper

git clone https://github.com/TaoLiveAIGC/TaoMate-H3.git
cd TaoMate-H3
pip install -e .

Install FFmpeg on Ubuntu or Debian:

sudo apt-get update
sudo apt-get install -y ffmpeg

Download the MiniMax H3 FL2VA model:

hf download MiniMaxAI/MiniMax-H3 \
  --include "model_index.json" "FL2VA/*" \
  --local-dir models/MiniMax-H3

TaoMate-H3 T2AV LoRA weights are available on Hugging Face. The T2AV release is the step-3000 generator EMA adapter (rank 128, alpha 128). T2AV inference downloads it automatically to models/TaoMate-H3 on first use. To download it in advance:

hf download TaoLiveAIGC/TaoMate-H3 \
  --include "config.json" "adapter_config.json" "adapter_model.safetensors" \
  --local-dir models/TaoMate-H3

For private or gated models, run hf auth login with an account that has access. Downloads also accept the standard HF_TOKEN environment variable.

The LoRA directory contains:

models/TaoMate-H3/
├── config.json
├── adapter_config.json
└── adapter_model.safetensors

⭐️ T2AV Inference

TaoMate-H3 accepts either one prompt through --prompt or one prompt per five-second block through --prompt-json.

Example prompt file:

{
  "prompts": [
    "prompt for seconds 0-5",
    "prompt for seconds 5-10"
  ],
  "seeds": [8301, 8301]
}

Run TaoMate-H3:

python -m taomate_h3 \
  --model-root models/MiniMax-H3 \
  --prompt-json examples/prompts_10s.json \
  --duration 10 \
  --resolution 768x1376 \
  --gpus 8 \
  --devices 0,1,2,3,4,5,6,7 \
  --seed 8301 \
  --output outputs/demo_10s

The command runs the complete pipeline and writes the final video to outputs/demo_10s/video.mp4. It starts its own local distributed workers, so no external torchrun command is needed.

To use a different local LoRA, add --adapter /path/to/adapter.

Common resolutions:

Format	Portrait	Landscape
480p	480x864	864x480
768p	768x1376	1376x768
1080p	1088x1920	1920x1088

For an exact 1080-pixel delivery edge, crop the generated 1088-pixel edge after inference.

Parameters

Parameter	Description	Default
--model-root	MiniMax H3 directory containing FL2VA/	Required
--adapter	Local LoRA directory; omit to download and reuse the official TaoMate-H3 adapter	models/TaoMate-H3
--prompt	One prompt reused for every five-second block	—
--prompt-json	JSON file with one prompt per five-second block	—
--duration	Total duration in seconds; must be a multiple of 5	5, or inferred from JSON
--resolution	WIDTHxHEIGHT; short edge 480, 768, or 1088; both edges divisible by 32	768x1376
--gpus	Local inference GPU count: 4 or 8	8
--devices	Comma-separated CUDA device IDs	0 to gpus-1
--seed	Authored request seed	8301
--output	New or empty output directory	Required

--prompt and --prompt-json are mutually exclusive.

Performance and Advantages

The following T2AV results were measured on one 8 × NVIDIA H20 96 GB node with TP2 × Ulysses4, a 480x864 canvas, a 10-second output, and seed 8301.

Metric	TaoMate-H3	MiniMax H3	Improvement
Pure DiT time	14.810 s	169.572 s	11.45× faster
First final chunk latent	6.148 s	170.052 s	27.66× faster
Benchmark first playable video	17.287 s	183.313 s	10.60× faster
Peak DiT memory allocated	31.37 GiB	32.03 GiB	-

Pure DiT time excludes model loading, text encoding, VAE decoding, and media encoding. First playable video includes Video VAE decoding and H.264 publication in the matched first-chunk publication benchmark. The table covers the Stage3 generation path and excludes the command's internal audio preparation. A 10-second TaoMate-H3 run contains 24 generation forwards and eight clean-KV updates.

🎬 FL2VA: real-time streaming first/last-frame control

The FL2VA inference subproject lives in src/taomate-fl2av. Its code is published separately in the TaoMate-H3 GitHub repository. It includes one example: seated_001 · seated presentation · family living-room organization.

Option	Native block frames	Native output	Requests
--time 30	[39] + [34] * 19 + [85]	770 frames / 32.083333s	Three continuous requests, seven blocks each
--time 10	[39] + [34] * 4 + [85]	260 frames / 10.833333s	One request, six blocks
--time 5	[39,85]	124 frames / 5.166667s	One request

All modes use 24fps, 768×1376, TP2/SP4 on eight GPUs, three-step denoising, recent-seven clean AV-KV and one-shot whole-clip decoding.

# Download checkpoint
hf download TaoLiveAIGC/TaoMate-H3-FL2VA \
  --include "config.json" "adapter_config.json" \
            "adapter_model.safetensors" "LICENSE" "NOTICE" \
  --local-dir models/TaoMate-H3-FL2VA

# Run Taomate-FL2VA
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python src/taomate-fl2av/infer.py \
  --time 30 --model-root /absolute/path/models/MiniMax-H3 \
  --checkpoint models/TaoMate-H3-FL2VA \
  --output outputs/seated_001_30

License

TaoMate-H3 is released under the MiniMax H3 Community License Agreement. Use and distribution must follow the terms of that license.

Citation

If you find our work helpful, feel free to cite us.

@misc{taomate2026,
      title={TaoMate: Anchor-Guided Memory Bridging Evolving and Reference States for Real-Time Audio-Video Digital Human Generation}, 
      author={Qijun Gan and Chenwei Zhang and Meiguang Jin and Junfeng Ma and Qiu Shen},
      year={2026},
      eprint={2607.24359},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.24359}, 
}

Acknowledgements

We thank the teams and contributors behind:

• MiniMax H3
• Qwen3-VL
• PyTorch
• FlashAttention
• Triton
• vLLM