xin1u/UltraFlash

⚡ Ultra Flash

Scaling Real-Time Streaming Video Generation to High Resolutions

     

~30 FPS at 1K  ·  ~18 FPS at 2K  ·  Single GPU  ·  4 Steps

---

🌟 Overview

While recent autoregressive video diffusion models achieve remarkable streaming quality, they remain confined to low resolutions (e.g. 480P), leaving efficient, scalable, real-time high-resolution video generation a fundamental open challenge.

Ultra Flash is the first framework to achieve real-time high-resolution streaming video generation, producing 1K video at ~30 FPS and 2K video at ~18 FPS on a single GPU. It cascades three key components after a low-resolution streaming generator:

	Contribution	What it does
1	Architecture-Preserving T2V-to-TV2V SR Training	Converts any pre-trained T2V model into a generative super-resolution model without architectural modification, paired with an AIGC-oriented degradation pipeline that preserves the base model's generative capability.
2	Causal Streaming Latent Upsampler	Ultralight causal memory network (~2M params) that upsamples latents with spatiotemporal coherence, adding <5% pipeline overhead.
3	Cascaded Streaming Optimization	Hybrid-reward-enhanced sparse causalization + single-step distillation, then cascaded streaming self-forcing preference optimization (DPO) with dynamic cache management.

---

📊 Efficiency

Method	Resolution	Steps	FPS ↑	Latency (ms) ↓	Streaming
Wan2.1	480×832	50	0.78	103,000	✗
CausVid	480×832	4	29.4	34	✓
Self Forcing	480×832	4	32.0	31	✓
Causal Forcing	480×832	4	31.2	32	✓
DummyForcing	480×832	4	28.0	36	✓
Self Forcing + FlashVSR	768×1408	5	15.0	67	✓
Ultra Flash (Ours)	960×1664 (1K)	4	30.0	40	✓
Ultra Flash (Ours)	1440×2496 (2K)	4	18.0	56	✓

---

🏗️ Architecture

Text Prompt
    │
    ▼
┌──────────────────────────┐
│  Self-Forcing Generator  │  480P, 4 denoising steps, streaming
│  (Wan2.1-1.3B causal)    │
└───────────┬──────────────┘
            │ LR latents (16×60×104)
            ▼
┌──────────────────────────┐
│  Causal Latent Upsampler │  2× or 3× spatial upsampling in latent space
│  (~2M params, Conv2D)    │  Causal memory for temporal coherence
└───────────┬──────────────┘
            │ HR latents (16×120×208)
            ▼
┌──────────────────────────┐
│  Sparse SR DiT           │  Single-step denoising, block-sparse attention
│  (1.3B, causal)          │  Adaptive top-k with local window
└───────────┬──────────────┘
            │ Refined HR latents
            ▼
┌──────────────────────────┐
│  Tiny / Ultra Decoder    │  Latent → Pixel (960×1664 or 1440×2496)
│  (Causal Memory Network) │  Sequential streaming mode
└───────────┬──────────────┘
            ▼
       2K Video Output (~18 FPS)

---

📦 Checkpoints in This Repository

File	Role
1K-Causal-Sparse-SR-Dit-fidelity-step_12800.pth	1K sparse causal SR DiT — single-step, block-sparse attention (main 1K model)
2K-Causal-Sparse-SR-Dit-fidelity-step_12800.pth	2K sparse causal SR DiT — single-step, block-sparse attention (main 2K model)
1K-ultraLatentUpsampler-step_19900.pth	Causal streaming latent upsampler for 1K
2K-ultraLatentUpsampler-step_28100.pth	Causal streaming latent upsampler for 2K
ultra-decoder-v3-ema_decoder.pth	Ultra Decoder V3 — high-resolution decoder with improved texture fidelity
v1.1-ultra-decoder-v3-ema_decoder.pth	Ultra Decoder V3 (v1.1 revision)
1K-Bi-Dense-SR-Dit-fidelity-step_12800.pth	Bidirectional dense SR DiT — ablation / reference (non-causal)
1K-teacher-NFE50-Dense-SR-Dit-fidelity-step_14700.pth	NFE-50 dense teacher used for distillation — research reference

The base Wan2.1-T2V-1.3B generator is not redistributed here — download it from Wan-AI/Wan2.1-T2V-1.3B.

---

🚀 Quick Start

1️⃣ Install

conda create -n ultraflash python=3.10 -y
conda activate ultraflash

git clone https://github.com/xin1u/UltraFlash.git && cd UltraFlash/inference
pip install -r requirements.txt
pip install flash-attn --no-build-isolation

# Block Sparse Attention (CUDA kernel, required for SR DiT)
git clone https://github.com/mit-han-lab/Block-Sparse-Attention.git
cd Block-Sparse-Attention && pip install -e . && cd ..

python setup.py develop

Requirements: NVIDIA GPU with 24+ GB memory (RTX 4090 / H200 / B200 tested) · Linux · 64 GB RAM · Python 3.10+

2️⃣ Download Checkpoints

# Base streaming generator
huggingface-cli download Wan-AI/Wan2.1-T2V-1.3B --local-dir inference/wan_models/Wan2.1-T2V-1.3B

# Ultra Flash checkpoints
huggingface-cli download xin1u/UltraFlash --local-dir inference/checkpoints

3️⃣ Generate

One-click inference (2K or 1K):

cd inference && bash inference.sh

Custom inference:

python inference.py \
    --config_path configs/self_forcing_dmd_4step.yaml \
    --checkpoint_path checkpoints/self_forcing_dmd.pt \
    --data_path prompts/examples.txt \
    --output_folder outputs/ \
    --use_ema \
    --tiny_decoder \
    --torch_compile \
    --compile_sr_dit

LR-only mode (480P, for comparison): add --lr_only and drop the SR flags.

---

⚙️ Key Arguments

Argument	Default	Description
--use_ema	False	Load EMA weights (recommended)
--num_output_frames	21	Number of latent frames (~5s video at 21 frames)
--torch_compile	False	Compile SF DiT (~1.5× speedup)
--compile_sr_dit	False	Compile SR DiT
--fp8	False	FP8 quantization for SF DiT
--tiny_decoder	True	Fast HR decoding (faster than Wan VAE)
--ultra_decoder_v3	False	Use Ultra Decoder V3 (better texture, replaces Tiny Decoder)
--sr_kv_len	3	SR DiT KV cache window length
--lr_only	False	Only generate LR video (skip SR cascade)
--dcm_adaptive_refresh	False	Enable IQA-based adaptive cache refresh
--no_dcm	False	Disable all DCM optimizations

Dynamic Cache Management (DCM)

Three inference-time optimizations, all essentially free in quality:

Optimization	Effect	Quality Impact
(i) LR Step Reduction	Saves 1 forward pass per chunk	Negligible
(ii) Adaptive Cache Refresh	Skips context rerun when quality suffices	Negligible
(iii) SR Cache Adaptation	Reduces SR KV cache memory over time	Minimal

---

🎬 Long Video Generation

The inference_long/ directory supports ~10-second videos (120 latent frames) using LongLive with LoRA adapters and the same SR cascade:

cd inference_long
python inference_sr.py \
    --config_path configs/longlive_inference_sr.yaml \
    --use_sparse_sr --tiny_decoder --torch_compile --compile_sr_dit

---

📰 News

• Aug 2026 — Inference code and checkpoints released on GitHub and Hugging Face. 🎉

📋 TODO

• ✅ Release inference code and checkpoints (short + long video)
• ⬜ Release training code and configs

---

🙏 Acknowledgements

This work builds upon these excellent open-source projects:

• FlashVSR — block-sparse attention for streaming video super-resolution
• Self-Forcing — real-time autoregressive video generation with self-rollout training
• TAEHV — tiny autoencoder for high-resolution video decoding
• Wan2.1 — video DiT foundation model

---

📞 Contact

Please feel free to contact us if there is any question (luxion@mail.ustc.edu.cn).

---

📜 Citation

@inproceedings{luxury2026ultraflash,
  title={Ultra Flash: Scaling Real-Time Streaming Video Generation to High Resolutions},
  author={Luxury and Huang, Jie and Fan, Zihao and Ma, Xiaoxiao and Li, Yuming and Zhuang, Jun-hao and Xue, Zeyue and Fu, Siming and Li, Haoran and Zhong, Mingchen and Zhang, Guohui and Ma, Shichen and Liu, Yijun and Shi, Jiaqi and Ma, Yanwen and Su, Yaofeng and Wang, Haoyu and Li, Yaowei and Zhang, Songchun and Jin, Weiyang and Bian, Yuxuan and Zhang, Shiyi and Xu, Haojun and Lu, Shuai and Han, Xin and Tang, Wei and Huang, Haoyang and Duan, Nan},
  booktitle={arXiv preprint},
  year={2026}
}

---

📄 License

Released under the Apache 2.0 License.