LTX-2.5 SDR-To-HDR (Modular Diffusers)

Modular Diffusers blocks that run Lightricks' LTX-2.5 SDR-To-HDR IC-LoRA on a stock diffusers install. An SDR clip goes in; a scene-linear HDR clip (Rec.709 or ACEScg primaries) or its ACEScct codes come out.

Seam keyframes are on by default, as in Lightricks' reference: at every 24- or 32-frame segment seam of the clip, the source frame is encoded on its own and held at keyframe_strength (0.95) as a guide, next to an empty keyframe slot that is generated with the video and then anchors the diffusion decoder.

This repo holds code and a pipeline index only. The weights are loaded from Lightricks/LTX-2.5-Diffusers (transformer, VAE, diffusion decoder, scheduler) and the IC-LoRA repo (adapter and precomputed scene embedding). See huggingface/diffusers#14981 for the discussion.

Requirements

  • Access to the gated Lightricks/LTX-2.5-Diffusers and Lightricks/LTX-2.5-22b-IC-LoRA-SDR-To-HDR repos (accept the LTX-2.x Community License, then hf auth login).
  • diffusers with the LTX-2.5 modular pipelines (LTX25ModularPipeline), peft for the LoRA.
  • trust_remote_code=True: the blocks are defined in block.py here, and the keyframe-aware diffusion decoder in keyframe_decoder.py.
  • kernels (pip install kernels) for the diffusion decoder's NATTEN attention on GPU.
  • Optional, for export: av with an FFmpeg build that includes libx265 (HLG .mp4), OpenEXR>=3.3 (EXR frames).

Usage

import torch
from diffusers import ComponentsManager, ModularPipeline
from diffusers.models.autoencoders.ltx2_diffusion_decoder import LTX2VideoVaeNeighborhoodNattenProcessor
from diffusers.pipelines.ltx2.pipeline_ltx2_ic_lora import LTX2ReferenceCondition
from diffusers.utils import load_video
from diffusers.utils.dynamic_modules_utils import get_class_from_dynamic_module

repo_id = "scenario-labs/ltx25-sdr-to-hdr-modular"
components = ComponentsManager()
components.enable_auto_cpu_offload(device="cuda")
pipe = ModularPipeline.from_pretrained(repo_id, trust_remote_code=True, components_manager=components)
# The VAE and the diffusion decoder run in float32, as in the reference.
# `diffusion_decoder` loads into `LTX25KeyframeDiffusionDecoderModel` (keyframe_decoder.py), see below.
pipe.load_components(dtype={"default": torch.bfloat16, "vae": torch.float32, "diffusion_decoder": torch.float32})
pipe.diffusion_decoder.set_attn_processor(LTX2VideoVaeNeighborhoodNattenProcessor())
pipe.diffusion_decoder.enable_tiling()

pipe.load_lora_weights(
    "Lightricks/LTX-2.5-22b-IC-LoRA-SDR-To-HDR",
    weight_name="ltx-2.5-22b-ic-lora-sdr-to-hdr-1.0.safetensors",
    adapter_name="sdr_to_hdr",
)
pipe.set_adapters("sdr_to_hdr", 1.0)

# An 8-bit sRGB clip of any size; 8k + 1 frames are generated (the largest count the clip covers by default).
sdr_video = load_video("/path/to/sdr.mp4")
frame_rate = 24.0
hdr = pipe(
    reference_conditions=LTX2ReferenceCondition(frames=sdr_video),
    frame_rate=frame_rate,
    output_colorspace="rec709",
    generator=torch.Generator("cuda").manual_seed(42),
    output="videos",
)[0]  # (F, H, W, 3) float32, scene-linear Rec.709

encode_hlg = get_class_from_dynamic_module(
    repo_id, "hdr_export.py", class_name="encode_hdr_tensor_to_hlg_mp4", trust_remote_code=True
)
encode_hlg(hdr, "sdr_to_hdr.mp4", frame_rate=frame_rate)

Seam keyframes are on by default (keyframe_strength=0.95). Pass seam_keyframes=False for the plain IC-LoRA (keyframe_strength=None in the reference; in Modular Diffusers a None input takes the default, so it cannot turn them off), and high_quality_hdr=True to generate twice the frames and keep every second one.

For an ACEScg EXR sequence, run with output_colorspace="acescg" and use export_to_exr_sequence from hdr_export.py (exr_colorspace="acescg").

To load from local copies, pass a per-component pretrained_model_name_or_path dict to load_components (add subfolder={"diffusion_decoder": None} when the decoder folder is not a diffusion_decoder subfolder of a base snapshot), and point the scene embedding at a local folder with pipe.register_to_config(scene_embedding_repo="/path/to/lora/folder") (or pass scene_video_context=).

Keyframe-aware diffusion decoder

diffusers does not decode keyframe planes yet (Lightricks is adding it), so keyframe_decoder.py provides an interim implementation: LTX25KeyframeDiffusionDecoderModel, a subclass of diffusers' LTX2VideoDiffusionDecoderModel that adds decode(..., keyframe_latents=, keyframe_frame_indices=), a port of the reference's keyframe decode (joint neighborhood attention included). Without keyframes, decode is the parent's. It also creates the checkpoint's learned keyframe tag decoder.type_emb, which Lightricks/LTX-2.5-Diffusers/diffusion_decoder holds but the stock class drops as an unexpected key, so every tensor of the checkpoint is loaded.

The blocks declare diffusion_decoder with that class and the base repo's diffusion_decoder subfolder, and this repo's modular_model_index.json deliberately has no diffusion_decoder entry, so load_components loads it with the keyframe-aware class. To load it yourself (e.g. from a local copy):

from diffusers.utils.dynamic_modules_utils import get_class_from_dynamic_module

KeyframeDecoder = get_class_from_dynamic_module(
    repo_id, "keyframe_decoder.py", class_name="LTX25KeyframeDiffusionDecoderModel", trust_remote_code=True
)
decoder = KeyframeDecoder.from_pretrained(
    "Lightricks/LTX-2.5-Diffusers", subfolder="diffusion_decoder", dtype=torch.float32
)
pipe.update_components(diffusion_decoder=decoder)

With the stock LTX2VideoDiffusionDecoderModel, the pipeline warns and decodes without the keyframe slots; without a diffusion decoder, the VAE decodes without them. keyframe_decoder.py will be retired once diffusers ships keyframe-aware decoding.

Validation

Plain IC-LoRA (seam_keyframes=False): on a B200 with the real weights, three clips (720x480 at 8 fps, 1000x560 padded to 1024x576, and a 60 fps clip for the RoPE cap) give bitwise-identical latents, transformer steps and HDR output to a classic LTX2HDRPipeline implementation of the same path, given the same input tensor.

Seam keyframes (49 frames with seams at 24 and 48, 97 frames with seams at 32, 64 and 96), high_quality_hdr and the no-seam fallback (17 frames): on a B200 with the real weights, latents, keyframe slot latents, every transformer step and the HDR output are bitwise identical to the classic implementation, given the same input tensor. A keyframe decode took under 32 s per call at 720x480 with a 63 GB peak.

Bitwise reproducibility of the default sRGB input needs the same CPU family: the sRGB to ACEScct primaries conversion runs on the CPU, and its 3x3 matrix product rounds differently across CPU models (about 60 dB PSNR between AMD and Intel hosts).

Inputs

Name Default
reference_conditions required The SDR clip: an LTX2ReferenceCondition, a list of them, or the frames (list of PIL images, (F, H, W, C) array, (F, C, H, W) tensor).
input_colorspace "srgb_gamma" srgb_gamma (sRGB-encoded, e.g. 8-bit video), srgb (scene-linear Rec.709), acescg (scene-linear AP1), acescct.
height, width the clip's Output size. Generation runs at the size rounded up to 32 (reflect-padded); the output is cropped back.
num_frames largest 8k + 1 in the clip Number of frames to generate.
frame_rate 24.0 Frame rate of the clip. RoPE is conditioned on at most 30 fps, as in the reference.
sigmas / timesteps distilled schedule The 8-step distilled schedule, used verbatim.
keyframe_strength 0.95 Strength of the seam guides, in [0, 1].
seam_keyframes True False runs the plain IC-LoRA, without seam keyframes.
high_quality_hdr False Frame-doubled source, 2N - 1 generated frames, every second frame kept (about twice the cost).
output_type "pt" pt, np, or latent (denormalized, padded latents).
output_colorspace "rec709" rec709 / acescg (scene-linear, [0, inf)) or acescct (codes in [0, 1]).
scene_video_context from the LoRA repo Overrides the precomputed scene embedding (config scene_embedding_repo / _filename / _revision).
generator, num_videos_per_prompt, latents, noise_scale, reference_downscale_factor, attention_kwargs As in the LTX-2 modular pipelines.

Output: videos, a float32 (B, F, H, W, 3) tensor (or array).

Blocks

LTX25SDRToHDRBlocks
  input              LTX25HDRInputStep              size from the clip, pad to 32, 30 fps RoPE cap, distilled sigmas,
                                                    seam positions, frame doubling
  scene_embedding    LTX25HDRSceneEmbeddingStep     precomputed scene embedding instead of a text encoder
  reference_encoder  LTX25HDRReferenceEncoderStep   ACEScct + reflect-pad, float32 VAE encode, seam frames encoded alone
  seam_keyframes     LTX25HDRSeamKeyframesStep      appends [guides | slots] after the reference tokens
  denoise
    prepare_latents        LTX2InContextPrepareLatentsStep   (diffusers)
    keyframe_marker        LTX25HDRKeyframeMarkerStep        first latent frame and seam slots marked
    set_timesteps          LTX2ConditionSetTimestepsStep     (diffusers)
    prepare_audio_latents  LTX25HDRPlaceholderAudioStep      one zero audio token, no RNG
    prepare_coords         LTX2ConditionPrepareCoordsStep    (diffusers)
    denoise                LTX2ConditionLoopBeforeDenoiser -> LTX25HDRLoopDenoiser -> LTX2ConditionLoopAfterDenoiser
  decode
    seam_slots             LTX25HDRSeamSlotsStep             cuts out and denormalizes the generated seam slots
    trim_condition_tokens  LTX2TrimConditionTokensStep       (diffusers)
    video_decode           LTX25HDRVideoDecodeStep           float32 keyframe-aware diffusion decoder (VAE fallback),
                                                             crop, every second frame if frame-doubled, ACEScct -> HDR

The denoiser makes one unguided, video-only transformer pass per step (isolate_modalities=True), so neither a text encoder, an audio VAE, a vocoder nor guiders are needed.

Limitations

  • Seam keyframes need a single reference clip and a transformer with the LTX-2.5 keyframe position embedding; clips shorter than 25 frames have no seam and run the plain IC-LoRA.
  • A keyframe decode always uses a pure-PyTorch joint attention (the NATTEN processor only serves plain decodes).
  • output_type="latent" returns the video latents only (with high_quality_hdr, those of the frame-doubled clip).
  • The scheduler must have use_dynamic_shifting=False (true for Lightricks/LTX-2.5-Diffusers); the blocks raise otherwise.
  • Output is HDR only (pt / np / latent); use hdr_export.py for HLG .mp4 or EXR files.

The model weights are subject to the LTX-2.x Community License; this repo's code is Apache-2.0.

Downloads last month
18
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for scenario-labs/ltx25-sdr-to-hdr-modular

Finetuned
(1)
this model