README.md

---
base_model:
- Wan-AI/Wan2.1-I2V-14B-480P-Diffusers
license: cc-by-nc-4.0
pipeline_tag: image-to-video
tags:
- Painting
library_name: diffusers
---

# Loomis Painter: Reconstructing the painting process

<p align="center">
  <a href='https://github.com/Markus-Pobitzer/wlp'>
    <img src='https://img.shields.io/badge/github-repo-blue?logo=github'></a>
  <a href='https://arxiv.org/abs/2511.17344'>
    <img src='https://img.shields.io/badge/Arxiv-Pdf-A42C25?style=flat&logo=arXiv&logoColor=white'></a>
  <a href='https://markus-pobitzer.github.io/lplp'>
    <img src='https://img.shields.io/badge/Project-Page-green?style=flat&logo=Google%20chrome&logoColor=white'></a>
</p>


<table>
  <tr>
    <td align="center">
      <img src="assets/base.gif" width="380" alt="Generated Video" />
      <br />
      <sub>Generated Video</sub>
    </td>
    <td align="center">
      <img src="assets/reference_image.png" width="380" alt="Input" title="Haystacks by Claude Monet. Source: Wikiart." />
      <br />
      <sub>Input</sub>
    </td>
  </tr>
</table>

## Base Model Inference

Before running the code make sure to have installed torch, diffusers, transformers, huggingface_hub, and pillow. You can also install the dependencies from the offical Loomis Portrait repo [link](https://github.com/Markus-Pobitzer/wlp).

```python
import torch
from diffusers import AutoencoderKLWan, WanImageToVideoPipeline
from diffusers.utils import export_to_video, load_image
from transformers import CLIPVisionModel
from huggingface_hub import hf_hub_download
from typing import List, Tuple, Union
from PIL import Image, ImageOps


def pil_resize(
    image: Image.Image,
    target_size: Tuple[int, int],
    pad_input: bool = False,
    padding_color: Union[str, int, Tuple[int, ...]] = "white",
) -> Image.Image:
    """Resizing it to the target size.

    Args:
        image: Input image to be processed.
        target_size: Target size (width, height).
        pad_input: If set resizes the image while keeping the aspect ratio and pads the unfilled part.
        padding_color: The color for the padded pixels.

    Returns:
        The resized image
    """
    if pad_input:
        # Resize image, keep aspect ratio
        image = ImageOps.contain(image, size=target_size)
        # Pad while keeping image in center
        image = ImageOps.pad(image, size=target_size, color=padding_color)
    else:
        image = image.resize(target_size)
    return image


def undo_pil_resize(
    image: Image.Image,
    target_size: Tuple[int, int],
) -> Image.Image:
    """Undo the resizing and padding of the input image to the a new image with size target_size.

    Args:
        image: Input image to be processed.
        target_size: Target size (width, height).

    Returns:
        The resized image
    """
    tmp_img = Image.new(mode="RGB", size=target_size)
    # Get the resized image size
    tmp_img = ImageOps.contain(tmp_img, size=image.size)

    # Undo padding by center cropping
    width, height = image.size
    tmp_width, tmp_height = tmp_img.size

    left = int(round((width - tmp_width) / 2.0))
    top = int(round((height - tmp_height) / 2.0))
    right = left + tmp_width
    bottom = top + tmp_height
    cropped = image.crop((left, top, right, bottom))

    # Undo resizing
    ret = cropped.resize(target_size)
    return ret

# Set to True if you have a GPU with less than 80GB VRAM --> Very slow inference!
enable_sequential_cpu_offload = True

# Download the LoRA file
lora_path = hf_hub_download(repo_id="Markus-Pobitzer/wlp-lora", filename="base.safetensors")
print(f"LoRA path: {lora_path}")

# Loads the pipeline
model_id = "Wan-AI/Wan2.1-I2V-14B-480P-Diffusers"
vae = AutoencoderKLWan.from_pretrained(model_id, subfolder="vae", torch_dtype=torch.float32)
image_encoder = CLIPVisionModel.from_pretrained(
    model_id, subfolder="image_encoder", torch_dtype=torch.float32
)
# Takes more than 100 GB of disk space
pipe = WanImageToVideoPipeline.from_pretrained(
    model_id, vae=vae, image_encoder=image_encoder, torch_dtype=torch.bfloat16
)

# Load LoRA
pipe.load_lora_weights(lora_path)
pipe.fuse_lora()

# Either offload or directly to GPU
if enable_sequential_cpu_offload:
    pipe.enable_sequential_cpu_offload()
else:
    pipe.to("cuda")


### INFERENCE ###
image = load_image(
    "https://uploads3.wikiart.org/images/claude-monet/haystacks-at-giverny.jpg"
)
og_size = image.size
height = 480
width = 832
# Resize and pad
ref_image = pil_resize(image, target_size=(width, height), pad_input=True)
prompt = "Painting process step by step."

output = pipe(
    image=ref_image,
    prompt=prompt,
    height=height,
    width=width,
    num_frames=81,
    output_type="pil",
    guidance_scale=1.0,
).frames[0]
# To original image size
output = [undo_pil_resize(img, og_size) for img in output][::-1]
# Save video
export_to_video(output, "output.mp4", fps=3)
```

### Art Media Transfer
To transfer from one art media to the other use following LoRA:

```python
lora_path = hf_hub_download(repo_id="Markus-Pobitzer/wlp-lora", filename="art_media_transfer.safetensors")
```

Make sure that you also change the prompt accordingly. The supported art medias are:
- acrylic
- colored pencils
- loomis
- pencil
- oil

The prompt has following format:
```python
art_media = "..."
painting_desc = "..."
prompt = f"<{art_media}> Painting process step by step. {painting_desc}"
```

For acrylic, colored pencils and oil the prompt can contain color descriptions, i.e.
```
prompt = f"<acrylic> Painting process step by step. The image depicts a serene landscape with a small brown and green island in the center of a body of water, surrounded by green trees and a few boats. The sky is blue with scattered clouds, and there are birds flying in the background."
```

For the loomis and pencil art media we left the color information out during fine tuning, i.e.
```
prompt = f"<pencil> Painting process step by step. The image depicts a serene landscape with a small island in the center of a body of water, surrounded by trees and a few boats. There are scattered clouds, and birds flying in the background."
```

Note that the loomis method only works on portrait photos/paintings and otherwise seems to fall back to an other art media.


## Citation

If you use this work, please cite:

```bibtex
@misc{pobitzer2025loomispainter,
      title={Loomis Painter: Reconstructing the Painting Process},
      author={Markus Pobitzer and Chang Liu and Chenyi Zhuang and Teng Long and Bin Ren and Nicu Sebe},
      year={2025},
      eprint={2511.17344},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2511.17344},
}
```