LeWAM: Latent Evolving World Action Model

Xueji Fang^1,2,3, Boqiang Duan³, Hua Wu³, Jingdong Wang^3,†,‡, Guo-Jun Qi^2,†

¹ Zhejiang University   ² Westlake University   ³ Baidu Inc.

^† Corresponding authors   ^‡ Project leader

Paper | Code | Model

Overview

LeWAM learns robot actions and visual dynamics in JEPA embedding space, without a pretrained video diffusion backbone. Our controlled comparison of frozen visual encoders finds I-JEPA-Huge most effective for action generation among the evaluated encoders.

• Joint action and world modeling. A single Transformer learns action generation and action-conditioned future embedding prediction. A structured attention mask prevents demonstrated actions from leaking into action generation; inference generates actions only.
• AdaFuse. Learned channel-wise weights combine features across frozen I-JEPA layers to form the current visual representation.
• DemoDPO. Demonstrations rank action candidates from a frozen reference policy, enabling offline preference refinement without additional environment interaction, reward supervision, or human preference labels.

With 0.4B trainable parameters, LeWAM achieves 92.28% average success on RoboTwin 2.0. Its policy is trained on RoboTwin demonstrations without additional robot pretraining data; the frozen I-JEPA encoder retains its ImageNet-22K pretraining.

This repository hosts the official model checkpoints for Latent evolving World Action Model.

News

• [2026-09-24] Our paper is now available on arXiv.
• [2026-09-23] We release the inference code and model weights.

RoboTwin 2.0 Results

Comparison with Other Models

LeWAM achieves the highest average success rate among the models listed below, with 0.4B trainable parameters and no additional robot pretraining.

All success rates are reported as percentages. Results are from Table 2 of the paper.

Model	Visual Encoder	Trainable Params.	Robot Pretraining	Clean ↑	Randomized ↑	Average ↑
π₀	SigLIP	3B	✓	65.92	58.40	62.16
π₀.₅	SigLIP	3B	✓	82.74	76.76	79.75
Motus w/o Pretrain	Wan2.2-VAE	6B	✗	77.56	77.00	77.28
LingBot-VA w/o Pretrain	Wan2.2-VAE	5B	✗	80.60	—	80.60
GigaWorld-Policy	Wan2.2-VAE	5B	✓	86.36	85.04	85.70
Motus	Wan2.2-VAE	6B	✓	88.66	87.02	87.84
Fast-WAM	Wan2.2-VAE	6B	✗	91.88	91.78	91.83
LingBot-VA	Wan2.2-VAE	5B	✓	92.90	91.50	92.20
LaWAM	DINOv3-B/16	1.7B	✓	92.64	89.80	91.22
LeWAM (Ours)	I-JEPA-H/14	0.4B	✗	93.14	91.42	92.28

Bold indicates the highest success rate in each column. — indicates an unreported result. The Average entry for LingBot-VA w/o Pretrain reflects its available Clean result only.

Robot Pretraining refers to additional robot pretraining, not visual-encoder pretraining. LeWAM's frozen I-JEPA encoder retains its ImageNet-22K pretraining.

Component Ablations

All LeWAM variants use the same frozen I-JEPA-H/14 encoder and 0.4B trainable parameters, without additional robot pretraining.

Variant	Clean ↑	Randomized ↑	Average ↑
Action Only	88.42	83.92	86.17
+ WM (FM)	88.18	84.24	86.21
+ WM	88.86	85.18	87.02
+ WM + AdaFuse	91.32	90.06	90.69
LeWAM (+ DemoDPO)	93.14	91.42	92.28

WM denotes world modeling through future embedding prediction. WM (FM) uses flow matching for future embedding prediction.

AdaFuse improves average success from 87.02% to 90.69%. DemoDPO further improves it to 92.28%, using demonstration-derived preferences without additional environment interaction.

For the complete experimental protocol, baseline configurations, encoder comparisons, and real-world manipulation results, see the paper.

Model Details

Property	Description
Model name	LeWAM
Expanded name	Latent Evolving World Action Model
Model type	JEPA-space World Action Model
Task	Robot manipulation
Vision encoder	Frozen I-JEPA-Huge
Trainable model size	Approximately 0.4B parameters
Action dimension	14 in the released RoboTwin configuration
Action prediction horizon	32 actions per inference call
Default inference steps	10
Scheduler	FlowMatchEulerDiscreteScheduler
Evaluation code	Custom LeWAM pipeline and RoboTwin evaluation runner

The 0.4B parameter count refers to trainable parameters, excluding the frozen vision encoder.

The vision encoder is based on facebook/ijepa_vith14_22k and remains frozen during LeWAM training.

Checkpoint Contents

Path	Contents
model_index.json	LeWAM pipeline configuration
predictor/	LeWAM predictor configuration and weights
vision_encoder/	Frozen I-JEPA-Huge encoder, configuration, and preprocessing files
scheduler/	Inference scheduler configuration
stats/dataset_stats.json	RoboTwin action statistics

Download the complete repository together, including the vision encoder and action statistics:

hf download XuejiFang/LeWAM --local-dir outputs/lewam

Getting Started

Installation

Clone the official code repository:

git clone https://github.com/XuejiFang/LeWAM.git
cd LeWAM

Set up the Python 3.10 evaluation environment with uv:

git submodule update --init --recursive
uv run --no-project --python 3.10 scripts/setup_environment.py

For evaluation, download the simulation assets into third_party/RoboTwin/assets, then update the embodiment configuration paths:

cd third_party/RoboTwin
../../.venvs/robotwin/bin/python scripts/update_embodiment_config_path.py
cd ../..

Evaluation requires an NVIDIA driver, CUDA 12.x toolkit, a C++ compiler, libvulkan1, and ffmpeg.

Inference Interface

LeWAM uses a custom robotics pipeline implemented in the official code repository. Its checkpoint format uses Diffusers components, but it is not a text-to-image model.

Use the LeWAM inference and evaluation code rather than a generic text-to-image DiffusionPipeline example. The released configuration predicts 32 actions per inference call using 10 inference steps. Action execution and replanning are handled by the evaluation runner.

Run Evaluation

Download the complete LeWAM checkpoint, including the I-JEPA encoder, to outputs/lewam, and evaluate a single task:

.venvs/robotwin/bin/hf download XuejiFang/LeWAM --local-dir outputs/lewam

.venvs/robotwin/bin/python experiments/robotwin/run_robotwin_manager.py \
  EVALUATION.task_name=adjust_bottle

Omit EVALUATION.task_name=adjust_bottle to evaluate all 50 tasks.

By default, evaluation uses seed 42 and 100 expert-validated scenes for each task and setting, covering demo_clean and demo_randomized. Results and generated scene lists are saved under outputs/evaluation/.

GPU selection, concurrency, and video settings are configured in configs/eval/sim_robotwin.yaml.

Custom Checkpoints

.venvs/robotwin/bin/python experiments/robotwin/run_robotwin_manager.py \
  ckpt=/path/to/epoch_10_ema \
  EVALUATION.dataset_stats_path=/path/to/dataset_stats.json

Fixed-Case Evaluation

.venvs/robotwin/bin/python experiments/robotwin/run_robotwin_manager.py \
  EVALUATION.cases_dir=/path/to/cases

For fixed-case evaluation, name each saved case list <task>__<setting>.json.

License

The bundled I-JEPA-Huge vision encoder is licensed under CC BY-NC 4.0. See the encoder's original model card and the documentation in vision_encoder/ for details.

This license statement applies to the bundled vision encoder. Check the applicable licenses for the LeWAM predictor, code, and other components before redistribution or commercial use.

Citation

If you use LeWAM in your research, please cite:

@article{fang2026lewam,
  title={Latent evolving World Action Model},
  author={Fang, Xueji and Duan, Boqiang and Wu, Hua and Wang, Jingdong and Qi, Guo-Jun},
  journal={arXiv preprint arXiv:2609.27455},
  year={2026}
}

Acknowledgements

We thank LeWM, Fast-WAM, and RoboTwin for their code and resources.