Hi-Token: Hierarchical Coordinate Tokenization for Generative Visual Grounding

Paper · Project page · Hugging Face · ModelScope

Overview

Hi-Token is a coordinate representation for generative visual grounding. It represents each bounding-box coordinate with separate axis tokens for the hundreds, tens, and ones digits. A box is expressed as 12 coordinate tokens drawn from 60 token types. This representation encodes numerical scale and axis identity within an autoregressive vision-language model.

Hi-GAR complements Hi-Token with geometry-aware GRPO training. It combines box overlap and coordinate accuracy at multiple scales. The paper calls the model trained with Hi-Token and Hi-GAR Hi-R1. Hi-GAR is used only during training.

This model card is shared between the Hugging Face and ModelScope releases.

Model details

Property	Description
Task	Referring-expression visual grounding
Input	An image and a referring expression
Output	A bounding box encoded with axis-specific digit tokens
Architecture	Qwen2.5-VL
Model class	Qwen2_5_VLForConditionalGeneration
Weight format	Safetensors, bfloat16, two shards
Saved Transformers version	4.56.2
Main paper setting	Qwen2.5-VL-3B-Instruct; coordinate SFT followed by Hi-GAR
Coordinate order	[x_min, y_min, x_max, y_max]
Coordinate range	Integer indices from 0 to 999

Download

Choose either hosting platform. The local checkpoint can be used with the same inference interface.

Hugging Face:

from huggingface_hub import snapshot_download

model_dir = snapshot_download(
    repo_id="xyzzzh/Hi-Token",
    local_dir="./Hi-Token",
)

ModelScope:

from modelscope import snapshot_download

model_dir = snapshot_download(
    "xyzzzh/Hi-Token",
    local_dir="./Hi-Token",
)

Inference

Install PyTorch and a Transformers version supporting Qwen2.5-VL, together with accelerate and pillow. Load the processor and tokenizer from this checkpoint so that the Hi-Token vocabulary is retained.

The grounding prompt used in the paper is:

Please provide the bounding box coordinate of the region this sentence describes: <expr>.

Replace <expr> with the referring expression. Decode generated responses with skip_special_tokens=False; otherwise the coordinate tokens may be removed.

For each coordinate, reconstruct its integer value as 100 × hundreds + 10 × tens + ones. The four coordinates follow the order x_min, y_min, x_max, y_max. Under the paper's 1,000-bin convention, dividing an integer index by 999 recovers the normalized coordinate.

For example, the coordinate triplet <x_hundreds_4><x_tens_7><x_ones_8> represents the x-axis index 478. All 60 axis-specific digit tokens are included in the released tokenizer.

The following example uses the local checkpoint downloaded above and an image saved as image.jpg. A CUDA GPU with enough memory for the model and image tokens is recommended.

import torch
from PIL import Image
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration

model_dir = "./Hi-Token"
processor = AutoProcessor.from_pretrained(model_dir)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    model_dir,
    torch_dtype="auto",
    device_map="auto",
).eval()

image = Image.open("image.jpg").convert("RGB")
expression = "the person on the left"
messages = [{
    "role": "user",
    "content": [
        {"type": "image"},
        {
            "type": "text",
            "text": (
                "Please provide the bounding box coordinate of the region "
                f"this sentence describes: {expression}."
            ),
        },
    ],
}]
prompt = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = processor(
    text=[prompt], images=[image], padding=True, return_tensors="pt"
).to(model.device)

with torch.inference_mode():
    generated = model.generate(
        **inputs, max_new_tokens=128, do_sample=False, use_cache=True
    )
response_ids = generated[:, inputs.input_ids.shape[1]:]
response = processor.batch_decode(
    response_ids,
    skip_special_tokens=False,
    clean_up_tokenization_spaces=False,
)[0]
print(response)

Results reported in the paper

The following table reports Hi-R1 results from the paper. Values are percentages; the full paper specifies the evaluation protocol and controlled comparisons.

Benchmark	mIoU	P@0.5	P@0.95
RefCOCO	84.3	93.1	33.4
RefCOCO+	81.8	89.9	32.9
RefCOCOg	80.3	86.4	39.4

Controlled experiments show gains across the evaluated IoU range and three backbones. Hi-GAR further reduces low-overlap predictions.

Intended use

The model supports visual grounding research and applications that locate image regions from natural language. Coordinate outputs should be parsed with the checkpoint's token vocabulary and interpreted in the normalized coordinate space. Grounding predictions can be incorrect, particularly for ambiguous descriptions and small targets.

Use of the checkpoint is subject to the applicable base-model and dataset terms. Public availability does not replace those terms.

Citation

@article{zhu2026hitoken,
  title   = {Hi-Token: Hierarchical Coordinate Tokenization for Generative Visual Grounding},
  author  = {Zhu, Xiuyuan and Lu, Ke and Dong, Kun and Jiao, Siwen and Wu, Hao and Du, Zijin and Mao, Shun and Zhang, Dongming and Xue, Jian},
  journal = {arXiv preprint arXiv:2608.03471},
  year    = {2026}
}

Contact

For questions about the paper or checkpoint: zhuxiuyuan22@mails.ucas.edu.cn.