Instructions to use graahand/qwen-vl-2b-lora with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries
PEFT
How to use graahand/qwen-vl-2b-lora with PEFT:
```
Task type is invalid.
```

How to use graahand/qwen-vl-2b-lora with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="graahand/qwen-vl-2b-lora")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("graahand/qwen-vl-2b-lora", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use graahand/qwen-vl-2b-lora with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "graahand/qwen-vl-2b-lora"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "graahand/qwen-vl-2b-lora",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/graahand/qwen-vl-2b-lora

SGLang

How to use graahand/qwen-vl-2b-lora with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "graahand/qwen-vl-2b-lora" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "graahand/qwen-vl-2b-lora",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "graahand/qwen-vl-2b-lora" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "graahand/qwen-vl-2b-lora",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use graahand/qwen-vl-2b-lora with Docker Model Runner:
```
docker model run hf.co/graahand/qwen-vl-2b-lora
```

Configuration Parsing Warning:Config file tokenizer_config.json cannot be fetched (too big)

Configuration Parsing Warning:Config file adapter_config.json cannot be fetched (too big)

Qwen3-VL-2B OCR LoRA

LoRA adapter fine-tuned for OCR-style key-value extraction from document images.

Model Details

Model Description

Developed by: graahand
Model type: Vision-language model adapter (LoRA)
Language(s) (NLP): English
License: Same as base model
Finetuned from model: Qwen/Qwen3-VL-2B-Instruct

Model Sources

Repository: https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct

Uses

Direct Use

Use this adapter with the base model for document OCR and key-value extraction.

Out-of-Scope Use

High-stakes decisions without human review
Extraction from heavily corrupted, blurred, or non-document images

Bias, Risks, and Limitations

May hallucinate values not present in the image.
Performance depends on image quality and layout similarity to training data.
Not audited for sensitive or personally identifying information.

How to Get Started with the Model

from transformers import AutoProcessor, AutoModelForVision2Seq
from peft import PeftModel
from PIL import Image
import torch

base_id = "Qwen/Qwen3-VL-2B-Instruct"
adapter_id = "graahand/qwen-vl-2b-lora"

model = AutoModelForVision2Seq.from_pretrained(base_id, device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)
processor = AutoProcessor.from_pretrained(adapter_id, trust_remote_code=True)

image = Image.open("your_image.png").convert("RGB")
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "Extract all key-value pairs from this document."},
        ],
    },
]

prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[prompt], images=[image], return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=128)

print(processor.tokenizer.decode(out[0], skip_special_tokens=True))

Training Details

Training Data

Custom OCR dataset with lines formatted as:

image_path<TAB>key: value

Training Procedure

Preprocessing

RGB conversion
Resize to 448x448

Training Hyperparameters

Epochs: 3
Train batch size: 4
Eval batch size: 4
Warmup steps: 500
Weight decay: 0.01
LoRA r: 8
LoRA alpha: 16
Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Precision: 4-bit NF4 quantization

Evaluation

No formal benchmark metrics reported yet.

Technical Specifications

Model Architecture and Objective

LoRA adapter for Qwen/Qwen3-VL-2B-Instruct for image-text-to-text generation.

Software

transformers
peft
torch

Model Card Contact

[More Information Needed]

Framework versions

PEFT 0.18.1

Downloads last month: 14

Model tree for graahand/qwen-vl-2b-lora

Base model

Qwen/Qwen3-VL-2B-Instruct

Adapter

(99)

this model