Qwen3-Embedding-0.6B-fp16-ov

Description

This is Qwen3-Embedding-0.6B model converted to the OpenVINO™ IR (Intermediate Representation) format with weights compressed to FP16.

Compatibility

The provided OpenVINO™ IR model is compatible with:

  • OpenVINO version 2026.0.0 and higher
  • Optimum Intel 1.27.0 and higher

Running Model Inference with Optimum Intel

  1. Install packages required for using Optimum Intel integration with the OpenVINO backend:
pip install "git+https://github.com/huggingface/optimum-intel.git" "torch==2.8" --extra-index-url https://download.pytorch.org/whl/cpu
  1. Run model inference:
import torch
import torch.nn.functional as F

from torch import Tensor
from transformers import AutoTokenizer

from optimum.intel import OVModelForFeatureExtraction


model_id = "OpenVINO/Qwen3-Embedding-0.6B-fp16-ov"

model = OVModelForFeatureExtraction.from_pretrained(model_id, export=False)


def last_token_pool(last_hidden_states: Tensor, attention_mask: Tensor) -> Tensor:
    left_padding = attention_mask[:, -1].sum() == attention_mask.shape[0]
    if left_padding:
        return last_hidden_states[:, -1]
    else:
        sequence_lengths = attention_mask.sum(dim=1) - 1
        batch_size = last_hidden_states.shape[0]
        return last_hidden_states[torch.arange(batch_size, device=last_hidden_states.device), sequence_lengths]


def get_detailed_instruct(task_description: str, query: str) -> str:
    return f"Instruct: {task_description}\nQuery:{query}"


# Each query must come with a one-sentence instruction that describes the task
task = "Given a web search query, retrieve relevant passages that answer the query"

queries = [get_detailed_instruct(task, "What is the capital of China?"), get_detailed_instruct(task, "Explain gravity")]
# No need to add instruction for retrieval documents
documents = [
    "The capital of China is Beijing.",
    "Gravity is a force that attracts two bodies towards each other. It gives weight to physical objects and is responsible for the movement of planets around the sun.",
]
input_texts = queries + documents

tokenizer = AutoTokenizer.from_pretrained(model_id, padding_side="left")

max_length = 8192

# Tokenize the input texts
batch_dict = tokenizer(
    input_texts,
    padding=True,
    truncation=True,
    max_length=max_length,
    return_tensors="pt",
)
batch_dict.to(model.device)
outputs = model(**batch_dict)
embeddings = last_token_pool(outputs.last_hidden_state, batch_dict["attention_mask"])
# normalize embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = embeddings[:2] @ embeddings[2:].T
print(scores.tolist())

For more examples and possible optimizations, refer to the Inference with Optimum Intel.

Running Model with OpenAI client and OpenVINO Model Server

1a. Deploy model on Windows using binary package:

mkdir C:\models
ovms.exe --rest_port 8000 --source_model OpenVINO/Qwen3-Embedding-0.6B-fp16-ov --model_repository_path C:\models

1b. Deploy model in a Docker container:

mkdir -p ${HOME}/models
export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi)
docker run -d ${GPU_ARGS} --user $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/models:/models openvino/model_server:latest-gpu --rest_port 8000 --model_repository_path /models --source_model OpenVINO/Qwen3-Embedding-0.6B-fp16-ov
2. Install the client library:

pip install openai "numpy<2"

3. Run the client:

from openai import OpenAI import numpy as np

client = OpenAI( base_url="http://localhost:8000/v1", api_key="unused" ) model = "OpenVINO/Qwen3-Embedding-0.6B-fp16-ov" embedding_responses = client.embeddings.create( input=[ "That is a happy person", "That is a very happy person" ], model=model, ) embedding_from_string1 = np.array(embedding_responses.data[0].embedding) embedding_from_string2 = np.array(embedding_responses.data[1].embedding) cos_sim = np.dot(embedding_from_string1, embedding_from_string2)/(np.linalg.norm(embedding_from_string1)*np.linalg.norm(embedding_from_string2)) print("Similarity score as cos_sim", cos_sim)


## Limitations

Check the original [model card](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) for limitations.

## Legal information

The original model is distributed under [Apache License Version 2.0](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md) license. More details can be found in [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B).

## Disclaimer

Intel is committed to respecting human rights and avoiding causing or contributing to adverse impacts on human rights. See [Intel’s Global Human Rights Principles](https://www.intel.com/content/dam/www/central-libraries/us/en/documents/policy-human-rights.pdf). Intel’s products and software are intended only to be used in applications that do not cause or contribute to adverse impacts on human rights.
Downloads last month
158
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for OpenVINO/Qwen3-Embedding-0.6B-fp16-ov

Finetuned
(585)
this model