sona-forge/clip-vit-h-14-image-fp16

Sona Forge — CLIP ViT-H/14 image encoder (ONNX FP16)

Vision-tower-only export of OpenCLIP ViT-H/14 used by IP-Adapter to produce image embeds for SD 1.5 conditioning. Used by the Sona Forge Android app. Pair with sona-forge/sd15-ipadapter-fp16.

ONNX shape

Input	Shape	dtype	Notes
pixel_values	[batch, 3, 224, 224]	FP16	center-cropped + NEAREST-resized + CLIP-mean/std-normalized

Output	Shape	dtype	Notes
image_embeds	[batch, 1024]	FP16	primary output
onnx::Gather_4496	[batch, seq, hidden]	FP16	secondary last_hidden_state passthrough; downstream consumers query by name (image_embeds) and ignore this

How it was made

Pinned conversion environment:

Package	Version
transformers	4.40.0
torch	2.3.0
onnx	1.16.0
numpy	<2 (ABI compat)

Conversion sequence:

1. Load transformers.CLIPVisionModelWithProjection.from_pretrained("h94/IP-Adapter", subfolder="models/image_encoder", torch_dtype=torch.float16). Note: this loads the OpenCLIP ViT-H/14 weights bundled inside the IP-Adapter repo as a convenience copy.
2. model.eval().
3. torch.onnx.export(model, dummy_pixel_values, opset_version=17, input_names=["pixel_values"], output_names=["image_embeds"], dynamic_axes={"pixel_values": {0: "batch"}, "image_embeds": {0: "batch"}}).

The text branch is not exported. Phase 6 spike characterisation used a deterministic synthetic 512×512 fixture; CLIP image-embeds norm = 21.9 (in the typical 15–25 range for natural portraits).

Files

File	Size	sha256
model.onnx	1,264,856,075 B (1206 MB)	038078ba22b67e57ec2ca5b466e0699f30e14a9613054be8c7fb24e537987689

Licence

OpenCLIP ViT-H/14 — MIT. Original training data is LAION-2B.

Preprocessing

The Sona Forge Android-side ClipImageEncoder uses NEAREST interpolation in the resize step, not BILINEAR. Reasoning: Pillow ≥ 10.0's Image.BILINEAR is anti-aliased by default and produces a small but consistent cosine drift (~0.04) versus a Kotlin pure-bilinear implementation. NEAREST on both sides clears the 0.999 cosine threshold the in-tree JVM golden test asserts. If you re-use this ONNX with a different preprocessor, verify cosine match against your own reference embedding.

Memory footprint

1.2 GB FP16 on disk; ORT CPU EP promotes to FP32 at session load (2.4 GB resident). On Android (NNAPI / XNNPack), FP16 runs natively.

Usage

import onnxruntime as ort
import numpy as np
from PIL import Image

clip = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])

CLIP_MEAN = np.array([0.48145466, 0.4578275, 0.40821073], dtype=np.float32)
CLIP_STD = np.array([0.26862954, 0.26130258, 0.27577711], dtype=np.float32)

img = Image.open("portrait.png").convert("RGB")
w, h = img.size
side = min(w, h)
img = img.crop(((w - side) // 2, (h - side) // 2, (w + side) // 2, (h + side) // 2))
img = img.resize((224, 224), Image.NEAREST)
arr = (np.asarray(img, dtype=np.float32) / 255.0 - CLIP_MEAN) / CLIP_STD
arr = arr.transpose(2, 0, 1)[None, ...].astype(np.float16)  # NCHW FP16

image_embeds = clip.run(["image_embeds"], {"pixel_values": arr})[0]  # (1, 1024)

Provenance

• Original weights: laion/CLIP-ViT-H-14-laion2B-s32B-b79K.
• Bundled-copy used by this conversion: h94/IP-Adapter/models/image_encoder.