silv4souz4/sam2.1-hiera-tiny-coreai-w8

Model Card for SAM 2.1 Hiera Tiny — Apple Core AI, 8-bit palettized (W8 "Identity V2")

Format conversion of Meta's facebook/sam2.1-hiera-tiny promptable image segmenter to Apple Core AI (.aimodel) with 8-bit weight palettization, for on-device inference on Apple silicon. Tested on macOS 27 (Apple M5); not tested on iOS (see Bias, Risks, and Limitations).

Model Details

Model Description

SAM 2 (Segment Anything Model 2) predicts object masks from point and box prompts. This repository contains the image path of SAM 2.1 Hiera Tiny — image encoder, prompt encoder and mask decoder — as three Core AI assets. The video memory components of SAM 2 are not included.

• Original model developed by: Meta FAIR (Ravi et al., 2024).
• Conversion by: silv4souz4 (independent; not affiliated with or endorsed by Meta or Apple).
• Model type: promptable image segmentation (Hiera image encoder + SAM prompt encoder + two-way transformer mask decoder).
• Base model: facebook/sam2.1-hiera-tiny (official checkpoint, SHA-256 7402e0d864fa82708a20fbd15bc84245c2f26dff0eb43a4b5b93452deb34be69).
• Relation to base: quantized (8-bit k-means palettization of the weights); architecture and outputs unchanged.
• License: Apache License 2.0, inherited from SAM 2 (see License and Attribution).

Model Sources

• Original code: https://github.com/facebookresearch/sam2
• Paper: https://arxiv.org/abs/2408.00714
• Other sizes of this conversion:
• silv4souz4/sam2.1-hiera-small-coreai-w8 — Small
• silv4souz4/sam2.1-hiera-base-plus-coreai-w8 — Base Plus
• silv4souz4/sam2.1-hiera-large-coreai-w8 — Large

Uses

Direct Use

Interactive or programmatic segmentation of objects in still images on Apple devices: one or more clicks (positive or negative) and/or a bounding box per object, producing three candidate masks with predicted IoU scores.

Downstream Use

As a building block in on-device apps: photo editing (object selection, background removal), annotation tools, pre-labelling of datasets, or as a mask source for inpainting and compositing.

Out-of-Scope Use

• Video object segmentation and tracking (the memory attention / memory encoder of SAM 2 are not converted).
• Prompt-free "segment everything" without a host-side point grid and post-processing (not provided here).
• Text- or class-conditioned segmentation (SAM 2 has no semantic labels).
• Any use that violates applicable law, including surveillance or identification of individuals without consent.

Bias, Risks, and Limitations

• The conversion inherits all limitations of SAM 2.1 described by its authors: masks may miss thin or fine structures, merge adjacent objects of similar appearance, or depend strongly on where the prompt is placed. SAM 2 produces masks, not labels, and is not a classifier. See the SAM 2 paper for the authors' fairness and limitation analyses.
• 8-bit palettization changes the weights slightly; individual masks can differ from the original, especially near ties between the three candidates.
• fp16 numerics differ across compute units (GPU, Neural Engine, CPU); validate on the unit and OS you ship.
• Assets were produced with a beta toolchain (macOS 27.0, Xcode 27 beta); later runtimes may change behaviour.
• iOS is untested for these exact assets. The Core AI format targets iOS 27 as well, but in a separate test the iOS Neural Engine compiler (--platform iOS --architecture h15p) rejected the SAM 2.1 Tiny image encoder in its original structure, so on iPhone the encoders may run on the GPU only; measure before relying on it.
• In an earlier on-device experiment with this Tiny encoder on an iPhone (architecture h15p), loading the portable asset spent tens of minutes in on-device specialization (the app looks frozen), while the AOT bundle compiled on a Mac in about a second and loaded in under 0.3 s; the iOS Neural Engine compiler rejected the encoder, so it ran on the GPU (about 1.4–1.5 s per image). These assets are otherwise untested on iOS.

Recommendations

Keep a human in the loop for consequential decisions; check masks visually; compare against the original PyTorch model on your target device; prefer box + click prompts for ambiguous objects (box prompts were markedly more stable in our app tests).

How to Get Started with the Model

Download the whole repository (assets, config.json and checksums):

hf download silv4souz4/sam2.1-hiera-tiny-coreai-w8 --local-dir sam2.1-hiera-tiny-coreai-w8

config.json is a machine-readable summary of the functions, tensor shapes, preprocessing and quantization below (it is also the file the Hub uses to count downloads of this repository).

File	Size	Entry point
sam2.1_hiera_tiny_encoder_w8_identity_v2.aimodel	27.5 MB	sam21_tiny_encoder_w8_v2_7402e0d864fa
sam2.1_hiera_tiny_prompt_w8_identity_v2.aimodel	19 KB	sam21_tiny_prompt_w8_v2_7402e0d864fa
sam2.1_hiera_tiny_decoder_w8_identity_v2.aimodel	8.5 MB	sam21_tiny_decoder_w8_v2_7402e0d864fa

Each asset has one function named sam21_<size>_<component>_w8_v2_<sha12> (first 12 hex digits of the SHA-256 of the original checkpoint), so different sizes can be loaded in the same process. SHA256SUMS.txt lists every file.

Function	Inputs	Outputs
encoder	image float16 [1,3,1024,1024]	high_res_0 [1,32,256,256], high_res_1 [1,64,128,128], image_embedding [1,256,64,64]
prompt	point_coords float32 [1,2,2], point_labels int32 [1,2]	sparse_prompt_embedding float16 [1,256,1,2]
decoder	image_embedding, high_res_0, high_res_1, sparse_prompt_embedding	low_res_masks [1,3,256,256], iou_scores [1,3], object_score_logits [1,1]

• Image: stretch (no letterbox) to 1024×1024, RGB, (v/255 − mean)/std with ImageNet mean [0.485, 0.456, 0.406] and std [0.229, 0.224, 0.225], NCHW float16 — as SAM2Transforms in the official code.
• Prompt: coordinates scaled per axis to the 1024 grid (x·1024/width, y·1024/height). Labels: 1 positive click, 0 negative click, 2/3 box top-left/bottom-right, −1 padding. One click = [click, padding]; a box = [top-left, bottom-right] with labels [2, 3]. The prompt asset has a fixed shape of 2 points.
• Output: take the mask with the highest iou_scores, upsample its 256×256 logits to the image size, threshold at 0.

import CoreAI

let url = URL(filePath: "sam2.1_hiera_tiny_encoder_w8_identity_v2.aimodel")
let encoder = try await AIModel(contentsOf: url, options: .default)
let fn = try encoder.loadFunction(named: "sam21_tiny_encoder_w8_v2_7402e0d864fa")!
var out = try await fn.run(inputs: ["image": image])   // NDArray float16 [1,3,1024,1024]
let embedding = out.remove("image_embedding")!.ndArray!

Ahead-of-time compilation (.aimodelc) — tutorial

The files here are portable .aimodel assets. Apple's documentation, Compiling Core AI models ahead of time, explains that every Core AI model must be specialized to the device before inference; the coreai-build tool moves the most expensive part of that specialization, model compilation, to your build machine, so the model loads faster in your app. It produces one .aimodelc per device architecture; you ship (or download) only the one that matches the device. According to Apple, AOT compilation targets devices that support Apple Intelligence (iPhone/iPad with A17 Pro or later, Mac with M1 or later, Apple Vision Pro); it is not supported on tvOS and watchOS.

1. Install the Metal Toolchain (Xcode > Settings > Components > Metal Toolchain, or):

xcodebuild -downloadComponent MetalToolchain

2. Compile (all architectures of a platform, or one with --architecture):

xcrun coreai-build compile sam2.1_hiera_tiny_encoder_w8_identity_v2.aimodel --platform macOS --min-deployment-version 27.0 --output compiled/
xcrun coreai-build compile sam2.1_hiera_tiny_encoder_w8_identity_v2.aimodel --platform iOS --min-deployment-version 27.0 --output compiled/

Repeat for the prompt and decoder assets. Output files are named <asset>.<arch>.aimodelc (for example .h17g for the Apple M5 used here; the iPhone we tested reports h15p); each works on any OS at or above --min-deployment-version. Run coreai-build compile --help for --preferred-compute and the other options.

3. Load the matching bundle — the same API as for .aimodel:

let arch = AIModel.deviceArchitectureName
let url = compiledDir.appending(path: "sam2.1_hiera_tiny_encoder_w8_identity_v2.\(arch).aimodelc")
let encoder = try await AIModel(contentsOf: url, options: .default)

Apple recommends hosting the per-architecture bundles and downloading only the matching one (for example with the BackgroundAssets framework). Some on-device specialization still remains after AOT, depending on the model and compute units.

What we measured (Apple M5, macOS 27.2, Swift runtime, image encoder, first load with an empty specialization cache):

Step	Portable .aimodel	AOT .aimodelc
Compile on the Mac (coreai-build, per asset)	—	0.1–3 s
First load (model + function), cold cache	0.4–0.5 s	0.1 s
Later loads (cache warm)	0.1 s	0.1 s

• AOT changes load time, not the math: on the 501-case benchmark the M5 AOT builds of Tiny and Large produced byte-identical outputs to their portable assets, in both loading orders. Inference latency was within run-to-run noise on this fanless machine — AOT is not a speed-up for inference.
• The benefit grows with model size and is largest on iPhone, where on-device specialization is slow; the cache is also rebuilt after every OS update, so a portable asset pays the cold cost again.
• When targeting iOS, capture stderr of coreai-build: a Neural Engine compile failure is reported only there (ANECCompileOffline … CompilationFailure) while the tool still exits 0 and the bundle runs on the GPU. Set MPSGRAPH_DISABLE_ANE_CACHING=1 when checking, so a cached earlier compile does not hide the message.

Training Details

Training Data

No training or fine-tuning was performed. The weights come from the official SAM 2.1 checkpoint, trained by Meta as described in the SAM 2 paper (the SA-V and SA-1B datasets among others).

Training Procedure (conversion)

1. Official checkpoint verified by SHA-256 (7402e0d864fa82708a20fbd15bc84245c2f26dff0eb43a4b5b93452deb34be69).
2. The image path was wrapped into three modules (encoder, prompt encoder, mask decoder) and exported with torch.export.
3. Compression: 8-bit k-means palettization (one 256-entry LUT per weight tensor) with coreai-opt. Kept in higher precision on purpose: encoder patch_embed.proj (uncompressed), the 7 scaled-dot-product-attention blocks of the mask decoder (fp32 with fp16 edges), and the whole prompt encoder (uncompressed; fp32 coordinates, int32 labels).
4. Conversion with coreai-torch (TorchConverter) to portable .aimodel; one uniquely named function per asset.

Evaluation

Testing Data, Factors & Metrics

COCO Panoptic val2017: 200 images selected with seed 42, 501 objects, one positive click per object taken from the ground-truth mask. Metric: mask IoU of the best of the three masks by predicted IoU, threshold 0. Original = official PyTorch fp32; W8 = this conversion on Core AI (default), Apple M5, macOS 27.0.

Results

	Original fp32	W8 Core AI
Mask IoU	0.638	0.64

Other metrics (original → W8): Dice 0.729 → 0.730, Boundary F1 (2 px) 0.414 → 0.414, oracle IoU 0.797 → 0.798.

Encoder median 273 ms, decoder median 2.8 ms per prompt (Apple M5, macOS 27.0, default; one machine, fanless laptop — latency varies with load and thermals; on macOS 27.2 we measured noticeably higher and unstable values).

Summary

The 8-bit conversion matches the original within ±0.002 IoU on this benchmark.

Environmental Impact

No training. Conversion and evaluation ran locally on a single Apple M5 laptop for a few hours; emissions are negligible compared with the original training reported by Meta.

Technical Specifications

Model Architecture and Objective

Unchanged from SAM 2.1: hierarchical Hiera image encoder with FPN neck (high-resolution features at 1/4 and 1/8 scale), SAM prompt encoder, two-way transformer mask decoder predicting 3 masks, IoU scores and an object-presence logit.

Compute Infrastructure

• Hardware: Apple M5 (Mac17,3), 24 GB unified memory.
• Software: conversion with macOS 27.0, Xcode 27 beta, coreai-torch 0.4.2, coreai-opt 0.2.1, PyTorch 2.11; AOT measurements with macOS 27.2 and coreai-build 3605.5.4.

Citation

If you use these assets, please cite the original SAM 2 work:

@article{ravi2024sam2,
  title={SAM 2: Segment Anything in Images and Videos},
  author={Ravi, Nikhila and Gabeur, Valentin and Hu, Yuan-Ting and Hu, Ronghang and Ryali, Chaitanya and Ma, Tengyu and Khedr, Haitham and R{\"a}dle, Roman and Rolland, Chloe and Gustafson, Laura and Mintun, Eric and Pan, Junting and Alwala, Kalyan Vasudev and Carion, Nicolas and Wu, Chao-Yuan and Girshick, Ross and Doll{\'a}r, Piotr and Feichtenhofer, Christoph},
  journal={arXiv preprint arXiv:2408.00714},
  url={https://arxiv.org/abs/2408.00714},
  year={2024}
}

The evaluation used COCO val2017 images with panoptic annotations (annotations CC BY 4.0; images subject to the Flickr Terms of Use):

@inproceedings{lin2014coco,
  title={Microsoft COCO: Common Objects in Context},
  author={Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C. Lawrence},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2014}
}
@inproceedings{kirillov2019panoptic,
  title={Panoptic Segmentation},
  author={Kirillov, Alexander and He, Kaiming and Girshick, Ross and Rother, Carsten and Doll{\'a}r, Piotr},
  booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2019}
}

License and Attribution

• License: Apache License 2.0 — full text in LICENSE, identical to the license of the SAM 2 model checkpoints ("The SAM 2 model checkpoints, SAM 2 demo code … and SAM 2 training code are licensed under Apache 2.0", SAM 2 README).
• Copyright of the original model: Meta Platforms, Inc. and affiliates.
• Modifications (Apache 2.0 §4(b)): these files are a derivative work. The original PyTorch weights were palettized to 8 bits (with the exceptions listed above) and converted to the Core AI format; the image path was split into three assets. No retraining. See NOTICE.
• Not included and not needed by these assets: the SAM 2 demo fonts (SIL OFL 1.1) and the optional cc_torch connected-components code (LICENSE_cctorch in the SAM 2 repository).
• "Apple", "Core AI" and "Meta" are trademarks of their respective owners; their use here is descriptive only.

More Information

Debug information embedded in the assets contains build-time file paths of the converting machine; it has no effect on inference.

Model Card Authors

silv4souz4

Model Card Contact

Please open a discussion in the Community tab of this repository.