smdesai/d1-omni-600M-CoreAI

d1-omni-600M — Core AI (Neural Engine and GPU)

LiquidAI/d1-omni-600M converted to Apple Core AI, re-authored to run on the Neural Engine, with GPU models for a Mac and as an iPhone fallback for long inputs. d1-omni-600M is a 587M-parameter decision model: give it a state — text, JSON, a photo or a voice clip — and named questions, and it returns typed answers (a yes/no probability, a pick among named options, or a level) read from its own distribution in one pass, with no generated tokens.

These are the portable models behind the D1Omni iPhone app (camera, voice and text decisions on device). Compile them for your device with compile.sh (one command, ~2.5 min).

	iPhone 17 Pro, Neural Engine
Fidelity vs the original model (1,011 golden questions)	100 % same answer; max probability difference 0.042
A 3-question support ticket	97 ms
A photo, 3 questions	0.76 s
A 10 s voice note	155 ms

Files

File	Size	What
ane/trunk_all.aimodel	604 MB	the decision trunk + head for the Neural Engine: 19 fixed-shape entrypoints over one copy of the weights, fp16
ane/vision.aimodel	180 MB	SigLIP2 vision tower + projector, one image crop per call, fp16
ane/audio.aimodel	218 MB	FastConformer audio encoder + adapter, a 30 s window, fp16
gpu/trunk_all.aimodel	599 MB	the trunk for the GPU (trunk_head, prefix_encode, trunk_text), fp16 weights + fp32 math, dynamic shapes
gpu/vision.aimodel, gpu/audio.aimodel	180 + 217 MB	GPU vision and audio
embed_tokens.f16	128 MB	the token embedding table, 65,536 × 1,024 fp16 (looked up on the host)
tokenizer.json, tokenizer_config.json, config.json		the original model's tokenizer and decision config (temperatures, lengths)
manifest.json		graph → model and entrypoint, the buckets, and the compiled bundles once compile.sh has run
compile.sh		compiles the models for a device and registers them in manifest.json
LICENSE		LFM Open License v1.0

Compile for your device

hf download smdesai/d1-omni-600M-CoreAI --local-dir d1-omni-600M-CoreAI && cd d1-omni-600M-CoreAI
bash compile.sh h18p ios     # iPhone 17 Pro / Pro Max (A19 Pro)
bash compile.sh h15c macos   # M3 Max — or your chip's Core AI architecture

compile.sh runs xcrun coreai-build compile (Xcode with Core AI) on the three Neural Engine models (--preferred-compute neural-engine) and, for iOS, the three GPU models (--preferred-compute gpu, the fallback), writing ane/<arch>/*.aimodelc and gpu/<arch>/*.aimodelc. coreai-build exits 0 even when the Neural Engine compiler rejects a graph, so the script counts the Neural Engine regions — one per entrypoint: 19 for the trunk, 1 each for vision and audio — and fails otherwise. The graphs compiled cleanly on all nine iOS architectures tried (h13g … h19p) as separate models; the combined trunk was verified on h15c and h18p.

On a Mac the gpu/*.aimodel files also run directly (Core AI specializes them on first load), which is the faster path there beyond the shortest inputs.

Loading. Only a bundle compiled with --preferred-compute neural-engine and loaded with default specialization options (Swift: AIModel(contentsOf:)) runs on the Neural Engine; a JIT-loaded .aimodel runs on the GPU whatever the preference. An iPhone app needs the com.apple.developer.kernel.increased-memory-limit entitlement (on the iPhone 17 Pro it raises the Neural Engine headroom from 1.9 to 4.75 GB and the GPU's from 3.3 to 6.2 GB).

How the graphs are used

All Neural Engine graphs are fp16 with fixed shapes: the host pads every input to a bucket. Padding is exact — padded positions are zeroed and masked — so a bucket costs compute, never accuracy.

Bundle	Entrypoint	Inputs → outputs
trunk_all	trunk_b1_l{128,256,512,1024,2048}_k{8,16}	embeds (1,L,1024), pad (1,L,1), keep_right (1,L,1), cos/sin (1,L,64), trunk_mask (1,1,L,L), head_mask (1,1,1,L), marker_onehot (1,K,L), qtype_onehot (1,1,3) → logits (1,K)
	prefix_p{512,1024,2048}	a media prefix once: embeds (1,P,1024), pad, cos/sin, mask (1,1,P,P), last (1,1,P) → k_cache, v_cache (6,1,8,P,64), conv_last (10,1,1024)
	text_p{P}_t{128,512}_k16	each question against that cache: embeds (1,T,1024), pad, cos/sin at positions P+t, mask (1,1,T,P+T), the caches, head_mask, marker_onehot (1,16,T), qtype_onehot → logits (1,16)
vision	main	patches (1,1024,768), row_w/col_w (1,1024,16), key_mask (1,1,1,1024) → tokens (1,256,1024)
audio	main	mel (1,128,3001), m0..m3 (1,1,1,{3001,1501,751,376}), att_mask (1,1,376,376) → prefix (1,376,1024)

Masks are built on the host with −40000 for masked positions (the Neural Engine mishandles −inf); RoPE tables (θ = 1e6) and keep_right (where the conv's right tap is cut at the last media position) are host inputs too.

One request, end to end (what the runtime does):

1. Prompt. Each question is its own sequence, <bos> state <q> instructions <opt><mask> option … <decide>, with the state serialized as the original model does (json.dumps(…, ensure_ascii=False) for JSON) and a picture or clip entering as a prefix of embeddings in front of the text. Token ids come from tokenizer.json.
2. Media. A picture: LFM2-VL tiling (up to 10 × 512 px tiles + a thumbnail, ≤ 256 tokens a crop; antialiased bilinear resize, 16 px patches in 2×2-block order, separable position-grid weights), one vision call per crop. A clip: 16 kHz, 128-bin log-mel, audio.
3. Trunk. Embeddings from embed_tokens.f16 (media rows from the vision/audio output), padded to a bucket; a text row ≤ 2,048 positions with ≤ 16 options runs on trunk_b1_l*_k*. With media and ≥ 2 questions the prefix runs once (prefix_p*) and each question reads the cache (text_p*).
4. Readout. logits are each option's score at its <mask> marker; the answer is a softmax over the options at the question type's temperature (config.json): P(yes), a choice with its probabilities, or the expected level.
5. Fallback. Longer text or more options run on the GPU models (trunk_head, up to 12,288 positions on the iPhone with the entitlement); prompts past that are refused.

The reference for every host step is the original model's own code (its encoder, prompt, vision and audio modules in LiquidAI/d1-omni-600M): a port that reproduces its token ids, pixel values, mel features and readout exactly gets the accuracy below.

Accuracy

Checked against the original PyTorch model on a 428-case golden set (1,013 questions: Fast Decisions text, images, audio, long contexts, JSON states):

Build	Same top answer	Max |Δp| text / image / audio
Neural Engine, compiled for h18p, iPhone 17 Pro	100 % (1,011 questions ¹)	0.029 / 0.042 / 0.013
GPU fallback, compiled for h18p, iPhone 17 Pro	100 % (1,011 ¹)	0.0028 / 0.0036 / 0.0032
Neural Engine, compiled with compile.sh h15c macos from these files, M3 Max (120 cases)	100 %	0.016 / 0.042 / 0.011
gpu/*.aimodel run directly, M3 Max	100 %	0.0028 / 0.0036 / 0.0032

¹ The 16,281-token case is refused on the iPhone (GPU limit 12,288 positions).

As the ports give the original model's answers, task accuracy is the original model's: on the golden set's 420 labelled Fast Decisions questions the reference (and the port) score 79.8 %.

Speed (iPhone 17 Pro, median, warm)

	Neural Engine	GPU fallback
trunk, 128 positions	14 ms	85 ms
trunk, 2,048 positions	374 ms	1,185 ms
vision, 1 crop	74 ms	185 ms
audio, 30 s window	49 ms	109 ms
support ticket, 3 questions, end to end	97 ms	497 ms
photo, 3 questions (prefix cache)	761 ms	2,325 ms
10 s voice note, 1 question	155 ms	576 ms

Modifications from the original

Per the LFM Open License v1.0, the changes made to the Work:

• Re-authored for Core AI. The PyTorch model was rewritten for export: the trunk and decision head over padded embeddings with host-built masks; the head reading options with one-hot selections; the vision tower over a fixed 1,024-patch slot in 2×2-block order; the audio encoder over a fixed 30 s window with BatchNorm folded.
• For the Neural Engine: Linear layers as 1×1 convolutions; RMSNorm as LayerNorm([x, −x]); SiLU, sigmoid and GELU in a stable x · e/(1+e) form; attention per head with probabilities scaled ×256 before the value product; audio subsampling with time on the width axis; all in fp16.
• Precision: Neural Engine bundles fp16; GPU fallback fp16 weights with fp32 math. The embedding table is fp16.
• Packaging: portable Core AI models (.aimodel) to be compiled per device; the 19 Neural Engine trunk entrypoints share one copy of the weights.

No weights were retrained or fine-tuned; the outputs above are the measured differences from the original.

License

LFM Open License v1.0 (LICENSE), the original model's licence. Commercial use is licensed only to entities with annual revenue under US$10 million (non-profit research excepted); see the licence for the full terms. Model by Liquid AI; this conversion is not affiliated with or endorsed by Liquid AI.