FluidInference/unlimited-ocr-coreml

Unlimited-OCR (Core ML)

Baidu's Unlimited-OCR (DeepSeek-OCR architecture: SAM-B + CLIP-L page encoder, 12-layer DeepseekV2 MoE decoder, 3B parameters / ~0.5B active) converted to Core ML for Apple silicon, including its multi-page mode: up to 8 pages go in as one prompt and the whole document comes out of a single generation as markdown, <PAGE>-separated, with <|det|>label [x1, y1, x2, y2]<|/det|> layout boxes (0–999 page coordinates). Runs on the Mac GPU, offline.

Runtime: UnlimitedOcrManager in FluidUse (Swift, macOS 15+), with a SwiftUI demo (UnlimitedOcrDemo): drop a PDF, every page is read in one pass while the page thumbnails highlight the block being written.

Packages

File	Role	Size	Precision
UnlimitedOcrPageEncoder.mlpackage	DeepEncoder + projector: one 1024 × 1024 page → 273 embeddings (16 rows of 16 + row newlines + view separator)	0.8 GB	fp16
UnlimitedOcrDecoder.mlpackage	decoder with a stateful K/V cache; functions prefill_P{384,640,1152,2304} (128 tokens per call) and decode_P… (1 token), weights shared	2.7 GB	8-bit weights (per block of 32), fp16 compute
embeddings.f16	input token embeddings (host-side gather)	0.3 GB	fp16
tokenizer.json, config.json	tokenizer; shapes, buckets, prompt token ids

Upstream's R-SWA attention keeps the whole prefix (prompt + page tokens) plus a ring of the last 128 generated tokens, so the cache has a fixed size per prefix bucket: k_cache / v_cache are Core ML state, [12 × (P + 128), 1280]. P = 384 / 640 / 1152 / 2304 holds 1 / 2 / 4 / 8 pages (each page is 273 tokens). Decoder inputs: x [1, C, 1280] embeddings, cos / sin [C, 128], additive mask [1, 1, C, P + 128 + C] (cached slots, then the C new tokens), write_idx [C] cache slots, out_index [1]; outputs the top-16 logits of each vocab half. The host (Swift runtime is the reference) assembles the prefix, places generated token t in slot P + t % 128, and applies upstream's 35-gram no-repeat ban.

Quality

Reference: upstream's own code (trust_remote_code), fp32 on CPU, greedy, same pixels.

	vs upstream fp32
1 page (Baidu paper p. 4), 865 tokens	identical
3 pages in one generation (p. 3–5), 2,461 tokens	identical text; 2 layout-box coordinates differ by 1
4-bit decoder (not shipped)	diverges at token 329

The page encoder must run on the GPU: on the Neural Engine its output is wrong (row cosine down to 0.66).

Speed (M5 Pro, 24 GB, GPU, Swift)

page encoder	~100 ms / page
prefill, 8 pages (2,189 tokens) incl. encoding	3.0 s
decode	~19 ms / token at 1 page, ~23 ms at 8 pages; flat over the whole read
Baidu paper, first 8 pages in one read: 6,727 tokens	160 s (first release; ~17% less decode time per token since)
memory	1.9 GB footprint + 2.7 GB mapped decoder weights

The decoder computes all 64 experts per token (routing weights zero the unselected ones), which keeps the graph static but reads every expert's weights on each step.

Usage (Swift)

import FluidUse
import PDFKit

let manager = try await UnlimitedOcrManager.load(from: bundleDirectory)
let pdf = PDFDocument(url: pdfURL)!
let pages = try (0..<min(pdf.pageCount, manager.maxPages)).map {
    try UnlimitedOcrPagePreprocessor.page(from: pdf.page(at: $0)!)
}
var text = ""
for try await event in manager.read(pages: pages) {
    if case .text(let delta) = event { text += delta }
}
print(UnlimitedOcrDocument(parsing: text).markdown)

License and credits

MIT, same as baidu/Unlimited-OCR (LICENSE is Baidu's). All model weights are Baidu's; this repository only changes their format and precision. Conversion by Fluid Inference.