Configuration Parsing Warning:In adapter_config.json: "peft.task_type" must be a string

Handwriting LoRA for Nemotron 3.5 Parse

A LoRA adapter (26.5 M parameters) for nvidia/Nemotron-3.5-Parse that teaches the document parser to read handwriting: notebook pages, phone photos of handwritten pages and chalkboards. It also writes handwritten formulas as LaTeX, in the model's own \( ... \) Formula blocks. Training used public, modern Latin-script handwriting (English, French, German) and handwritten formulas.

  • The files at the top level apply the learned change at 0.75 strength. This is the recommended version; see why 0.75.
  • full-strength/ holds the adapter exactly as trained.
  • Licence: CC BY-NC-SA 4.0, non-commercial only, because several training sets are non-commercial. See licence and attribution.

Results

All test sets are held out from training: unseen writers, test splits and boards. There are 15 pages per set, so differences of 1 to 2 points are noise. Lower is better unless marked.

Test set Measure Base model This adapter (×0.75) Full strength
English handwritten pages (IAM test writers) CER 5.7 3.7 3.5
French handwritten pages (RIMES test) CER / accent errors 7.5 / 29.2 4.5 / 20.2 4.5 / 22.3
German handwritten pages (held-out rows) CER / accent errors 5.4 / 23.0 4.0 / 11.5 4.4 / 14.9
Real phone photos of handwritten pages (GNHK test) CER 25.7 21.2 20.1
Handwritten math pages (MathWriting test) LaTeX CER / formulas found (higher is better) 94.1 / 2.5% 41.3 / 41.5% 41.2 / 44.1%
Chalk writing on held-out chalkboards CER / formulas found (higher is better) 72.1 / 0% 41.0 / 27.3% 42.4 / 29.9%
Printed textbook pages: drift from the base model CER / worst page 0.1 / 0% 3.4 / 14% 18.4 / 100%

How the measures work:

  • CER is the character error rate after whitespace and Unicode normalisation.
  • Accent errors are edits that touch a non-ASCII letter (é, ö, ß, ...).
  • LaTeX CER is measured on Formula blocks after equivalent spellings are normalised.
  • Formulas found is the share of reference formulas that appear, within 25% edit distance, in some output block.
  • Print drift is the CER against the base model's own output on 15 printed (Hungarian) textbook pages. It measures how much the adapter changes the reading of normal print, where 0 means unchanged.

How the test pages were made:

  • Paper, math and chalkboard pages are composed from held-out real handwriting lines and formulas, the same way as in training.
  • The GNHK pages are real photos, used as they are.
  • The chalkboard and print sets come from private photos and books and are not released.
  • Private Hungarian notes (5 pages, 34 words) were also tested: word recall was 26.5% for the base model and 29.4% with the adapter, which is too small a sample to show a change.

Why 0.75 strength

The full-strength adapter improved handwriting everywhere, but on printed pages it sometimes fell into repetition loops. One textbook page came out as "Marok, Marok, Marok, ..." and lost all its text. Rather than retrain, the learned change was rescaled. A LoRA adapter adds ΔW = (α/r)·B·A to each weight it targets, so multiplying every B by s applies s·ΔW: s = 0 is the base model and s = 1 is the trained adapter. This is the same idea as interpolating between the original and the fine-tuned weights, as in WiSE-FT (Wortsman et al., Robust fine-tuning of zero-shot models, CVPR 2022).

Variant IAM CER GNHK CER Math LaTeX CER / found Chalkboard CER Print drift / worst page
Base model 5.7 25.7 94.1 / 2.5% 72.1 0.1 / 0%
×0.5 4.1 18.4 48.6 / 21.2% 44.7 1.3 / 6%
×0.65 3.8 20.5 44.9 / 36.4% 49.9 2.2 / 12%
×0.75 3.7 21.2 41.3 / 41.5% 41.0 3.4 / 14%
×1 (as trained) 3.5 20.1 41.2 / 44.1% 42.4 18.4 / 100%
Vision-encoder half only 4.0 18.6 54.0 / 44.1% 40.9 3.0 / 14%
Decoder half only 5.1 30.5 97.1 / 0% 71.6 3.7 / 15%
Vision ×1 + decoder ×0.5 3.4 37.7 39.0 / 40.7% 40.5 3.1 / 14%

What this shows:

  • At 0.75 the handwriting gains stay and the print damage mostly goes. No page loops, and the worst page is a caricature with handwriting inside the picture, which the base model also misreads.
  • The handwriting skill sits in the vision encoder. The vision half alone keeps nearly all of the text gains and finds as many formulas as the full adapter. The decoder half alone does almost nothing; its contribution is precise LaTeX (41 instead of 54 LaTeX CER).
  • The loops come from the combination. Each half alone drifts about 3% on print. Together at full strength the drift is 18%, and unequal strengths create new loops: with vision ×1 and decoder ×0.5, one GNHK photo came out as the same empty text block repeated, a 321% error on that page.

How to use

import torch
from PIL import Image
from peft import PeftModel
from transformers import AutoModel, AutoProcessor, GenerationConfig

BASE = "nvidia/Nemotron-3.5-Parse"
ADAPTER = "Kavic/nemotron-3.5-parse-handwriting-lora"

model = AutoModel.from_pretrained(BASE, trust_remote_code=True, dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, ADAPTER).merge_and_unload()   # subfolder="full-strength" for the trained version
model = model.to("cuda").eval()
processor = AutoProcessor.from_pretrained(BASE, trust_remote_code=True)
gen = GenerationConfig.from_pretrained(BASE)
gen.max_new_tokens = 4096

prompt = "</s><s><predict_bbox><predict_classes><output_markdown><predict_text_in_pic>"
image = Image.open("page.jpg").convert("RGB")
inputs = processor(images=[image], text=prompt, return_tensors="pt", add_special_tokens=False).to("cuda")
inputs["pixel_values"] = inputs["pixel_values"].to(torch.bfloat16)
with torch.inference_mode():
    out = model.generate(**inputs, generation_config=gen)
print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=False))

The output uses the base model's block format: <x_..><y_..>text<x_..><y_..><class_Text>, with formulas as \( ... \) in <class_Formula> blocks. Tested with transformers 5.22.2, peft 0.21.3 and timm 1.0.31. Merging is optional: without merge_and_unload() the adapter runs unmerged, which is slightly slower. For another strength, load full-strength/ and multiply every lora_B tensor by your factor before loading; that is the only edit behind the top-level files.

Training

  • Pages are generated on the fly from real handwriting:

    Share Page type What it contains
    30% Paper IAM, RIMES and German lines of one hand on lined, grid or plain paper, with photo effects (tilt, perspective, shading, blur, noise, JPEG)
    20% Chalkboard The same lines and MathWriting formulas as white chalk on photos of empty classroom chalkboards (writing erased)
    20% Printed anchors Pages rendered from Hungarian, German and English Wikipedia text in several fonts and layouts, with the base model's own output as the target, to keep print reading unchanged
    17% GNHK Real phone photos of handwritten pages, with their line boxes
    13% Math MathWriting formulas in the \( ... \) format, with some text lines, on paper

    The chalkboard photos are private and not released.

  • Held out from training: IAM and RIMES test splits, the last ~400 rows of the German set, the MathWriting test set, the GNHK test set and every 10th chalkboard.

  • LoRA settings:

    • Decoder: r = 64, α = 128 on q_proj, k_proj, v_proj, out_proj, fc1 and fc2 of all 10 decoder layers.
    • Vision encoder: r = 32, α = 64 on qkv, proj, fc1 and fc2 of all 27 RADIO blocks.
    • 26.5 M trainable parameters; the base model stays frozen.
  • Optimisation:

    • AdamW with learning rate 1e-4 (0.5× on the encoder), 30 warm-up steps, then cosine decay to 10%.
    • 1,500 steps of 8 pages each, which is 12,000 generated pages.
    • bf16, gradient clipping at 1.0.
  • Compute: one A100 40 GB on Google Colab for about 2 h 20 min (0.69 s per page).

Limitations

  • The benchmark is small, and most test sets are composed pages rather than real scans. Treat differences under 2 points as noise.
  • Loops can still happen. The ×0.75 version drifts 3.4% from the base model on printed pages. For exact reading of clean print, the base model alone remains the safer choice.
  • Not tuned for Hungarian handwriting. No public data exists; the private Hungarian notes test showed no measurable change.
  • The chalkboard backgrounds come from one school's boards, so results on other boards may be weaker. Whiteboards and marker writing are not covered.
  • Drawings are not labelled, so sketches on boards are not described.

Licence and attribution

This adapter is released under CC BY-NC-SA 4.0: non-commercial use, with attribution, and adaptations must use the same licence. That is the strictest of the training-data terms below. Whether the licences of training data bind trained weights is legally unsettled; this choice is the conservative one.

  • Base model: Nemotron 3.5 Parse by NVIDIA, under the OpenMDW License Agreement 1.1. Its licence text and copyright notice are included as LICENSE-OpenMDW-1.1-Nemotron-3.5-Parse.txt. Its tokenizer is MIT-licensed (see the base repository). The adapter only works together with the base model.

  • Training data:

    Dataset Copy used Terms of the original Cite
    IAM Handwriting Database via Teklia/IAM-line Non-commercial research use U.-V. Marti and H. Bunke, The IAM-database: an English sentence database for offline handwriting recognition, IJDAR 5, 39–46 (2002)
    RIMES via Teklia/RIMES-2011-line CC BY-NC 4.0 Grosicki et al., RIMES, Mitek Systems, doi:10.5281/zenodo.10812725
    German handwriting fhswf/german_handwriting AFL-3.0 FH Südwestfalen
    GNHK via radhakrishnadeshpande/gnhk-pagelevel CC BY 4.0 A. W. C. Lee, J. Chung and M. Lee, GNHK: A Dataset for English Handwriting in the Wild, ICDAR 2021, doi:10.1007/978-3-030-86337-1_27
    MathWriting via deepcopy/MathWriting-human CC BY-NC-SA 4.0, Google LLC P. Gervais, A. Fadeeva and A. Maksai, MathWriting: A Dataset For Handwritten Mathematical Expression Recognition, arXiv:2404.10690
    Wikipedia text (printed anchors only) — CC BY-SA 4.0 —
  • What is not released: the chalkboard photos, the printed textbook pages and the private notes used for testing.

Downloads last month
11
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Kavic/nemotron-3.5-parse-handwriting-lora

Adapter
(2)
this model

Datasets used to train Kavic/nemotron-3.5-parse-handwriting-lora

Paper for Kavic/nemotron-3.5-parse-handwriting-lora