Llama-ML-Expert-DPO-1b

The final, preference-aligned model in the LLM-ArXiv-Domain-Expert pipeline. Built on top of danivpv/Llama-ML-Expert-Instruct-1b via Direct Preference Optimization (DPO), aligning outputs toward the authoritative, academic writing style of the source ArXiv ML papers.

Model Details


Base model	`danivpv/Llama-ML-Expert-Instruct-1b` (SFT), itself built on `unsloth/Llama-3.2-1B-bnb-4bit`
Fine-tuning method	DPO via `unsloth`'s `PatchDPOTrainer`, new LoRA adapters — rank 32, alpha 32
LoRA target modules	`q_proj, k_proj, v_proj, up_proj, down_proj, o_proj, gate_proj`
Learning rate	2e-6
Epochs	1
DPO beta	0.5
Optimizer	`adamw_8bit`
Prompt format	Alpaca (`### Instruction: ... ### Response: ...`)

Training Data

danivpv/ml-arxiv-dpo — instruction/chosen/rejected triples, where chosen is a verbatim excerpt from the source ArXiv paper and rejected is a factually-grounded but LLM-paraphrased answer. The model is optimized to prefer authentic academic phrasing over generic LLM-speak — see the dataset card for the full rationale and its tradeoffs.

How to Use

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="danivpv/Llama-ML-Expert-DPO-1b",
    max_seq_length=2048,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

prompt = """### Instruction:
{your ML question here}

### Response:
"""
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Relationship to Instruct-1b

This model is a second-stage fine-tune, not a replacement — it trades a small amount of generative flexibility for closer adherence to source-paper phrasing. If you need more open-ended generation, Llama-ML-Expert-Instruct-1b (SFT-only) may be the better starting point.

Limitations

Same 1B-scale limitations as the base Instruct model: strong for domain-specialized short-form Q&A, not a general-purpose reasoning model.
DPO training used 1 epoch at a small learning rate — a conservative alignment pass, not an exhaustive preference-optimization run.