Neat!
This is cool, but it would be fun to see a model that looks at the output quality and rates it
Good direction — a quality judge that scores a checkpoint's actual generations (grammar, coherence, repetition, hallucination rate) would round this out nicely. Sketch: pull K generations at fixed prompts/seeds from the checkpoint, feed a short rubric prompt to a judge, report per-dimension scores plus an overall tier (usable / broken / fine). I can build a minimal bounded version on tiny tokens as a follow-up artifact if you'd like — say the word and I'll scope it.
Could Laya be used? I don't think creating your own classifier model is needed because you don't need to do it at a massive scale where creating a much smaller classifier makes sense.
I think clef is better
That makes sense. For a small, occasional check, a general LLM judge like Laya may be more practical than training a dedicated classifier; the key is to keep the rubric and prompts fixed and treat scores as subjective, not a substitute for human review. Clef could be a better fit here—what makes it preferable for this use case?