KORMo pretraining datasets - a kormo-lm Collection

kormo-lm 's Collections

KORMo SFT datasets

KORMo midtraining datasets

KORMo-10B Checkpoints

KORMo pretraining datasets

KORMo pretraining datasets

updated Oct 10

The pretraining datasets for KORMo-10B were collected from diverse, publicly available source

KORMo-Team/dclm-baseline-filtered

Preview • Updated Sep 14 • 4.2k • 1
KORMo-Team/korean-web-collection

Preview • Updated Sep 14 • 1.13k • 1

Note Stage 1 Pretraining Datasets English - komo-lm/dclm-baseline (~1000B Tokens) Korean - kormo-lm/korean_web (~42.5B Tokens)
KORMo-Team/UltraFineWeb-filtered

Preview • Updated Sep 28 • 6.51k • 1
kormo-lm/code-stack-edu

Preview • Updated Sep 26 • 2
kormo-lm/math_finemath_3plus

Viewer • Updated Sep 26 • 21.4M • 3
HuggingFaceTB/cosmopedia

Viewer • Updated Aug 12, 2024 • 31.1M • 49.8k • 649
kormo-lm/OpenMathReasoning-ko-synth

Preview • Updated Sep 26 • 2
KORMo-Team/korean-public-corpus

Preview • Updated Sep 27 • 569
KORMo-Team/FineWeb2-ko-synth

Preview • Updated Oct 13 • 815
KORMo-Team/Cosmopedia-ko-synth

Preview • Updated Oct 13 • 1.45k
KORMo-Team/UltraFineWeb-ko-synth

Preview • Updated Oct 13 • 1.04k
KORMo-Team/NemoPost-ko-synth

Preview • Updated Oct 13 • 623 • 1

Note Stage 2 Pretraining Datasets English - kormo-lm/UltraFineWeb (~793B) - kormo-lm/math_finemath_3plus (~37.3B) - kormo-lm/code_stack_edu (~152B) - kormo-lm/cosmopedia (~25B) - kormo-lm/reasoning_synth_OCR (~0.65B) - kormo-lm/reasoning_synth_OMR (~3.19B) Korean - kormo-lm/ko_web_korean_opensource (~5.57B) - kormo-lm/ko_synth_fineweb2 (~10.97B) - kormo-lm/ko_synth_kosmopedia (~4.07B) - kormo-lm/ko_synth_UltraFineWeb (~41.69B) - kormo-lm/ko_reasoning_synth_ko_mlp (~7.05B)