KORMo pretraining datasets

KORMo-Team 's Collections

KORMo-10B

KORMo midtraining datasets

KORMo SFT datasets

updated Oct 13

The pretraining datasets for KORMo-10B were collected from diverse, publicly available source.

Upvote

KORMo-Team/dclm-baseline-filtered

Preview • Updated Sep 14 • 4.2k • 1
KORMo-Team/korean-web-collection

Preview • Updated Sep 14 • 1.13k • 1

Note Stage 1 Pretraining Datasets English - KORMo-Team/dclm-baseline-filtered (~1000B Tokens) Korean - KORMo-Team/korean-web-collection (~42.5B Tokens)
KORMo-Team/UltraFineWeb-filtered

Preview • Updated Sep 28 • 6.51k • 1
HuggingFaceTB/stack-edu

Viewer • Updated Mar 20 • 167M • 2.17k • 60
HuggingFaceTB/smollm-corpus

Viewer • Updated Sep 6, 2024 • 237M • 17.1k • 405
HuggingFaceTB/finemath

Viewer • Updated Feb 6 • 48.3M • 11.6k • 344
HuggingFaceTB/cosmopedia

Viewer • Updated Aug 12, 2024 • 31.1M • 49.8k • 649
nvidia/OpenCodeReasoning

Viewer • Updated May 4 • 753k • 4.1k • 519
nvidia/OpenMathReasoning

Viewer • Updated May 27 • 5.68M • 17.2k • 390

Note Stage 2 Pretraining Datasets English - KORMo-Team/UltraFineWeb-english-filtered (~793B Tokens) - HuggingFaceTB/finemath (~37.3B Tokens, sampling) - HuggingFaceTB/stack-edu & HuggingFaceTB/smollm-corpus(python-edu) (~152B Tokens, sampling) - HuggingFaceTB/cosmopedia (~25B Tokens) - nvidia/OpenCodeReasoning (~0.65B Tokens, sampling) - nvidia/OpenMathReasoning (~3.19B Tokens, sampling)
KORMo-Team/korean-public-corpus

Preview • Updated Sep 27 • 569
KORMo-Team/FineWeb2-ko-synth

Preview • Updated Oct 13 • 815
KORMo-Team/Cosmopedia-ko-synth

Preview • Updated Oct 13 • 1.45k
KORMo-Team/UltraFineWeb-ko-synth

Preview • Updated Oct 13 • 1.04k
KORMo-Team/NemoPost-ko-synth

Preview • Updated Oct 13 • 623 • 1

Note Stage 2 Pretraining Datasets Korean - KORMo-Team/korean-public-corpus (~5.57B Tokens) - KORMo-Team/FineWeb2-ko-synth (~10.97B Tokens) - KORMo-Team/Cosmopedia-ko-synth (~4.07B Tokens) - KORMo-Team/UltraFineWeb-ko-synth (~41.69B Tokens) - KORMo-Team/NemoPost-ko-synth (~7.05B Tokens)

Upvote