| | --- |
| | license: mit |
| | datasets: |
| | - HuggingFaceFW/fineweb-edu |
| | - common-pile/arxiv_papers_filtered |
| | - tiiuae/falcon-refinedweb |
| | - manu/project_gutenberg |
| | - nampdn-ai/tiny-textbooks |
| | - SciPhi/textbooks-are-all-you-need-lite |
| | - abehandlerorg/ccnews |
| | base_model: |
| | - openai-community/gpt2 |
| | pipeline_tag: text-generation |
| | --- |
| | |
| | # GPT-2 from Scratch |
| |
|
| | This model implements the GPT-2 architecture (125M parameters) trained from scratch. |
| |
|
| | ## Model Description |
| |
|
| | - **Model type:** GPT-2 (125M parameters) |
| | - **Architecture:** Transformer-based autoregressive language model following the original GPT-2 design |
| | - **Training data:** Uses multiple datasets (check tags) - 18Billion tokens. |
| | - **Language:** English |
| |
|
| |
|
| | ## Performance and Evaluation |
| |
|
| | | Dataset | Metric | thecr7guy/gpt2-pretrain | GPT-2 (baseline) | |
| | |----------------|-----------|------------|------------------| |
| | | HellaSwag | acc | **0.291** | 0.289 | |
| | | SciQ | acc | **0.754** | 0.752 | |
| | | Winogrande | acc | 0.491 | **0.516** | |
| | | TruthfulQA MC1 | acc | **0.236** | 0.228 | |
| | | MMLU (overall) | acc | **0.230** | 0.229 | |
| | | - Humanities | acc | 0.242 | 0.242 | |
| | | - Social Sci. | acc | 0.217 | 0.217 | |
| | | - STEM | acc | 0.213 | 0.213 | |
| | | - Other | acc | **0.239** | 0.238 | |
| |
|
| | ## Training Details |
| |
|
| | - **Training corpus:** Approximately 18B tokens (120GB) |
| | - **Training duration:** 1 epochs (approximately 8 hours total) |
| | - **Hardware:** 8× NVIDIA A100 PCE GPUs via runpod.io |
| | - **Estimated cost:** $ (8*13.52) for complete training |
| | - **Token context:** 1024 tokens |
| | |
| | ### Hyperparameters |
| | |
| | - context_len: 1024 |
| | - seed: 42 |
| | - epochs: 2 |
| | - batch_size: 64 |
| | - total_batch_size: 524288 tokens |
| | - grad_clip: 1.0 |
| | - optimizer: "adamw" |
| | - max_lr: 6.0e-4 |
| | - min_lr: 6.0e-5 |
| | - beta1: 0.9 |
| | - beta2: 0.95 |
| | - weight_decay: 0.1 |
| | |
| | |
| | . |
| | |
| | ## Commands used during installation |
| | |
| | - pip install wandb |
| | - pip install tiktoken |
| | - pip install --upgrade huggingface_hub |
| | - pip install torchinfo |
| | - pip install datasets |
| | - sudo apt update && sudo apt install tmux |
| | - tmux new -s training |
| | - wandb login |
| | - CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 NCCL_P2P_DISABLE=1 \ |
| | torchrun --standalone --nproc_per_node=8 train.py |
| | |
| | ## Contact |
| | |
| | GitHub: [thecr7guy2](https://github.com/thecr7guy2) |
