aloobun
/

Reyna-CoT-4B-v0.1

Text Generation

text-generation-inference

Model card Files Files and versions

aloobun commited on Feb 23, 2024

Commit

d186533

·

verified ·

1 Parent(s): 0066ec8

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -17,7 +17,7 @@ datasets:
 - Finetuned [Qwen/Qwen1.5-4B](https://huggingface.co/Qwen/Qwen1.5-4B), on variety of CoT tasks including Reasoning, Closed Book Question Answering, Ethics, and more.
 - Datasets : Curated from - [kaist-ai/CoT-Collection](https://huggingface.co/datasets/kaist-ai/CoT-Collection), [euclaise/TinyCoT](https://huggingface.co/datasets/euclaise/TinyCoT) and a very small subset from [teknium/OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5).
 - This marks the fourth model in this series. This experiment aims to improve Chain of Thought (CoT) capabilities on smaller language models.
-- In the next run, I may rerun the finetuning experiment using an iterative rationale-bootstrapping procedure inspired by euclaise/Memphis-CoT-3B.
 - Hyperparameter: adamw with eps of 1e-8, cosine decay with 20% warmup, lr=2e-5
 ## Benchamrks:

 - Finetuned [Qwen/Qwen1.5-4B](https://huggingface.co/Qwen/Qwen1.5-4B), on variety of CoT tasks including Reasoning, Closed Book Question Answering, Ethics, and more.
 - Datasets : Curated from - [kaist-ai/CoT-Collection](https://huggingface.co/datasets/kaist-ai/CoT-Collection), [euclaise/TinyCoT](https://huggingface.co/datasets/euclaise/TinyCoT) and a very small subset from [teknium/OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5).
 - This marks the fourth model in this series. This experiment aims to improve Chain of Thought (CoT) capabilities on smaller language models.
+- I may rerun the finetuning experiment(with a more balanced dataset), using an iterative rationale-bootstrapping procedure inspired by euclaise/Memphis-CoT-3B.
 - Hyperparameter: adamw with eps of 1e-8, cosine decay with 20% warmup, lr=2e-5
 ## Benchamrks: