File size: 2,090 Bytes
8b189f0 dc3d26e 8b189f0 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
---
license: apache-2.0
tags:
- qwen3
- finetuned
- qlora
- text-generation
- rag
- embedding
- reranker
- base-model
- thinking-model
library_name: transformers
---
# 🤖 Qwen3 Finetuned Model Series (QLoRA)
This repository contains multiple variants of **Qwen3**-based models fine-tuned via **QLoRA**, including **base generative models**, **Thinking models**, and **RAG companion models** (Embedding + Reranker). All models are developed to support iGEM teams and synthetic biology research groups with functionalities such as experimental protocol assistance, iGEM rule explanations, and competition strategy guidance. They are suitable for dialogue, reasoning, and Retrieval-Augmented Generation (RAG) scenarios.
> ✅ **Overall Evaluation Conclusion**: After balancing multiple dimensions (performance, reasoning quality, resource consumption), the **4B-parameter base model demonstrates the best overall performance** and is recommended as the default choice.
---
## 📦 Model Overview
| Model Type | Parameters | Description |
|------------------ |-------- |--------- |
| **Base Models** | 0.6B,1.7B,4B,8B,14B | Standard text generation models for general dialogue and instruction following |
| **Thinking Model** | 14B | Enables "Chain-of-Thought" capability, suitable for complex reasoning tasks |
| **Embedding Model** | 0.6B | Used for vector retrieval in RAG (sentence embedding) |
| **Reranker Model** | 0.6B | Used for re-ranking in RAG (cross-encoder style reranking) |
---
All models are fine-tuned from the original **Qwen3** base weights.
---
## ⚙️ Finetuning Configuration (QLoRA)
- **Quantization**: 4-bit (NF4)
- **Training Epochs**: 4
- **Per-device Batch Size**: 2
- **Gradient Accumulation Steps**: 8 (effective batch size = 16)
- **Learning Rate Warmup Steps**: 4
- **LoRA Configuration**:
- `rank (r)`: 8
- `alpha`: 256
- `target_modules`: `["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]`
- **training framework**：`transformers` + `peft` + `bitsandbytes` |