---
library_name: peft
license: apache-2.0
base_model: Intelligent-Internet/II-Medical-8B
tags:
- axolotl
- base_model:adapter:Intelligent-Internet/II-Medical-8B
- lora
- transformers
datasets:
- ruslanmv/HealthCareMagic-100k
- medalpaca/medical_meadow_mediqa
- medalpaca/medical_meadow_medical_flashcards
- ruslanmv/icliniq-7k
- keivalya/MedQuad-MedicalQnADataset
- mohammad2928git/complete_medical_symptom_dataset
- gamino/wiki_medical_terms
pipeline_tag: text-generation
model-index:
- name: medical-llm-out
results: []
---
[
](https://github.com/axolotl-ai-cloud/axolotl)
See axolotl config
axolotl version: `0.16.0.dev0`
```yaml
base_model: Intelligent-Internet/II-Medical-8B
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
chat_template: tokenizer_default
load_in_8bit: false
load_in_4bit: true
strict: false
datasets:
# --- Standard Alpaca Datasets (No mapping needed) ---
- path: ruslanmv/HealthCareMagic-100k
type: alpaca
- path: medalpaca/medical_meadow_mediqa
type: alpaca
- path: medalpaca/medical_meadow_medical_flashcards
type: alpaca
# --- Custom Mapped Hugging Face Datasets ---
- path: ruslanmv/icliniq-7k
type:
system_prompt: "You are a helpful medical assistant."
field_instruction: input
field_output: answer_icliniq
format: "{instruction}"
no_input_format: "{instruction}"
- path: keivalya/MedQuad-MedicalQnADataset
type:
system_prompt: "You are a helpful medical assistant."
field_instruction: Question
field_output: Answer
format: "{instruction}"
no_input_format: "{instruction}"
- path: mohammad2928git/complete_medical_symptom_dataset
type:
system_prompt: "You are a helpful medical diagnostic assistant. Based on the patient's symptoms, identify the most likely condition."
field_instruction: text
field_output: Name
format: "{instruction}"
no_input_format: "{instruction}"
- path: gamino/wiki_medical_terms
type: completion
field: page_text
dataset_prepared_path: last_run_prepared
val_set_size: 0.05
output_dir: ./medical-llm-out
sequence_len: 4096
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true
adapter: lora
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules:
- q_proj
- v_proj
- k_proj
- o_proj
- gate_proj
- down_proj
- up_proj
# --- NVIDIA B200 Optimizations (Maximum Speed) ---
gradient_accumulation_steps: 1 # No need to accumulate, the GPU can handle it raw
micro_batch_size: 16 # Massively increased to saturate the 180GB VRAM
eval_batch_size: 8 # Faster evaluations
num_epochs: 3
optimizer: paged_adamw_32bit
lr_scheduler: cosine
learning_rate: 2e-4
train_on_inputs: false
group_by_length: false
bf16: true # Blackwell thrives on bfloat16
fp16: false
tf32: true
gradient_checkpointing: true
logging_steps: 1
flash_attention: true # Extremely fast on Blackwell
warmup_steps: 10
evals_per_epoch: 4
saves_per_epoch: 1
weight_decay: 0.0
```
# medical-llm-out
This model is a fine-tuned version of [Intelligent-Internet/II-Medical-8B](https://huggingface.co/Intelligent-Internet/II-Medical-8B) on the ruslanmv/HealthCareMagic-100k, the medalpaca/medical_meadow_mediqa, the medalpaca/medical_meadow_medical_flashcards, the ruslanmv/icliniq-7k, the keivalya/MedQuad-MedicalQnADataset, the mohammad2928git/complete_medical_symptom_dataset and the gamino/wiki_medical_terms datasets.
It achieves the following results on the evaluation set:
- Loss: 1.4660
- Ppl: 4.3319
- Memory/max Active (gib): 75.35
- Memory/max Allocated (gib): 75.35
- Memory/device Reserved (gib): 169.19
## Model description
More information needed
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- optimizer: Use OptimizerNames.PAGED_ADAMW with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- training_steps: 21090
### Training results
| Training Loss | Epoch | Step | Validation Loss | Ppl | Active (gib) | Allocated (gib) | Reserved (gib) |
|:-------------:|:------:|:-----:|:---------------:|:-------:|:------------:|:---------------:|:--------------:|
| No log | 0 | 0 | 3.0292 | 20.6798 | 75.32 | 75.32 | 82.41 |
| 1.0604 | 0.2501 | 1758 | 1.6494 | 5.2037 | 75.35 | 75.35 | 138.78 |
| 1.6010 | 0.5001 | 3516 | 1.5858 | 4.8834 | 75.35 | 75.35 | 172.28 |
| 1.5152 | 0.7502 | 5274 | 1.5469 | 4.6968 | 75.35 | 75.35 | 163.01 |
| 1.5167 | 1.0003 | 7032 | 1.5192 | 4.5687 | 75.35 | 75.35 | 170.67 |
| 1.3191 | 1.2504 | 8790 | 1.5054 | 4.5060 | 75.35 | 75.35 | 129.5 |
| 1.4320 | 1.5004 | 10548 | 1.4885 | 4.4306 | 75.35 | 75.35 | 163.71 |
| 1.5285 | 1.7505 | 12306 | 1.4749 | 4.3708 | 75.35 | 75.35 | 138.78 |
| 1.5745 | 2.0006 | 14064 | 1.4639 | 4.3228 | 75.35 | 75.35 | 163.01 |
| 1.3795 | 2.2506 | 15822 | 1.4719 | 4.3577 | 75.35 | 75.35 | 157.6 |
| 1.5165 | 2.5007 | 17580 | 1.4682 | 4.3413 | 75.35 | 75.35 | 108.64 |
| 1.0412 | 2.7508 | 19338 | 1.4660 | 4.3319 | 75.35 | 75.35 | 169.19 |
### Framework versions
- PEFT 0.18.1
- Transformers 5.3.0
- Pytorch 2.9.1+cu128
- Datasets 4.5.0
- Tokenizers 0.22.2