Instructions to use mixtao/MixTAO-7Bx2-MoE-Instruct-v7.0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mixtao/MixTAO-7Bx2-MoE-Instruct-v7.0 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mixtao/MixTAO-7Bx2-MoE-Instruct-v7.0")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mixtao/MixTAO-7Bx2-MoE-Instruct-v7.0")
model = AutoModelForCausalLM.from_pretrained("mixtao/MixTAO-7Bx2-MoE-Instruct-v7.0", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mixtao/MixTAO-7Bx2-MoE-Instruct-v7.0 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mixtao/MixTAO-7Bx2-MoE-Instruct-v7.0"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mixtao/MixTAO-7Bx2-MoE-Instruct-v7.0",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/mixtao/MixTAO-7Bx2-MoE-Instruct-v7.0

SGLang

How to use mixtao/MixTAO-7Bx2-MoE-Instruct-v7.0 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mixtao/MixTAO-7Bx2-MoE-Instruct-v7.0" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mixtao/MixTAO-7Bx2-MoE-Instruct-v7.0",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mixtao/MixTAO-7Bx2-MoE-Instruct-v7.0" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mixtao/MixTAO-7Bx2-MoE-Instruct-v7.0",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use mixtao/MixTAO-7Bx2-MoE-Instruct-v7.0 with Docker Model Runner:
```
docker model run hf.co/mixtao/MixTAO-7Bx2-MoE-Instruct-v7.0
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

MixTAO-7Bx2-MoE-Instruct

MixTAO-7Bx2-MoE-Instruct is a Mixure of Experts (MoE).

💻 Usage

text-generation-webui - Model Tab

Chat template

{%- for message in messages %}
    {%- if message['role'] == 'system' -%}
        {{- message['content'] + '\n\n' -}}
    {%- else -%}
        {%- if message['role'] == 'user' -%}
            {{- name1 + ': ' + message['content'] + '\n'-}}
        {%- else -%}
            {{- name2 + ': ' + message['content'] + '\n' -}}
        {%- endif -%}
    {%- endif -%}
{%- endfor -%}

Instruction template ：Alpaca

Change this according to the model/LoRA that you are using. Used in instruct and chat-instruct modes.

{%- set ns = namespace(found=false) -%}
{%- for message in messages -%}
    {%- if message['role'] == 'system' -%}
        {%- set ns.found = true -%}
    {%- endif -%}
{%- endfor -%}
{%- if not ns.found -%}
    {{- '' + 'Below is an instruction that describes a task. Write a response that appropriately completes the request.' + '\n\n' -}}
{%- endif %}
{%- for message in messages %}
    {%- if message['role'] == 'system' -%}
        {{- '' + message['content'] + '\n\n' -}}
    {%- else -%}
        {%- if message['role'] == 'user' -%}
            {{-'### Instruction:\n' + message['content'] + '\n\n'-}}
        {%- else -%}
            {{-'### Response:\n' + message['content'] + '\n\n' -}}
        {%- endif -%}
    {%- endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
    {{-'### Response:\n'-}}
{%- endif -%}

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	76.55
AI2 Reasoning Challenge (25-Shot)	74.23
HellaSwag (10-Shot)	89.37
MMLU (5-Shot)	64.54
TruthfulQA (0-shot)	74.26
Winogrande (5-shot)	87.77
GSM8k (5-shot)	69.14

Downloads last month: 18

Safetensors

Model size

13B params

Tensor type

BF16

Model tree for mixtao/MixTAO-7Bx2-MoE-Instruct-v7.0

Quantizations

3 models

Spaces using mixtao/MixTAO-7Bx2-MoE-Instruct-v7.0 2

Evaluation results

normalized accuracy on AI2 Reasoning Challenge (25-Shot)
test set Open LLM Leaderboard

74.230
normalized accuracy on HellaSwag (10-Shot)
validation set Open LLM Leaderboard

89.370
accuracy on MMLU (5-Shot)
test set Open LLM Leaderboard

64.540
mc2 on TruthfulQA (0-shot)
validation set Open LLM Leaderboard

74.260
accuracy on Winogrande (5-shot)
validation set Open LLM Leaderboard

87.770
accuracy on GSM8k (5-shot)
test set Open LLM Leaderboard

69.140