Instructions to use ree2raz/CyberSecQwen-4B-AWQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ree2raz/CyberSecQwen-4B-AWQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ree2raz/CyberSecQwen-4B-AWQ")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ree2raz/CyberSecQwen-4B-AWQ")
model = AutoModelForCausalLM.from_pretrained("ree2raz/CyberSecQwen-4B-AWQ")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use ree2raz/CyberSecQwen-4B-AWQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ree2raz/CyberSecQwen-4B-AWQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ree2raz/CyberSecQwen-4B-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ree2raz/CyberSecQwen-4B-AWQ

SGLang

How to use ree2raz/CyberSecQwen-4B-AWQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ree2raz/CyberSecQwen-4B-AWQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ree2raz/CyberSecQwen-4B-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ree2raz/CyberSecQwen-4B-AWQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ree2raz/CyberSecQwen-4B-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ree2raz/CyberSecQwen-4B-AWQ with Docker Model Runner:
```
docker model run hf.co/ree2raz/CyberSecQwen-4B-AWQ
```

CyberSecQwen-4B-AWQ

4-bit AWQ quantized version of CyberSecQwen-4B.

Quantization

Parameter	Value
Method	AWQ (group_size=128, zero_point=True)
Weight precision	4-bit
Compute dtype	float16
Calibration samples	320 CTI-Bench prompts (256 RCM + 64 MCQ, chat-template formatted)
Quantization tool	autoawq
Calibration hardware	Modal A100

CTI-Bench Evaluation

Evaluated under the Foundation-Sec-8B protocol:

Temperature 0.3, max_tokens 512, concurrency 32
5 independent trials, zero-shot (no system prompt)
vLLM v0.20.1 with awq_marlin kernel on Modal L4 GPU

Task	AWQ 4-bit	GGUF Q4_K_M	FP16 Reference
CTI-MCQ (2,500 items)	0.5921 ± 0.0083	0.5368 ± 0.0048	0.5868 ± 0.0029
CTI-RCM (1,000 items)	0.5814 ± 0.0025	0.6254 ± 0.0063	0.6664 ± 0.0023

Key findings:

CTI-MCQ: AWQ 4-bit matches or slightly exceeds FP16 performance (+0.5 points). Better than GGUF Q4_K_M.
CTI-RCM: AWQ 4-bit degrades by 8.5 percentage points vs FP16. GGUF Q4_K_M does better on this task (-4.1 pts).
AWQ is best for MCQ (general language), GGUF is best for RCM (task-specific classification).

Trial results

CTI-MCQ

Trial	Seed	Accuracy
1	42	0.6016
2	43	0.5984
3	44	0.5936
4	45	0.5780
5	46	0.5888

CTI-MCQ

Trial	Seed	Accuracy
1	42	0.6016
2	43	0.5984
3	44	0.5936
4	45	0.5780
5	46	0.5888

CTI-RCM

Trial	Seed	Accuracy
1	42	0.5790
2	43	0.5830
3	44	0.5790
4	45	0.5840
5	46	0.5820

Quantization variants

Variant	CTI-MCQ	CTI-RCM	Size	Engine
AWQ 4-bit	0.5921	0.5814	2.7 GB	vLLM
GGUF Q4_K_M	0.5368	0.6254	2.5 GB	llama.cpp

Choose AWQ for MCQ/general chat, GGUF for vulnerability classification.

Usage with vLLM

vllm serve ree2raz/CyberSecQwen-4B-AWQ --quantization awq_marlin --dtype float16

Model Size

Format	Size
Original FP16	~8 GB
AWQ 4-bit	~2.7 GB

Citation

@misc{{cybersecqwen2026,
  title  = {{CyberSecQwen-4B: A Compact CTI Specialist Fine-Tuned from Qwen3-4B-Instruct-2507 on AMD MI300X}},
  author = {{Mulia, Samuel}},
  year   = {{2026}},
  publisher = {{Hugging Face}},
  url    = {{https://huggingface.co/athena129/CyberSecQwen-4B}}
}}

Evaluation Infrastructure

GitHub repository — Modal scripts for quantization + evaluation.

Downloads last month: 226

Safetensors

Model size

4B params

Tensor type

I32

F16

Model tree for ree2raz/CyberSecQwen-4B-AWQ

Base model

Qwen/Qwen3-4B-Instruct-2507

Adapter

lablab-ai-amd-developer-hackathon/CyberSecQwen-4B

Quantized

(2)

this model

Paper for ree2raz/CyberSecQwen-4B-AWQ

Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Report

Paper • 2504.21039 • Published Apr 28, 2025 • 17