Instructions to use Vortex5/Violet-Starlight-12B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Vortex5/Violet-Starlight-12B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Vortex5/Violet-Starlight-12B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Vortex5/Violet-Starlight-12B")
model = AutoModelForCausalLM.from_pretrained("Vortex5/Violet-Starlight-12B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Vortex5/Violet-Starlight-12B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Vortex5/Violet-Starlight-12B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Vortex5/Violet-Starlight-12B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Vortex5/Violet-Starlight-12B

SGLang

How to use Vortex5/Violet-Starlight-12B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Vortex5/Violet-Starlight-12B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Vortex5/Violet-Starlight-12B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Vortex5/Violet-Starlight-12B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Vortex5/Violet-Starlight-12B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Vortex5/Violet-Starlight-12B with Docker Model Runner:
```
docker model run hf.co/Vortex5/Violet-Starlight-12B
```

Additional info on intent or use case of merges

by jebcarter - opened Jan 26

Discussion

jebcarter

Jan 26

Good morning!

I love the flood of Nemo merges coming out here - still near the top of the charts for local writing and often better (in my eyes) than the big API models - but I'd love some more information about what the intended flavor or use of some of these merges are?

At least a "hey I liked how model X did Thing Y so I merged with model Alpha whose Thing Z I dig" or sommat. :D

You are cooking strong contenders for unseating Muse with the combination of high grade prose and good instruct following but ye gads its a lot of these. :)

Vortex5

Owner Jan 26

Good morning!

I love the flood of Nemo merges coming out here - still near the top of the charts for local writing and often better (in my eyes) than the big API models - but I'd love some more information about what the intended flavor or use of some of these merges are?

At least a "hey I liked how model X did Thing Y so I merged with model Alpha whose Thing Z I dig" or sommat. :D

You are cooking strong contenders for unseating Muse with the combination of high grade prose and good instruct following but ye gads its a lot of these. :)

Hey thanks, currently intended use section is to just fill to footer space in card. I don't want to claim anything like X model is better at Y in descriptions etc. Because everyone uses different settings and prompts this can be subjective.

If there was a more neutral way to describe models I may be into that, that disregards any personal bias.

I do sometimes test on same seed with a neutral prompt and a LLM tells me the kind of prose. I could put this somewhere on card to describe general model "flavor", I would be open to any ideas you have.

jebcarter

Jan 26

I completely understand where you are coming from - I still have flashbacks to Psyonic-Cetacean and Psyfighter's releases.

Not neutral, but as a starter what I'm usually looking at is:

What's in the training data? Adventure, story, chat?
Long or short outputs by default.
Dry or purple outputs by default.
Spatial tracking (The McGuffin is a Thing inside a drawer. The drawer is locked. I get the McGuffin -> does the model understand the drawer needs to be unlocked and opened? Just opened? Does it write about smashing the drawer and just taking it)
I know lots of people care about turn taking but that is so down to prompts/harness/text vs chat completions that I don't think there's any reliable way to go after it.

In a more narrow scope, what are you as a model cooker GENERALLY trying to work for? Chat? Story? Mix?

Vortex5

Owner Jan 26

I try to create unique merges with good prose for roleplay/story that are also smart.

I look for the same things usually:

Does it output too much probably broken in some way
Does it output too little and not match example also bad
Detail retrieval
Instruction following
Prose and pacing

As for the card, I am not sure what to say.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment