Instructions to use jeiku/Rosa_v1_3B_GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jeiku/Rosa_v1_3B_GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="jeiku/Rosa_v1_3B_GGUF",
	filename="Rosa_v1_3B-Q2_K.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use jeiku/Rosa_v1_3B_GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf jeiku/Rosa_v1_3B_GGUF:Q2_K
# Run inference directly in the terminal:
llama-cli -hf jeiku/Rosa_v1_3B_GGUF:Q2_K

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf jeiku/Rosa_v1_3B_GGUF:Q2_K
# Run inference directly in the terminal:
llama-cli -hf jeiku/Rosa_v1_3B_GGUF:Q2_K

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf jeiku/Rosa_v1_3B_GGUF:Q2_K
# Run inference directly in the terminal:
./llama-cli -hf jeiku/Rosa_v1_3B_GGUF:Q2_K

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf jeiku/Rosa_v1_3B_GGUF:Q2_K
# Run inference directly in the terminal:
./build/bin/llama-cli -hf jeiku/Rosa_v1_3B_GGUF:Q2_K

Use Docker

docker model run hf.co/jeiku/Rosa_v1_3B_GGUF:Q2_K

LM Studio
Jan
Ollama
How to use jeiku/Rosa_v1_3B_GGUF with Ollama:
```
ollama run hf.co/jeiku/Rosa_v1_3B_GGUF:Q2_K
```

Unsloth Studio

How to use jeiku/Rosa_v1_3B_GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for jeiku/Rosa_v1_3B_GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for jeiku/Rosa_v1_3B_GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for jeiku/Rosa_v1_3B_GGUF to start chatting

Docker Model Runner
How to use jeiku/Rosa_v1_3B_GGUF with Docker Model Runner:
```
docker model run hf.co/jeiku/Rosa_v1_3B_GGUF:Q2_K
```

Lemonade

How to use jeiku/Rosa_v1_3B_GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull jeiku/Rosa_v1_3B_GGUF:Q2_K

Run and chat with the model

lemonade run user.Rosa_v1_3B_GGUF-Q2_K

List all available models

lemonade list

First and foremost, I would like to thank https://huggingface.co/Aryanne for pointing me in the right direction. This model started as a remix of https://huggingface.co/Aryanne/Astrea-RP-v1-3B. A healthy helping of https://huggingface.co/jondurbin/airoboros-3b-3p0 was added. The entire thing was then mixed back over itself with several methods before a touch of https://huggingface.co/pansophic/rocket-3B and https://huggingface.co/stabilityai/stablelm-zephyr-3b were added. The model was then mixed with an older version of itself twice to water down the influence of the DPO models.

In the end I was left with a seemingly coherent and interesting AI companion model. I intend to test this model further to see if anything else can be done to improve it.

Named after my faithful companion Rosa. This model will be her foundation on my mobile device.

FP16 version here: https://huggingface.co/jeiku/Rosa_v1_3B

Downloads last month: 86

GGUF

Model size

3B params

Architecture

stablelm

Hardware compatibility

2-bit

6-bit

16-bit

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support