Instructions to use ethzanalytics/gpt-j-6B-8bit-sharded with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ethzanalytics/gpt-j-6B-8bit-sharded with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ethzanalytics/gpt-j-6B-8bit-sharded")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ethzanalytics/gpt-j-6B-8bit-sharded")
model = AutoModelForCausalLM.from_pretrained("ethzanalytics/gpt-j-6B-8bit-sharded")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ethzanalytics/gpt-j-6B-8bit-sharded with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ethzanalytics/gpt-j-6B-8bit-sharded"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ethzanalytics/gpt-j-6B-8bit-sharded",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ethzanalytics/gpt-j-6B-8bit-sharded

SGLang

How to use ethzanalytics/gpt-j-6B-8bit-sharded with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ethzanalytics/gpt-j-6B-8bit-sharded" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ethzanalytics/gpt-j-6B-8bit-sharded",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ethzanalytics/gpt-j-6B-8bit-sharded" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ethzanalytics/gpt-j-6B-8bit-sharded",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ethzanalytics/gpt-j-6B-8bit-sharded with Docker Model Runner:
```
docker model run hf.co/ethzanalytics/gpt-j-6B-8bit-sharded
```

ethzanalytics/gpt-j-6B-8bit-sharded

This is a version of hivemind/gpt-j-6B-8bit for low-RAM loading, i.e., free Colab runtimes :)

shards are <= 1000MB each
a demo notebook of how to use it is here

Please refer to the original model card for hivemind/gpt-j-6B-8bit for all details.

Usage

NOTE: PRIOR to loading the model, you need to "patch" it to be compatible with loading 8bit weights etc. See the original model card above for details on how to do this.

install transformers, accelerate, and bitsandbytes if needed:

pip install transformers accelerate bitsandbytes

Patch the model, load using device_map="auto":

import transformers 
from transformers import AutoTokenizer

"""
CODE TO PATCH GPTJForCausalLM GOES HERE
"""

tokenizer = AutoTokenizer.from_pretrained("ethzanalytics/gpt-j-6B-8bit-sharded")

model = GPTJForCausalLM.from_pretrained(
    "ethzanalytics/gpt-j-6B-8bit-sharded",
    device_map="auto",
)

Take a look at the notebook for details.

Downloads last month: 14

Safetensors

Model size

6B params

Tensor type

F32