Instructions to use meta-llama/Llama-3.1-70B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use meta-llama/Llama-3.1-70B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="meta-llama/Llama-3.1-70B-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-70B-Instruct")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-70B-Instruct", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use meta-llama/Llama-3.1-70B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "meta-llama/Llama-3.1-70B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-llama/Llama-3.1-70B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/meta-llama/Llama-3.1-70B-Instruct

SGLang

How to use meta-llama/Llama-3.1-70B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "meta-llama/Llama-3.1-70B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-llama/Llama-3.1-70B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "meta-llama/Llama-3.1-70B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-llama/Llama-3.1-70B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use meta-llama/Llama-3.1-70B-Instruct with Docker Model Runner:
```
docker model run hf.co/meta-llama/Llama-3.1-70B-Instruct
```

what is a way to verify the model I am running is performing as expected?

#18

by MarkWard0110 - opened Jul 29, 2024

Discussion

MarkWard0110

Jul 29, 2024

I would like to know if there is a way to verify that the model I am executing performs the same as the one published by meta-llama. A way to verify the model and the software hosting the model.
For example, how would I verify that the output of a Llama 3.1-70b-instruct model pulled from Ollama's model library and executed by the Ollama host (which has various hardware profiles) would be similar to the model and environment used for benchmarks?

There are subtle things that might affect the model, such as how the host was compiled, which gguf version was selected, and the hardware features available.
I am asking because I have a situation where Llama 3.1 70B instruct q8_0 can't complete a task completed by Llama 3 70B instruct q8_0 and q6_K.

My assumption is that 3.1 is supposed to be better than Llama 3, yet I do not observe this. This is why I am looking for some way to confirm that the model is performing as designed.

I know each of the gguf versions will perform differently from the original model. However, I assume the fp16 would perform the same as the original model.

Dampfinchen

Jul 31, 2024

3.1 is a small update to L3, mainly increasing context size, coding performance and tool usage.

For the groundbreaking changes you will have to wait for LLama 4. I'm pretty sure it will have native multimodal capabilities, similar to GPT4o, and also much better text based performance.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment