Instructions to use LoftQ/CodeLlama-7b-hf-4bit-64rank with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LoftQ/CodeLlama-7b-hf-4bit-64rank with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="LoftQ/CodeLlama-7b-hf-4bit-64rank")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("LoftQ/CodeLlama-7b-hf-4bit-64rank")
model = AutoModelForCausalLM.from_pretrained("LoftQ/CodeLlama-7b-hf-4bit-64rank", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use LoftQ/CodeLlama-7b-hf-4bit-64rank with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LoftQ/CodeLlama-7b-hf-4bit-64rank"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LoftQ/CodeLlama-7b-hf-4bit-64rank",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/LoftQ/CodeLlama-7b-hf-4bit-64rank

SGLang

How to use LoftQ/CodeLlama-7b-hf-4bit-64rank with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "LoftQ/CodeLlama-7b-hf-4bit-64rank" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LoftQ/CodeLlama-7b-hf-4bit-64rank",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "LoftQ/CodeLlama-7b-hf-4bit-64rank" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LoftQ/CodeLlama-7b-hf-4bit-64rank",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use LoftQ/CodeLlama-7b-hf-4bit-64rank with Docker Model Runner:
```
docker model run hf.co/LoftQ/CodeLlama-7b-hf-4bit-64rank
```

CodeLlama-7b-hf-4bit-64rank / README.md

LoftQ

Create README.md

8da3fb9 verified over 2 years ago

preview code

Raw History Blame Contribute Delete

3.03 kB

	---
	license: mit
	language:
	- en
	pipeline_tag: text-generation
	tags:
	- quantization
	- lora
	- loftq
	- llama
	---
	# LoftQ Initialization

	\| [Paper](https://arxiv.org/abs/2310.08659) \| [Code](https://github.com/yxli2123/LoftQ) \| [PEFT Example](https://github.com/huggingface/peft/tree/main/examples/loftq_finetuning) \|

	LoftQ (LoRA-fine-tuning-aware Quantization) provides a quantized backbone Q and LoRA adapters A and B, given a full-precision pre-trained weight W.

	This model, `CodeLlama-7b-hf-4bit-64rank`, is obtained from [CodeLLAMA-7b](https://huggingface.co/meta-llama/CodeLlama-7b-hf).
	The backbone is under `LoftQ/CodeLlama-7b-hf-4bit-64rank` and LoRA adapters are under the `subfolder='loftq_init'`.

	## Model Info
	### Backbone
	- Stored format: `torch.bfloat16`
	- Size: ~ 14 GiB
	- Loaded format: bitsandbytes nf4
	- Size loaded on GPU: ~3.5 GiB

	### LoRA adapters
	- rank: 64
	- lora_alpha: 16
	- target_modules: ["down_proj", "up_proj", "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj"]

	## Usage

	Training. Here's an example of loading this model and preparing for the LoRA fine-tuning.

	```python
	import torch
	from transformers import AutoModelForCausalLM, BitsAndBytesConfig
	from peft import PeftModel

	MODEL_ID = "LoftQ/CodeLlama-7b-hf-4bit-64rank"

	base_model = AutoModelForCausalLM.from_pretrained(
	MODEL_ID,
	torch_dtype=torch.bfloat16, # you may change it with different models
	quantization_config=BitsAndBytesConfig(
	load_in_4bit=True,
	bnb_4bit_compute_dtype=torch.bfloat16, # bfloat16 is recommended
	bnb_4bit_use_double_quant=False,
	bnb_4bit_quant_type='nf4',
	),
	)
	peft_model = PeftModel.from_pretrained(
	base_model,
	MODEL_ID,
	subfolder="loftq_init",
	is_trainable=True,
	)

	# Do training with peft_model ...
	```

	Inference. Here is an example code for inference after the model has been fine-tuned on [GSM8K](https://huggingface.co/datasets/gsm8k).

	```python
	import torch
	from transformers import AutoModelForCausalLM, BitsAndBytesConfig
	from peft import PeftModel

	MODEL_ID = "LoftQ/CodeLlama-7b-hf-4bit-64rank"

	base_model = AutoModelForCausalLM.from_pretrained(
	MODEL_ID,
	torch_dtype=torch.bfloat16, # you may change it with different models
	quantization_config=BitsAndBytesConfig(
	load_in_4bit=True,
	bnb_4bit_compute_dtype=torch.bfloat16, # bfloat16 is recommended
	bnb_4bit_use_double_quant=False,
	bnb_4bit_quant_type='nf4',
	),
	)
	peft_model = PeftModel.from_pretrained(
	base_model,
	MODEL_ID,
	subfolder="gsm8k",
	is_trainable=True,
	)

	# Do inference with peft_model ...
	```

	See the full code at our [Github Repo]((https://github.com/yxli2123/LoftQ))


	## Citation

	```bibtex
	@article{li2023loftq,
	title={Loftq: Lora-fine-tuning-aware quantization for large language models},
	author={Li, Yixiao and Yu, Yifan and Liang, Chen and He, Pengcheng and Karampatziakis, Nikos and Chen, Weizhu and Zhao, Tuo},
	journal={arXiv preprint arXiv:2310.08659},
	year={2023}
	}
	```