Error message while downloading the model
I am trying to download the model on Google Colab with the following command:
"
Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("osunlp/TableLlama")
model = AutoModelForCausalLM.from_pretrained("osunlp/TableLlama")
"
However, I get the following error message:
"ValueError: Couldn't instantiate the backend tokenizer from one of:
(1) a tokenizers library serialization file,
(2) a slow tokenizer instance to convert or
(3) an equivalent slow tokenizer class to instantiate and convert.
You need to have sentencepiece installed to convert a slow tokenizer to a fast one."
transformers version: 4.35.2
I tried "transformers version: 4.35.2" this version. You can try to install sentencepiece package using "!pip install sentencepiece" on Colab.
I can download it by the following codes:
from transformers import AutoModel, AutoTokenizer
def download_model(model_name):
# Replace "model_name" with the specific model you want to download
model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
# Save the model and tokenizer to a directory
model.save_pretrained('./model_directory/')
tokenizer.save_pretrained('./model_directory/')
if __name__ == "__main__":
model_name = "osunlp/TableLlama"
download_model(model_name)