NameError: name 'init_empty_weights' is not defined when using load_in_8bit=True
Running the code:
model = OPTForCausalLM.from_pretrained("facebook/opt-350m", device_map='auto', load_in_8bit=True).to(device)
produces the following error:
NameError Traceback (most recent call last)
<ipython-input-7-2a869e732be4> in <module>
3 device = "cuda"
4
----> 5 model = OPTForCausalLM.from_pretrained("facebook/opt-350m", device_map='auto', load_in_8bit=True).to(device)
6 tokenizer = GPT2Tokenizer.from_pretrained("facebook/opt-350m")
/usr/local/lib/python3.8/dist-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
2271 init_contexts = [deepspeed.zero.Init(config_dict_or_path=deepspeed_config())] + init_contexts
2272 elif load_in_8bit or low_cpu_mem_usage:
-> 2273 init_contexts.append(init_empty_weights())
2274
2275 with ContextManagers(init_contexts):
NameError: name 'init_empty_weights' is not defined
I have both bitsandbytes and accelerate installed.
Hey @linkanjarad , what is your version of accelerate? cc @ybelkada
Hey @linkanjarad
Thanks for the issue!
Could you please make sure you are using the latest version of accelerate and transformers? pip install --upgrade accelerate & pip install --upgrade transformers
And also it's not recommended to call .to(device) when you load a 8bit model - you will most likely get an error. So just calling:
model = OPTForCausalLM.from_pretrained("facebook/opt-350m", device_map='auto', load_in_8bit=True)
is enough
Hey @linkanjarad
Thanks for the issue!
Could you please make sure you are using the latest version ofaccelerateandtransformers?pip install --upgrade accelerate&pip install --upgrade transformers
@ybelkada Hi, thanks for pointing out my redundancy on the usage of both device_map='auto' and .to(device), will keep that in mind. I tried using the latest version of accelerate and transformers and now it works! Thanks! Apologies for not trying the most obvious solution 😅
Great! Let us know if you face into any issue in the future