Cant inference Via Vllm Docker container.
i cant run this via docker and tutorial for this model.
always error : ValueError: Cannot find model module. 'DotsOCRForCausalLM' is not a registered model in the Transformers library (only relevant if the model is meant to be in Transformers) and 'AutoModel' is not present in the model config's 'auto_map' (relevant if the model is custom).
i change config.json to this and dont work : "architectures": [
"DotsOCRForCausalLM"
],
"model_type": "dots_ocr",
"auto_map": {
"AutoConfig": "configuration_dots.DotsOCRConfig",
"AutoModelForCausalLM": "modeling_dots_ocr_vllm.DotsOCRForCausalLM"
},
Please help
check vllm-inference in README.md and my fix #33 and #34.
note that in modeling_dots_ocr_vllm.py, any vllm>0.9.1 will be rejected by:
def patch_vllm_chat_placeholder():
import vllm
# return when vllm version > 0.9.1
if not (vllm.__version_tuple__[0]==0 and vllm.__version_tuple__[1] <= 9 and vllm.__version_tuple__[2] <= 1):
return
from vllm.entrypoints.chat_utils import BaseMultiModalItemTracker
ori = BaseMultiModalItemTracker._placeholder_str
def _placeholder_str(self, modality, current_count: int) -> Optional[str]:
hf_config = self._model_config.hf_config
model_type = hf_config.model_type
if modality in ("image",) and model_type in ["dots_ocr"]:
return "<|img|><|imgpad|><|endofimg|>"
return ori(self, modality, current_count)
BaseMultiModalItemTracker._placeholder_str = _placeholder_str
Infact vllm==0.11.0 has formally supported dots.ocr, you can use it on any SM80 or above device.