ERROR: Should have a `model_type` key in its config.json
Device：8*NVIDIA H20-3e（141GB，Hopper）
Docker Image：lmsysorg/sglang:v0.5.13.post1
Driver: 580.95.05
'''
/usr/local/lib/python3.12/dist-packages/torchao/quantization/quant_api.py:1731: SyntaxWarning: invalid escape sequence '.'
"""Configuration class for applying different quantization configs to modules or parameters based on their fully qualified names (FQNs).
Unable to import torchao Tensor objects. This may affect loading checkpoints serialized with torchao
[2026-06-25 17:17:48] kill_process_tree called: parent_pid=1, include_parent=False, pid=1
Traceback (most recent call last):
File "/usr/local/bin/sglang", line 6, in
sys.exit(main())
^^^^^^
File "/sgl-workspace/sglang/python/sglang/cli/main.py", line 40, in main
serve(args, extra_argv)
File "/sgl-workspace/sglang/python/sglang/cli/serve.py", line 126, in serve
server_args = prepare_server_args(dispatch_argv)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/server_args.py", line 8009, in prepare_server_args
return ServerArgs.from_cli_args(raw_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/server_args.py", line 7284, in from_cli_args
return cls(**{attr: getattr(args, attr) for attr in attrs})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 399, in init
File "/sgl-workspace/sglang/python/sglang/srt/server_args.py", line 955, in post_init
self._handle_piecewise_cuda_graph()
File "/sgl-workspace/sglang/python/sglang/srt/server_args.py", line 1350, in _handle_piecewise_cuda_graph
if self.get_model_config().is_piecewise_cuda_graph_disabled_model:
^^^^^^^^^^^^^^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/server_args.py", line 7334, in get_model_config
self.model_config = ModelConfig.from_server_args(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/configs/model_config.py", line 389, in from_server_args
return ModelConfig(
^^^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/configs/model_config.py", line 201, in init
self.hf_config = get_config(
^^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/utils/common.py", line 3508, in wrapper
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/utils/hf_transformers/config.py", line 249, in get_config
config = parser.parse(
^^^^^^^^^^^^^
File "/sgl-workspace/sglang/python/sglang/srt/utils/hf_transformers/config.py", line 63, in parse
config = AutoConfig.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/modelscope/utils/hf_util/patcher.py", line 289, in from_pretrained
module_obj = module_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/configuration_auto.py", line 419, in from_pretrained
raise ValueError(
ValueError: Unrecognized model in /root/.cache/modelscope/hub/models/PhalaCloud/GLM-5.2-W4AFP8. Should have a model_type key in its config.json.
'''
The repo’s config.json does contain model_type (deepseek_v3). I've verified it on both HF and the ModelScope mirror. If your local cache has a stale or partially-downloaded copy (likely from an interrupted first pull). Please delete ~/.cache/modelscope/hub/models/PhalaCloud/GLM-5.2-W4AFP8 and re-download.
Re-pulling will also pick up the newly added generation_config.json (temperature 1.0 / top_p 0.95 defaults). This is recommended, as serving without top-p truncation can cause occasional repetition loops in thinking mode.