Instructions to use Inferact/Qwen3.8-27B-MXFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Inferact/Qwen3.8-27B-MXFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Inferact/Qwen3.8-27B-MXFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Inferact/Qwen3.8-27B-MXFP4")
model = AutoModelForMultimodalLM.from_pretrained("Inferact/Qwen3.8-27B-MXFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Inferact/Qwen3.8-27B-MXFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Inferact/Qwen3.8-27B-MXFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Inferact/Qwen3.8-27B-MXFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Inferact/Qwen3.8-27B-MXFP4

SGLang

How to use Inferact/Qwen3.8-27B-MXFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Inferact/Qwen3.8-27B-MXFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Inferact/Qwen3.8-27B-MXFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Inferact/Qwen3.8-27B-MXFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Inferact/Qwen3.8-27B-MXFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Inferact/Qwen3.8-27B-MXFP4 with Docker Model Runner:
```
docker model run hf.co/Inferact/Qwen3.8-27B-MXFP4
```

vllm load failed because of dimension mismatch

by hnhyzz - opened 23 days ago

Discussion

hnhyzz

23 days ago

(APIServer pid=1) INFO 08-17 11:38:58 [api_utils.py:345]
(APIServer pid=1) INFO 08-17 11:38:58 [api_utils.py:345] █ █ █▄ ▄█
(APIServer pid=1) INFO 08-17 11:38:58 [api_utils.py:345] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.1.dev1+g27a9509ca
(APIServer pid=1) INFO 08-17 11:38:58 [api_utils.py:345] █▄█▀ █ █ █ █ model /models/Qwen-fp4/
(APIServer pid=1) INFO 08-17 11:38:58 [api_utils.py:345] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=1) INFO 08-17 11:38:58 [api_utils.py:345]
(APIServer pid=1) INFO 08-17 11:38:58 [api_utils.py:273] non-default args: {'model_tag': '/models/Qwen-fp4/', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'api_key': ['xxxxxx'], 'model': '/models/Qwen-fp4/', 'dtype': 'bfloat16', 'max_model_len': 32000, 'served_model_name': ['Qwen'], 'attention_backend': 'TRITON_ATTN', 'reasoning_parser': 'qwen3', 'pipeline_parallel_size': 2, 'gpu_memory_utilization': 0.96, 'enable_prefix_caching': True, 'language_model_only': True, 'limit_mm_per_prompt': {'image': 0}, 'mm_encoder_attn_backend': 'TORCH_SDPA', 'skip_mm_profiling': True, 'max_num_batched_tokens': 4096, 'max_num_seqs': 4, 'compilation_config': {'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': [], 'ir_enable_torch_wrap': None, 'splitting_ops': None, 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': None, 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': None, 'pass_config': {}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': None, 'static_all_moe_layers': []}}
(APIServer pid=1) INFO 08-17 11:38:58 [model.py:679] Resolved architecture: Qwen3_5ForConditionalGeneration
(APIServer pid=1) INFO 08-17 11:38:58 [model.py:1972] Using max model len 32000
(APIServer pid=1) [ERROR] min_frames is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
(APIServer pid=1) [ERROR] max_frames is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
(APIServer pid=1) INFO 08-17 11:39:00 [registry.py:141] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.
(APIServer pid=1) INFO 08-17 11:39:00 [model.py:882] Disabled mm_prefix attention mode because multimodal inputs are configuration-disabled. Attention backends without mm_prefix support may now be selected.
(APIServer pid=1) INFO 08-17 11:39:01 [scheduler.py:242] Chunked prefill is enabled with max_num_batched_tokens=4096.
(APIServer pid=1) INFO 08-17 11:39:01 [config.py:598] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled
(APIServer pid=1) INFO 08-17 11:39:01 [kernel.py:306] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=1) WARNING 08-17 11:39:05 [rocm.py:134] Using CUDA_VISIBLE_DEVICES on ROCm is deprecated and support will be removed in vLLM v0.26.0. Please use HIP_VISIBLE_DEVICES instead.
WARNING 08-17 11:39:07 [rocm.py:134] Using CUDA_VISIBLE_DEVICES on ROCm is deprecated and support will be removed in vLLM v0.26.0. Please use HIP_VISIBLE_DEVICES instead.
(EngineCore pid=223) INFO 08-17 11:39:15 [core.py:121] Initializing a V1 LLM engine (v0.1.dev1+g27a9509ca) with config: model='/models/Qwen-fp4/', speculative_config=None, tokenizer='/models/Qwen-fp4/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32000, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=2, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=mxfp4, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=Qwen, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 8, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
(EngineCore pid=223) INFO 08-17 11:39:15 [multiproc_executor.py:149] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=10.176.26.38 (local), world_size=2, local_world_size=2
WARNING 08-17 11:39:16 [rocm.py:134] Using CUDA_VISIBLE_DEVICES on ROCm is deprecated and support will be removed in vLLM v0.26.0. Please use HIP_VISIBLE_DEVICES instead.
WARNING 08-17 11:39:17 [rocm.py:134] Using CUDA_VISIBLE_DEVICES on ROCm is deprecated and support will be removed in vLLM v0.26.0. Please use HIP_VISIBLE_DEVICES instead.
[ERROR] min_frames is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
[ERROR] max_frames is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
[ERROR] min_frames is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
[ERROR] max_frames is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
INFO 08-17 11:39:27 [registry.py:141] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.
INFO 08-17 11:39:27 [registry.py:141] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.
(Worker pid=402) INFO 08-17 11:39:27 [parallel_state.py:1638] world_size=2 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_eec79e82294b49b496ef69a7d79f4ad2 backend=nccl
(Worker pid=403) INFO 08-17 11:39:27 [parallel_state.py:1638] world_size=2 rank=1 local_rank=1 distributed_init_method=file:///tmp/vllm_dist_eec79e82294b49b496ef69a7d79f4ad2 backend=nccl
(Worker pid=402) INFO 08-17 11:39:27 [pynccl.py:113] vLLM is using nccl==2.27.7
(Worker pid=402) INFO 08-17 11:39:30 [cuda_communicator.py:266] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'pp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=402) INFO 08-17 11:39:30 [parallel_state.py:1982] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(Worker_PP0 pid=402) INFO 08-17 11:39:31 [gpu_model_runner.py:5395] Starting to load model /models/Qwen-fp4/...
(Worker_PP0 pid=402) INFO 08-17 11:39:31 [rocm.py:742] Using backend AttentionBackendEnum.TORCH_SDPA for vit attention
(Worker_PP0 pid=402) WARNING 08-17 11:39:31 [activation.py:808] [ROCm] PyTorch's native GELU with tanh approximation is unstable. Falling back to GELU(approximate='none').
(Worker_PP0 pid=402) INFO 08-17 11:39:31 [mm_encoder_attention.py:375] Using AttentionBackendEnum.TORCH_SDPA for MMEncoderAttention.
(Worker_PP0 pid=402) INFO 08-17 11:39:31 [qwen_gdn_linear_attn.py:153] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(Worker_PP0 pid=402) INFO 08-17 11:39:31 [rocm.py:639] Using TRITON_ATTN backend (selected via --attention-backend).
(Worker_PP0 pid=402) WARNING 08-17 11:39:32 [compilation.py:1347] Op 'sparse_attn_indexer' not present in model, enabling with '+sparse_attn_indexer' has no effect
(Worker_PP0 pid=402) INFO 08-17 11:39:32 [weight_utils.py:858] Filesystem type for checkpoints: EXT4. Checkpoint size: 25.75 GiB. Available RAM: 407.85 GiB.
(Worker_PP0 pid=402) INFO 08-17 11:39:32 [weight_utils.py:881] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards: 0% Completed | 0/7 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 14% Completed | 1/7 [00:03<00:18, 3.07s/it]
Loading safetensors checkpoint shards: 29% Completed | 2/7 [00:03<00:07, 1.55s/it]
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] WorkerProc failed to start.
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] Traceback (most recent call last):
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/v1/executor/multiproc_executor.py", line 894, in worker_main
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] worker = WorkerProc(*args, **kwargs)
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] return func(*args, **kwargs)
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/v1/executor/multiproc_executor.py", line 663, in init
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] self.worker.load_model()
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/v1/worker/gpu_worker.py", line 443, in load_model
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] return func(*args, **kwargs)
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/v1/worker/gpu_model_runner.py", line 5411, in load_model
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] self.model = model_loader.load_model(
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] return func(*args, **kwargs)
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/model_loader/base_loader.py", line 64, in load_model
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] self.load_weights(model, model_config)
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] return func(*args, **kwargs)
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/model_loader/default_loader.py", line 427, in load_weights
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] loaded_weights = model.load_weights(self.get_all_weights(model_config, model))
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/qwen3_5.py", line 588, in load_weights
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] return original_load_weights(self, weights, *args, **kwargs)
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 423, in load_weights
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] autoloaded_weights = set(self._load_module("", self.module, weights))
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 357, in _load_module
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] yield from self._load_module(
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 330, in _load_module
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] loaded_params = module_load_weights(weights)
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/qwen3_5.py", line 428, in load_weights
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] return original_load_weights(self, weights, *args, **kwargs)
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 423, in load_weights
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] autoloaded_weights = set(self._load_module("", self.module, weights))
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 357, in _load_module
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] yield from self._load_module(
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 330, in _load_module
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] loaded_params = module_load_weights(weights)
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/qwen3_5.py", line 284, in load_weights
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] return original_load_weights(self, weights, *args, **kwargs)
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 423, in load_weights
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] autoloaded_weights = set(self._load_module("", self.module, weights))
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 357, in _load_module
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] yield from self._load_module(
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 357, in _load_module
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] yield from self._load_module(
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 357, in _load_module
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] yield from self._load_module(
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] [Previous line repeated 1 more time]
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 366, in _load_module
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] yield from self._load_param(
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 280, in _load_param
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] weight_loader(param, weight_data)
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/layers/linear.py", line 1633, in weight_loader_v2
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] param.load_row_parallel_weight(loaded_weight=loaded_weight)
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/parameter.py", line 222, in load_row_parallel_weight
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] loaded_weight = loaded_weight.narrow(
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^
(Worker_PP0 pid=402) ERROR 08-17 11:39:36 [multiproc_executor.py:927] RuntimeError: start (0) + length (6144) exceeds dimension size (3072).
Loading safetensors checkpoint shards: 29% Completed | 2/7 [00:04<00:10, 2.07s/it]
(Worker_PP0 pid=402)
(EngineCore pid=223) INFO 08-17 11:39:36 [multiproc_executor.py:455] [shutdown] Executor: waiting for worker exit count=2
[rank0]:[W817 11:39:36.483573035 ProcessGroupNCCL.cpp:1647] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] WorkerProc failed to start.
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] Traceback (most recent call last):
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/v1/executor/multiproc_executor.py", line 894, in worker_main
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] worker = WorkerProc(*args, **kwargs)
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] return func(*args, **kwargs)
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/v1/executor/multiproc_executor.py", line 663, in init
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] self.worker.load_model()
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/v1/worker/gpu_worker.py", line 443, in load_model
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] return func(*args, **kwargs)
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/v1/worker/gpu_model_runner.py", line 5411, in load_model
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] self.model = model_loader.load_model(
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] return func(*args, **kwargs)
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/model_loader/base_loader.py", line 64, in load_model
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] self.load_weights(model, model_config)
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] return func(*args, **kwargs)
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/model_loader/default_loader.py", line 427, in load_weights
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] loaded_weights = model.load_weights(self.get_all_weights(model_config, model))
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/qwen3_5.py", line 588, in load_weights
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] return original_load_weights(self, weights, *args, **kwargs)
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 423, in load_weights
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] autoloaded_weights = set(self._load_module("", self.module, weights))
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 357, in _load_module
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] yield from self._load_module(
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 330, in _load_module
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] loaded_params = module_load_weights(weights)
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/qwen3_5.py", line 428, in load_weights
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] return original_load_weights(self, weights, *args, **kwargs)
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 423, in load_weights
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] autoloaded_weights = set(self._load_module("", self.module, weights))
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 357, in _load_module
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] yield from self._load_module(
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 330, in _load_module
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] loaded_params = module_load_weights(weights)
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/qwen3_5.py", line 284, in load_weights
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] return original_load_weights(self, weights, *args, **kwargs)
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 423, in load_weights
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] autoloaded_weights = set(self._load_module("", self.module, weights))
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 357, in _load_module
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] yield from self._load_module(
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 357, in _load_module
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] yield from self._load_module(
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 357, in _load_module
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] yield from self._load_module(
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] [Previous line repeated 1 more time]
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 366, in _load_module
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] yield from self._load_param(
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/models/utils.py", line 280, in _load_param
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] weight_loader(param, weight_data)
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/layers/linear.py", line 1633, in weight_loader_v2
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] param.load_row_parallel_weight(loaded_weight=loaded_weight)
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] File "/workspace/vllm/model_executor/parameter.py", line 222, in load_row_parallel_weight
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] loaded_weight = loaded_weight.narrow(
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] ^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1 pid=403) ERROR 08-17 11:39:37 [multiproc_executor.py:927] RuntimeError: start (0) + length (6144) exceeds dimension size (3072).
(EngineCore pid=223) INFO 08-17 11:39:39 [multiproc_executor.py:462] [shutdown] Executor: all workers exited gracefully
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] EngineCore failed to start.
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] Traceback (most recent call last):
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] File "/workspace/vllm/v1/engine/core.py", line 1312, in run_engine_core
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] File "/workspace/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] return func(*args, **kwargs)
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] File "/workspace/vllm/v1/engine/core.py", line 1070, in init
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] super().init(
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] File "/workspace/vllm/v1/engine/core.py", line 132, in init
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] self.model_executor = executor_class(vllm_config)
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] File "/workspace/vllm/v1/executor/multiproc_executor.py", line 115, in init
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] super().init(vllm_config)
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] File "/workspace/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] return func(*args, **kwargs)
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] File "/workspace/vllm/v1/executor/abstract.py", line 109, in init
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] self._init_executor()
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] File "/workspace/vllm/v1/executor/multiproc_executor.py", line 210, in _init_executor
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] self.workers = WorkerProc.wait_for_ready(unready_workers)
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] File "/workspace/vllm/v1/executor/multiproc_executor.py", line 791, in wait_for_ready
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] raise e from None
(EngineCore pid=223) ERROR 08-17 11:39:39 [core.py:1343] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment