meet error ,when vllm service get a request. model ：official vllm-ascend deepseek-v3-W8A8 model.

[StevenBrown008]

Hardware: 2 node , 8* 910B per node, 64G mem per card.
CANN Toolkit: 8.2rc1
Ascend Driver: 25.2.0
PyTorch: 2.5.1, Torch-npu: 2.5.1.post1.dev20250619
vLLM: v0.9.2 & vLLM-Ascend: v0.9.2rc1

image build with this repo's dockerfile.

detail error:

ntegration.vllm.vllm_v1_adapter)
(VllmWorker rank=1 pid=1034) [2025-09-08 21:48:38,604] LMCache INFO: Storing KV cache for 6 out of 6 tokens (skip_leading_tokens=0) for request cmpl-f870c93bda2647ab91bf1cbed5859bab-0 (vllm_v1_adapter.py:709:lmcache.integration.vllm.vllm_v1_adapter)
(VllmWorker rank=0 pid=895) [2025-09-08 21:48:38,604] LMCache INFO: Storing KV cache for 6 out of 6 tokens (skip_leading_tokens=0) for request cmpl-f870c93bda2647ab91bf1cbed5859bab-0 (vllm_v1_adapter.py:709:lmcache.integration.vllm.vllm_v1_adapter)
(VllmWorker rank=3 pid=1807) [2025-09-08 21:48:38,604] LMCache INFO: Storing KV cache for 6 out of 6 tokens (skip_leading_tokens=0) for request cmpl-f870c93bda2647ab91bf1cbed5859bab-0 (vllm_v1_adapter.py:709:lmcache.integration.vllm.vllm_v1_adapter)
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] WorkerProc hit an exception.
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] WorkerProc hit an exception.
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] output = func(*args, **kwargs)
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] output = func(*args, **kwargs)
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 217, in execute_model
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 217, in execute_model
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] output = self.model_runner.execute_model(scheduler_output,
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] output = self.model_runner.execute_model(scheduler_output,
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] WorkerProc hit an exception.
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] output = func(*args, **kwargs)
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1421, in execute_model
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1421, in execute_model
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 217, in execute_model
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] WorkerProc hit an exception.
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] num_scheduled_tokens_np, finished_sending, finished_recving) = (self._process_reqs(
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] num_scheduled_tokens_np, finished_sending, finished_recving) = (self._process_reqs(
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] output = self.model_runner.execute_model(scheduler_output,
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1147, in _process_reqs
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1147, in _process_reqs
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] output = func(*args, **kwargs)
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.maybe_wait_for_kv_save()
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.maybe_wait_for_kv_save()
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1633, in maybe_wait_for_kv_save
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 217, in execute_model
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1633, in maybe_wait_for_kv_save
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] get_kv_transfer_group().wait_for_save()
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] output = self.model_runner.execute_model(scheduler_output,
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] get_kv_transfer_group().wait_for_save()
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1421, in execute_model
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/integration/vllm/lmcache_connector_v1.py", line 99, in wait_for_save
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/integration/vllm/lmcache_connector_v1.py", line 99, in wait_for_save
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] num_scheduled_tokens_np, finished_sending, finished_recving) = (self._process_reqs(
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self._lmcache_engine.wait_for_save()
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self._lmcache_engine.wait_for_save()
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/integration/vllm/vllm_v1_adapter.py", line 731, in wait_for_save
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/integration/vllm/vllm_v1_adapter.py", line 731, in wait_for_save
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1147, in _process_reqs
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.lmcache_engine.store(
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.lmcache_engine.store(
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.maybe_wait_for_kv_save()
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1421, in execute_model
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1633, in maybe_wait_for_kv_save
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] num_scheduled_tokens_np, finished_sending, finished_recving) = (self._process_reqs(
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] get_kv_transfer_group().wait_for_save()
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/integration/vllm/lmcache_connector_v1.py", line 99, in wait_for_save
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/cache_engine.py", line 237, in store
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1147, in _process_reqs
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/cache_engine.py", line 237, in store
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self._lmcache_engine.wait_for_save()
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.gpu_connector.batched_from_gpu(memory_objs, starts, ends, **kwargs)
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.gpu_connector.batched_from_gpu(memory_objs, starts, ends, **kwargs)
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/integration/vllm/vllm_v1_adapter.py", line 731, in wait_for_save
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/gpu_connector.py", line 293, in batched_from_gpu
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/gpu_connector.py", line 293, in batched_from_gpu
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.lmcache_engine.store(
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.from_gpu(memory_obj, start, end, **kwargs)
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.from_gpu(memory_obj, start, end, **kwargs)
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.maybe_wait_for_kv_save()
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/gpu_connector.py", line 248, in from_gpu
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/gpu_connector.py", line 248, in from_gpu
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1633, in maybe_wait_for_kv_save
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] kv_cache_pointers = self._initialize_pointers(self.kvcaches)
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] kv_cache_pointers = self._initialize_pointers(self.kvcaches)
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] get_kv_transfer_group().wait_for_save()
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/integration/vllm/lmcache_connector_v1.py", line 99, in wait_for_save
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/cache_engine.py", line 237, in store
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache-Ascend/lmcache_ascend/v1/npu_connector.py", line 22, in _initialize_pointers
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache-Ascend/lmcache_ascend/v1/npu_connector.py", line 22, in _initialize_pointers
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self._lmcache_engine.wait_for_save()
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.gpu_connector.batched_from_gpu(memory_objs, starts, ends, **kwargs)
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.kv_cache_pointers.numpy()[:] = [t.data_ptr() for t in kv_caches]
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.kv_cache_pointers.numpy()[:] = [t.data_ptr() for t in kv_caches]
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/integration/vllm/vllm_v1_adapter.py", line 731, in wait_for_save
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/gpu_connector.py", line 293, in batched_from_gpu
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.lmcache_engine.store(
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.from_gpu(memory_obj, start, end, **kwargs)
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache-Ascend/lmcache_ascend/v1/npu_connector.py", line 22, in
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache-Ascend/lmcache_ascend/v1/npu_connector.py", line 22, in
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/gpu_connector.py", line 248, in from_gpu
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.kv_cache_pointers.numpy()[:] = [t.data_ptr() for t in kv_caches]
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.kv_cache_pointers.numpy()[:] = [t.data_ptr() for t in kv_caches]
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] kv_cache_pointers = self._initialize_pointers(self.kvcaches)
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] AttributeError: 'tuple' object has no attribute 'data_ptr'
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] AttributeError: 'tuple' object has no attribute 'data_ptr'
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/cache_engine.py", line 237, in store
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache-Ascend/lmcache_ascend/v1/npu_connector.py", line 22, in _initialize_pointers
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.gpu_connector.batched_from_gpu(memory_objs, starts, ends, **kwargs)
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.kv_cache_pointers.numpy()[:] = [t.data_ptr() for t in kv_caches]
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/gpu_connector.py", line 293, in batched_from_gpu
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] output = func(*args, **kwargs)
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] output = func(*args, **kwargs)
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.from_gpu(memory_obj, start, end, **kwargs)
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache-Ascend/lmcache_ascend/v1/npu_connector.py", line 22, in
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/gpu_connector.py", line 248, in from_gpu
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.kv_cache_pointers.numpy()[:] = [t.data_ptr() for t in kv_caches]
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 217, in execute_model
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 217, in execute_model
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] kv_cache_pointers = self._initialize_pointers(self.kvcaches)
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] output = self.model_runner.execute_model(scheduler_output,
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] output = self.model_runner.execute_model(scheduler_output,
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] AttributeError: 'tuple' object has no attribute 'data_ptr'
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache-Ascend/lmcache_ascend/v1/npu_connector.py", line 22, in _initialize_pointers
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.kv_cache_pointers.numpy()[:] = [t.data_ptr() for t in kv_caches]
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] output = func(*args, **kwargs)
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache-Ascend/lmcache_ascend/v1/npu_connector.py", line 22, in
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1421, in execute_model
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1421, in execute_model
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.kv_cache_pointers.numpy()[:] = [t.data_ptr() for t in kv_caches]
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 217, in execute_model
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] num_scheduled_tokens_np, finished_sending, finished_recving) = (self._process_reqs(
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] num_scheduled_tokens_np, finished_sending, finished_recving) = (self._process_reqs(
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] output = self.model_runner.execute_model(scheduler_output,
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] AttributeError: 'tuple' object has no attribute 'data_ptr'
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1147, in _process_reqs
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1147, in _process_reqs
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.maybe_wait_for_kv_save()
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.maybe_wait_for_kv_save()
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1633, in maybe_wait_for_kv_save
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1633, in maybe_wait_for_kv_save
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] output = func(*args, **kwargs)
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] get_kv_transfer_group().wait_for_save()
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] get_kv_transfer_group().wait_for_save()
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1421, in execute_model
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/integration/vllm/lmcache_connector_v1.py", line 99, in wait_for_save
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/integration/vllm/lmcache_connector_v1.py", line 99, in wait_for_save
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 217, in execute_model
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] num_scheduled_tokens_np, finished_sending, finished_recving) = (self._process_reqs(
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self._lmcache_engine.wait_for_save()
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self._lmcache_engine.wait_for_save()
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] output = self.model_runner.execute_model(scheduler_output,
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/integration/vllm/vllm_v1_adapter.py", line 731, in wait_for_save
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/integration/vllm/vllm_v1_adapter.py", line 731, in wait_for_save
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1147, in _process_reqs
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.lmcache_engine.store(
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.lmcache_engine.store(
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.maybe_wait_for_kv_save()
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1633, in maybe_wait_for_kv_save
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] get_kv_transfer_group().wait_for_save()
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1421, in execute_model
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/integration/vllm/lmcache_connector_v1.py", line 99, in wait_for_save
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/cache_engine.py", line 237, in store
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/cache_engine.py", line 237, in store
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] num_scheduled_tokens_np, finished_sending, finished_recving) = (self._process_reqs(
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self._lmcache_engine.wait_for_save()
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.gpu_connector.batched_from_gpu(memory_objs, starts, ends, **kwargs)
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.gpu_connector.batched_from_gpu(memory_objs, starts, ends, **kwargs)
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/integration/vllm/vllm_v1_adapter.py", line 731, in wait_for_save
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/gpu_connector.py", line 293, in batched_from_gpu
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/gpu_connector.py", line 293, in batched_from_gpu
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1147, in _process_reqs
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.lmcache_engine.store(
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.from_gpu(memory_obj, start, end, **kwargs)
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.from_gpu(memory_obj, start, end, **kwargs)
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.maybe_wait_for_kv_save()
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/gpu_connector.py", line 248, in from_gpu
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/gpu_connector.py", line 248, in from_gpu
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1633, in maybe_wait_for_kv_save
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] kv_cache_pointers = self._initialize_pointers(self.kvcaches)
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] kv_cache_pointers = self._initialize_pointers(self.kvcaches)
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] get_kv_transfer_group().wait_for_save()
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/integration/vllm/lmcache_connector_v1.py", line 99, in wait_for_save
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/cache_engine.py", line 237, in store
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache-Ascend/lmcache_ascend/v1/npu_connector.py", line 22, in _initialize_pointers
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache-Ascend/lmcache_ascend/v1/npu_connector.py", line 22, in _initialize_pointers
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self._lmcache_engine.wait_for_save()
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.gpu_connector.batched_from_gpu(memory_objs, starts, ends, **kwargs)
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.kv_cache_pointers.numpy()[:] = [t.data_ptr() for t in kv_caches]
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.kv_cache_pointers.numpy()[:] = [t.data_ptr() for t in kv_caches]
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/integration/vllm/vllm_v1_adapter.py", line 731, in wait_for_save
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/gpu_connector.py", line 293, in batched_from_gpu
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.lmcache_engine.store(
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.from_gpu(memory_obj, start, end, **kwargs)
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache-Ascend/lmcache_ascend/v1/npu_connector.py", line 22, in
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache-Ascend/lmcache_ascend/v1/npu_connector.py", line 22, in
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/gpu_connector.py", line 248, in from_gpu
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.kv_cache_pointers.numpy()[:] = [t.data_ptr() for t in kv_caches]
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.kv_cache_pointers.numpy()[:] = [t.data_ptr() for t in kv_caches]
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] kv_cache_pointers = self._initialize_pointers(self.kvcaches)
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522] AttributeError: 'tuple' object has no attribute 'data_ptr'
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522] AttributeError: 'tuple' object has no attribute 'data_ptr'
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/cache_engine.py", line 237, in store
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache-Ascend/lmcache_ascend/v1/npu_connector.py", line 22, in _initialize_pointers
(VllmWorker rank=2 pid=1205) ERROR 09-08 21:48:38 [multiproc_executor.py:522]
(VllmWorker rank=3 pid=1807) ERROR 09-08 21:48:38 [multiproc_executor.py:522]
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.gpu_connector.batched_from_gpu(memory_objs, starts, ends, **kwargs)
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.kv_cache_pointers.numpy()[:] = [t.data_ptr() for t in kv_caches]
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/gpu_connector.py", line 293, in batched_from_gpu
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.from_gpu(memory_obj, start, end, **kwargs)
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache-Ascend/lmcache_ascend/v1/npu_connector.py", line 22, in
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache/lmcache/v1/gpu_connector.py", line 248, in from_gpu
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.kv_cache_pointers.numpy()[:] = [t.data_ptr() for t in kv_caches]
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] kv_cache_pointers = self._initialize_pointers(self.kvcaches)
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] AttributeError: 'tuple' object has no attribute 'data_ptr'
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache-Ascend/lmcache_ascend/v1/npu_connector.py", line 22, in _initialize_pointers
(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522]
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.kv_cache_pointers.numpy()[:] = [t.data_ptr() for t in kv_caches]
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] File "/workspace/LMCache-Ascend/lmcache_ascend/v1/npu_connector.py", line 22, in
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.kv_cache_pointers.numpy()[:] = [t.data_ptr() for t in kv_caches]
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] ^^^^^^^^^^
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522] AttributeError: 'tuple' object has no attribute 'data_ptr'
(VllmWorker rank=0 pid=895) ERROR 09-08 21:48:38 [multiproc_executor.py:522]
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.9.2) with config: model='/data/models/DeepSeek-V3-W8A8', speculative_config=None, tokenizer='/data/models/DeepSeek-V3-W8A8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=ascend, enforce_eager=False, kv_cache_dtype=auto, device_config=npu, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=1024, served_model_name=deepseek_v3, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":["all"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null},
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=cmpl-f870c93bda2647ab91bf1cbed5859bab-0,prompt_token_ids_len=6,mm_inputs=[],mm_hashes=[],mm_positions=[],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.0, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=50, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None),block_ids=([1],),num_computed_tokens=0,lora_request=None)], scheduled_cached_reqs=CachedRequestData(req_ids=[], resumed_from_preemption=[], new_token_ids=[], new_block_ids=[], num_computed_tokens=[]), num_scheduled_tokens={cmpl-f870c93bda2647ab91bf1cbed5859bab-0: 6}, total_num_scheduled_tokens=6, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[1], finished_req_ids=[], free_encoder_input_ids=[], structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=LMCacheConnectorMetadata(requests=[ReqMeta(req_id='cmpl-f870c93bda2647ab91bf1cbed5859bab-0', token_ids=Tensor(shape=torch.Size([6]), device=cpu,dtype=torch.int64), slot_mapping=Tensor(shape=torch.Size([6]), device=cpu,dtype=torch.int64), is_last_prefill=true, save_spec=SaveSpec(skip_leading_tokens=0, can_save=true), load_spec=null, disagg_spec=null)], lookup_requests_in_step=['cmpl-f870c93bda2647ab91bf1cbed5859bab-0']))
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, kv_cache_usage=0.006134969325153339, prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0), spec_decoding_stats=None, num_corrupted_reqs=0)
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] EngineCore encountered a fatal error.
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] Traceback (most recent call last):
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] File "/workspace/vllm/vllm/v1/engine/core.py", line 579, in run_engine_core
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] engine_core.run_busy_loop()
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] File "/workspace/vllm/vllm/v1/engine/core.py", line 938, in run_busy_loop
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] executed = self._process_engine_step()
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] File "/workspace/vllm/vllm/v1/engine/core.py", line 631, in _process_engine_step
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] outputs, model_executed = self.step_fn()
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] ^^^^^^^^^^^^^^
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] File "/workspace/vllm/vllm/v1/engine/core.py", line 235, in step
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] model_output = self.execute_model(scheduler_output)
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] File "/workspace/vllm/vllm/v1/engine/core.py", line 221, in execute_model
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] raise err
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] File "/workspace/vllm/vllm/v1/engine/core.py", line 212, in execute_model
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] return self.model_executor.execute_model(scheduler_output)
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 158, in execute_model
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] (output, ) = self.collective_rpc(
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] ^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 215, in collective_rpc
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] result = get_response(w, dequeue_timeout)
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 202, in get_response
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] raise RuntimeError(
(EngineCore_0 pid=666) ERROR 09-08 21:48:38 [core.py:588] RuntimeError: Worker failed with error ''tuple' object has no attribute 'data_ptr'', please check the stack trace above for the root cause
(EngineCore_0 pid=666) /workspace/vllm/vllm/v1/engine/core.py:766: ResourceWarning: Destroying context with unclosed socket <zmq.Socket(zmq.PUSH) at 0xfffc9f9be200>
(EngineCore_0 pid=666) with ExitStack() as stack, zmq.Context() as ctx:
(EngineCore_0 pid=666) /workspace/vllm/vllm/v1/engine/core.py:766: ResourceWarning: Destroying context with unclosed socket <zmq.Socket(zmq.PUSH) at 0xfffc9f9be120>
(EngineCore_0 pid=666) with ExitStack() as stack, zmq.Context() as ctx:
(EngineCore_0 pid=666) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffdead13cb0>
(EngineCore_0 pid=666) w.worker_response_mq = None
(EngineCore_0 pid=666) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdead27ad0>
(EngineCore_0 pid=666) w.worker_response_mq = None
(EngineCore_0 pid=666) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffde92a6c10>
(EngineCore_0 pid=666) w.worker_response_mq = None
(EngineCore_0 pid=666) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdead279b0>
(EngineCore_0 pid=666) w.worker_response_mq = None
ERROR 09-08 21:48:38 [async_llm.py:419] AsyncLLM output_handler failed.
ERROR 09-08 21:48:38 [async_llm.py:419] Traceback (most recent call last):
ERROR 09-08 21:48:38 [async_llm.py:419] File "/workspace/vllm/vllm/v1/engine/async_llm.py", line 378, in output_handler
ERROR 09-08 21:48:38 [async_llm.py:419] outputs = await engine_core.get_output_async()
ERROR 09-08 21:48:38 [async_llm.py:419] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 09-08 21:48:38 [async_llm.py:419] File "/workspace/vllm/vllm/v1/engine/core_client.py", line 740, in get_output_async
ERROR 09-08 21:48:38 [async_llm.py:419] raise self._format_exception(outputs) from None
ERROR 09-08 21:48:38 [async_llm.py:419] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
INFO 09-08 21:48:38 [async_llm.py:345] Request cmpl-f870c93bda2647ab91bf1cbed5859bab-0 failed (engine dead).
(EngineCore_0 pid=666) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffde91ff070>
(EngineCore_0 pid=666) w.worker_response_mq = None
(EngineCore_0 pid=666) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdead27830>
(EngineCore_0 pid=666) w.worker_response_mq = None
INFO: 127.0.0.1:22498 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error

[StevenBrown008]

need for help , meet "AttributeError: 'tuple' object has no attribute 'data_ptr'" . gentle man ,please cc .@matthewygf@YaoJiayi @chloroethylene @sammshen

[StevenBrown008]

@maobaolong

[matthewygf]

Thanks for the issue.
Just to confirm a few things:

1. node 920B <--- is this correct ?
2. did you turn on torchair ?
3. are you sure the vllm-ascend version is 0.9.2rc1

we have encountered this issue in vllm-ascend v0.10.x for other models, which we plan to contribute a fix soon.

(VllmWorker rank=1 pid=1034) ERROR 09-08 21:48:38 [multiproc_executor.py:522] self.kv_cache_pointers.numpy()[:] = [t.data_ptr() for t in kv_caches]
The issue in v0.10.x is due to kvcaches layout changes into tuple instead of tensor under graph mode.

[StevenBrown008]

Thanks a lot for your answer !

1. node hardware: 8* 910b .Sorry, I will fix the description.
2. yes. torchair is on.
3. yes. the image is build based on Dockerfile.a2.openEuler in this repo.

Here is two detailed shell file for start vllm service integrated with lmcache in docker :

on node0 start shell:

#!/bin/sh
nic_name="bond0.1318"
local_ip="10.151.18.93"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export HCCL_BUFFSIZE=1024
vllm serve /data/models/DeepSeek-V3-W8A8
--host 0.0.0.0
--port 8004
--data-parallel-size 4
--data-parallel-size-local 2
--data-parallel-address $local_ip
--data-parallel-rpc-port 13389
--tensor-parallel-size 4
--seed 1024
--served-model-name deepseek_v3
--enable-expert-parallel
--max-num-seqs 16
--max-model-len 32768
--max-num-batched-tokens 4096
--quantization ascend
--trust-remote-code
--no-enable-prefix-caching
--gpu-memory-utilization 0.9
--kv-transfer-config '{"kv_connector":"LMCacheAscendConnectorV1Dynamic","kv_role":"kv_both", "kv_connector_module_path":"lmcache_ascend.integration.vllm.lmcache_ascend_connector_v1"}'
--additional-config '{"ascend_scheduler_config":{"enabled":true},"torchair_graph_config":{"enabled": true}}'

on node1 start shell :

#!/bin/sh
nic_name="bond0.1318"
local_ip="10.151.18.95"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export VLLM_USE_V1=1
export HCCL_BUFFSIZE=1024
vllm serve /data/models/DeepSeek-V3-W8A8
--host 0.0.0.0
--port 8004
--headless
--data-parallel-size 4
--data-parallel-size-local 2
--data-parallel-start-rank 2
--data-parallel-address "10.151.18.93"
--data-parallel-rpc-port 13389
--tensor-parallel-size 4
--seed 1024
--served-model-name deepseek_v3
--max-num-seqs 16
--max-model-len 32768
--max-num-batched-tokens 4096
--enable-expert-parallel
--trust-remote-code
--quantization ascend
--no-enable-prefix-caching
--gpu-memory-utilization 0.92
--kv-transfer-config '{"kv_connector":"LMCacheAscendConnectorV1Dynamic","kv_role":"kv_both", "kv_connector_module_path":"lmcache_ascend.integration.vllm.lmcache_ascend_connector_v1"}'
--additional-config '{"ascend_scheduler_config":{"enabled":true},"torchair_graph_config":{"enabled": true}}'

The above configuration is learn from a reference of vllm-ascend official document:
https://vllm-ascend.readthedocs.io/en/v0.9.1-dev/tutorials/multi_node.html
Did I have some configuration error ? Look forward to your reply !
@matthewygf

[matthewygf]

Thanks a lot for your answer !

1. node hardware: 8* 910b .Sorry, I will fix the description.
2. yes. torchair is on.
3. yes. the image is build based on Dockerfile.a2.openEuler in this repo.

Here is two detailed shell file for start vllm service integrated with lmcache in docker :

on node0 start shell:

#!/bin/sh nic_name="bond0.1318" local_ip="10.151.18.93" export HCCL_IF_IP=$local_ip export GLOO_SOCKET_IFNAME=$nic_name export TP_SOCKET_IFNAME=$nic_name export HCCL_SOCKET_IFNAME=$nic_name export OMP_PROC_BIND=false export OMP_NUM_THREADS=100 export HCCL_BUFFSIZE=1024 vllm serve /data/models/DeepSeek-V3-W8A8 --host 0.0.0.0 --port 8004 --data-parallel-size 4 --data-parallel-size-local 2 --data-parallel-address $local_ip --data-parallel-rpc-port 13389 --tensor-parallel-size 4 --seed 1024 --served-model-name deepseek_v3 --enable-expert-parallel --max-num-seqs 16 --max-model-len 32768 --max-num-batched-tokens 4096 --quantization ascend --trust-remote-code --no-enable-prefix-caching --gpu-memory-utilization 0.9 --kv-transfer-config '{"kv_connector":"LMCacheAscendConnectorV1Dynamic","kv_role":"kv_both", "kv_connector_module_path":"lmcache_ascend.integration.vllm.lmcache_ascend_connector_v1"}' --additional-config '{"ascend_scheduler_config":{"enabled":true},"torchair_graph_config":{"enabled": true}}'

on node1 start shell :

#!/bin/sh nic_name="bond0.1318" local_ip="10.151.18.95" export HCCL_IF_IP=$local_ip export GLOO_SOCKET_IFNAME=$nic_name export TP_SOCKET_IFNAME=$nic_name export HCCL_SOCKET_IFNAME=$nic_name export OMP_PROC_BIND=false export OMP_NUM_THREADS=100 export VLLM_USE_V1=1 export HCCL_BUFFSIZE=1024 vllm serve /data/models/DeepSeek-V3-W8A8 --host 0.0.0.0 --port 8004 --headless --data-parallel-size 4 --data-parallel-size-local 2 --data-parallel-start-rank 2 --data-parallel-address "10.151.18.93" --data-parallel-rpc-port 13389 --tensor-parallel-size 4 --seed 1024 --served-model-name deepseek_v3 --max-num-seqs 16 --max-model-len 32768 --max-num-batched-tokens 4096 --enable-expert-parallel --trust-remote-code --quantization ascend --no-enable-prefix-caching --gpu-memory-utilization 0.92 --kv-transfer-config '{"kv_connector":"LMCacheAscendConnectorV1Dynamic","kv_role":"kv_both", "kv_connector_module_path":"lmcache_ascend.integration.vllm.lmcache_ascend_connector_v1"}' --additional-config '{"ascend_scheduler_config":{"enabled":true},"torchair_graph_config":{"enabled": true}}'

The above configuration is learn from a reference of vllm-ascend official document: https://vllm-ascend.readthedocs.io/en/v0.9.1-dev/tutorials/multi_node.html Did I have some configuration error ? Look forward to your reply ! @matthewygf

No problem, thanks for trying out the Ascend-plugin. The issue is due to using torchair, where using torchair , the kvcache manager will change the kvcache shapes. We are in the process of investigating and fixing this.

[StevenBrown008]

Thanks for your answer. I have removed the torchair config and set enforce_eager for vllm , the above error was resolved. But meet new error when request come to vllm:

`INFO: Started server process [3104]
INFO: Waiting for application startup.
INFO: Application startup complete.

INFO 09-09 23:19:53 [logger.py:43] Received request cmpl-d0a2ca7c46204247acb08e143efbe02a-0: prompt: 'The future of AI is', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.0, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=50, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: [0, 671, 3988, 294, 7703, 344], prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None.
INFO 09-09 23:19:53 [async_llm.py:270] Added request cmpl-d0a2ca7c46204247acb08e143efbe02a-0.
(EngineCore_0 pid=3249) [2025-09-09 23:19:54,168] LMCache INFO: Reqid: cmpl-d0a2ca7c46204247acb08e143efbe02a-0, Total tokens 6, LMCache hit tokens: 0, need to load: 0 (vllm_v1_adapter.py:803:lmcache.integration.vllm.vllm_v1_adapter)
(EngineCore_0 pid=3249) ('Warning: torch.save with "_use_new_zipfile_serialization = False" is not recommended for npu tensor, which may bring unexpected errors and hopefully set "_use_new_zipfile_serialization = True"', 'if it is necessary to use this, please convert the npu tensor to cpu tensor for saving')
(VllmWorker rank=1 pid=3616) [2025-09-09 23:19:54,432] LMCache INFO: Post-initializing LMCacheEngine (cache_engine.py:148:lmcache.v1.cache_engine)
(VllmWorker rank=0 pid=3479) [2025-09-09 23:19:54,518] LMCache INFO: Post-initializing LMCacheEngine (cache_engine.py:148:lmcache.v1.cache_engine)
(VllmWorker rank=2 pid=3786) [2025-09-09 23:19:54,549] LMCache INFO: Post-initializing LMCacheEngine (cache_engine.py:148:lmcache.v1.cache_engine)
(VllmWorker rank=3 pid=4387) [2025-09-09 23:19:54,554] LMCache INFO: Post-initializing LMCacheEngine (cache_engine.py:148:lmcache.v1.cache_engine)
(VllmWorker rank=2 pid=3786) [2025-09-09 23:19:59,332] LMCache INFO: Storing KV cache for 6 out of 6 tokens (skip_leading_tokens=0) for request cmpl-d0a2ca7c46204247acb08e143efbe02a-0 (vllm_v1_adapter.py:709:lmcache.integration.vllm.vllm_v1_adapter)
(VllmWorker rank=3 pid=4387) [2025-09-09 23:19:59,332] LMCache INFO: Storing KV cache for 6 out of 6 tokens (skip_leading_tokens=0) for request cmpl-d0a2ca7c46204247acb08e143efbe02a-0 (vllm_v1_adapter.py:709:lmcache.integration.vllm.vllm_v1_adapter)
(VllmWorker rank=0 pid=3479) [2025-09-09 23:19:59,332] LMCache INFO: Storing KV cache for 6 out of 6 tokens (skip_leading_tokens=0) for request cmpl-d0a2ca7c46204247acb08e143efbe02a-0 (vllm_v1_adapter.py:709:lmcache.integration.vllm.vllm_v1_adapter)
(VllmWorker rank=1 pid=3616) [2025-09-09 23:19:59,332] LMCache INFO: Storing KV cache for 6 out of 6 tokens (skip_leading_tokens=0) for request cmpl-d0a2ca7c46204247acb08e143efbe02a-0 (vllm_v1_adapter.py:709:lmcache.integration.vllm.vllm_v1_adapter)
(VllmWorker rank=2 pid=3786) [2025-09-09 23:19:59,333] LMCache INFO: Stored 6 out of total 6 tokens. size: 0.0004 gb, cost 1.1872 ms, throughput: 0.3307 GB/s; offload_time: 1.1417 ms, put_time: 0.0455 ms (cache_engine.py:251:lmcache.v1.cache_engine)
(VllmWorker rank=1 pid=3616) [2025-09-09 23:19:59,334] LMCache INFO: Stored 6 out of total 6 tokens. size: 0.0004 gb, cost 1.3902 ms, throughput: 0.2825 GB/s; offload_time: 1.3286 ms, put_time: 0.0616 ms (cache_engine.py:251:lmcache.v1.cache_engine)
(VllmWorker rank=3 pid=4387) [2025-09-09 23:19:59,334] LMCache INFO: Stored 6 out of total 6 tokens. size: 0.0004 gb, cost 1.5782 ms, throughput: 0.2488 GB/s; offload_time: 1.4997 ms, put_time: 0.0785 ms (cache_engine.py:251:lmcache.v1.cache_engine)
(VllmWorker rank=0 pid=3479) [2025-09-09 23:19:59,350] LMCache INFO: Stored 6 out of total 6 tokens. size: 0.0004 gb, cost 17.5528 ms, throughput: 0.0224 GB/s; offload_time: 17.4806 ms, put_time: 0.0722 ms (cache_engine.py:251:lmcache.v1.cache_engine)
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] EngineCore encountered a fatal error.
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] Traceback (most recent call last):
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] File "/workspace/vllm/vllm/v1/engine/core.py", line 579, in run_engine_core
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] engine_core.run_busy_loop()
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] File "/workspace/vllm/vllm/v1/engine/core.py", line 938, in run_busy_loop
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] executed = self._process_engine_step()
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] File "/workspace/vllm/vllm/v1/engine/core.py", line 631, in _process_engine_step
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] outputs, model_executed = self.step_fn()
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] ^^^^^^^^^^^^^^
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] File "/workspace/vllm/vllm/v1/engine/core.py", line 234, in step
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] scheduler_output = self.scheduler.schedule()
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] File "/workspace/vllm-ascend/vllm_ascend/core/scheduler.py", line 405, in schedule
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] meta = self.connector.build_connector_meta(scheduler_output)
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] File "/workspace/LMCache/lmcache/integration/vllm/lmcache_connector_v1.py", line 162, in build_connector_meta
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] return self._lmcache_engine.build_connector_meta(scheduler_output)
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] File "/workspace/LMCache/lmcache/integration/vllm/vllm_v1_adapter.py", line 971, in build_connector_meta
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] raise ValueError(
(EngineCore_0 pid=3249) ERROR 09-09 23:19:59 [core.py:588] ValueError: Request cmpl-d0a2ca7c46204247acb08e143efbe02a-0 is not in _unfinished_requests, but it is scheduled to be cached
(EngineCore_0 pid=3249) /workspace/vllm/vllm/v1/engine/core.py:766: ResourceWarning: Destroying context with unclosed socket <zmq.Socket(zmq.PUSH) at 0xfffca6195e10>
(EngineCore_0 pid=3249) with ExitStack() as stack, zmq.Context() as ctx:
(EngineCore_0 pid=3249) /workspace/vllm/vllm/v1/engine/core.py:766: ResourceWarning: Destroying context with unclosed socket <zmq.Socket(zmq.PUSH) at 0xfffca6195d30>
(EngineCore_0 pid=3249) with ExitStack() as stack, zmq.Context() as ctx:
(EngineCore_0 pid=3249) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffdeb977a10>
(EngineCore_0 pid=3249) w.worker_response_mq = None
(EngineCore_0 pid=3249) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed context <zmq.Context() at 0xfffded4cbb30>
(EngineCore_0 pid=3249) w.worker_response_mq = None
ERROR 09-09 23:19:59 [async_llm.py:419] AsyncLLM output_handler failed.
ERROR 09-09 23:19:59 [async_llm.py:419] Traceback (most recent call last):
ERROR 09-09 23:19:59 [async_llm.py:419] File "/workspace/vllm/vllm/v1/engine/async_llm.py", line 378, in output_handler
ERROR 09-09 23:19:59 [async_llm.py:419] outputs = await engine_core.get_output_async()
ERROR 09-09 23:19:59 [async_llm.py:419] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 09-09 23:19:59 [async_llm.py:419] File "/workspace/vllm/vllm/v1/engine/core_client.py", line 740, in get_output_async
ERROR 09-09 23:19:59 [async_llm.py:419] raise self._format_exception(outputs) from None
ERROR 09-09 23:19:59 [async_llm.py:419] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
INFO 09-09 23:19:59 [async_llm.py:345] Request cmpl-d0a2ca7c46204247acb08e143efbe02a-0 failed (engine dead).
(EngineCore_0 pid=3249) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffdeb9b2580>
(EngineCore_0 pid=3249) w.worker_response_mq = None
(EngineCore_0 pid=3249) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed context <zmq.Context() at 0xfffded4cb8f0>
(EngineCore_0 pid=3249) w.worker_response_mq = None
INFO: 127.0.0.1:46174 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error
(EngineCore_0 pid=3249) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffdeb92e350>
(EngineCore_0 pid=3249) w.worker_response_mq = None
(EngineCore_0 pid=3249) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed context <zmq.Context() at 0xfffded4cb890>
(EngineCore_0 pid=3249) w.worker_response_mq = None
(EngineCore_0 pid=3249) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffdeba5e430>
(EngineCore_0 pid=3249) w.worker_response_mq = None
(EngineCore_0 pid=3249) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed context <zmq.Context() at 0xfffded4cba10>
(EngineCore_0 pid=3249) w.worker_response_mq = None
(VllmWorker rank=0 pid=3479) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed socket <zmq.Socket(zmq.XPUB) at 0xfffdd35152b0>
(VllmWorker rank=0 pid=3479) self.worker_response_mq = None
(VllmWorker rank=0 pid=3479) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdd3512090>
(VllmWorker rank=0 pid=3479) self.worker_response_mq = None
(VllmWorker rank=2 pid=3786) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed socket <zmq.Socket(zmq.XPUB) at 0xfffdccb65400>
(VllmWorker rank=2 pid=3786) self.worker_response_mq = None
(VllmWorker rank=2 pid=3786) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdccb62090>
(VllmWorker rank=2 pid=3786) self.worker_response_mq = None
(VllmWorker rank=0 pid=3479) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed socket <zmq.Socket(zmq.XPUB) at 0xfffd672da0b0>
(VllmWorker rank=0 pid=3479) self.mq_broadcaster = None
(VllmWorker rank=0 pid=3479) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed context <zmq.Context() at 0xfffd672e37d0>
(VllmWorker rank=0 pid=3479) self.mq_broadcaster = None
(VllmWorker rank=2 pid=3786) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffd608da200>
(VllmWorker rank=2 pid=3786) self.mq_broadcaster = None
(VllmWorker rank=2 pid=3786) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed context <zmq.Context() at 0xfffd60941b50>
(VllmWorker rank=2 pid=3786) self.mq_broadcaster = None
(VllmWorker rank=1 pid=3616) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed socket <zmq.Socket(zmq.XPUB) at 0xfffdd6d79780>
(VllmWorker rank=1 pid=3616) self.worker_response_mq = None
(VllmWorker rank=1 pid=3616) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdd6d76090>
(VllmWorker rank=1 pid=3616) self.worker_response_mq = None
(VllmWorker rank=3 pid=4387) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed socket <zmq.Socket(zmq.XPUB) at 0xfffde6be14e0>
(VllmWorker rank=3 pid=4387) self.worker_response_mq = None
(VllmWorker rank=3 pid=4387) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed context <zmq.Context() at 0xfffde6bde090>
(VllmWorker rank=3 pid=4387) self.worker_response_mq = None
(VllmWorker rank=1 pid=3616) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffd6ab9e2e0>
(VllmWorker rank=1 pid=3616) self.mq_broadcaster = None
(VllmWorker rank=1 pid=3616) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed context <zmq.Context() at 0xfffd6aba7890>
(VllmWorker rank=1 pid=3616) self.mq_broadcaster = None
(VllmWorker rank=3 pid=4387) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffd7a946200>
(VllmWorker rank=3 pid=4387) self.mq_broadcaster = None
(VllmWorker rank=3 pid=4387) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed context <zmq.Context() at 0xfffd7a94f7d0>
(VllmWorker rank=3 pid=4387) self.mq_broadcaster = None
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] WorkerProc hit an exception.
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] output = func(*args, **kwargs)
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 300, in execute_dummy_batch
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] max_num_tokens, with_prefill = runner._get_forward_metadata_across_dp(
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 561, in _get_forward_metadata_across_dp
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] dist.all_reduce(forward_metadata,
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 2506, in all_reduce
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] work.wait()
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [10.151.18.93]:29989
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] output = func(*args, **kwargs)
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 300, in execute_dummy_batch
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] max_num_tokens, with_prefill = runner._get_forward_metadata_across_dp(
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 561, in _get_forward_metadata_across_dp
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] dist.all_reduce(forward_metadata,
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 2506, in all_reduce
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] work.wait()
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522] RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [10.151.18.93]:29989
(VllmWorker rank=0 pid=3478) ERROR 09-09 23:19:59 [multiproc_executor.py:522]
(EngineCore_1 pid=3384) ERROR 09-09 23:19:59 [core.py:588] EngineCore encountered a fatal error.
(EngineCore_1 pid=3384) ERROR 09-09 23:19:59 [core.py:588] Traceback (most recent call last):
(EngineCore_1 pid=3384) ERROR 09-09 23:19:59 [core.py:588] File "/workspace/vllm/vllm/v1/engine/core.py", line 579, in run_engine_core
(EngineCore_1 pid=3384) ERROR 09-09 23:19:59 [core.py:588] engine_core.run_busy_loop()
(EngineCore_1 pid=3384) ERROR 09-09 23:19:59 [core.py:588] File "/workspace/vllm/vllm/v1/engine/core.py", line 949, in run_busy_loop
(EngineCore_1 pid=3384) ERROR 09-09 23:19:59 [core.py:588] self.execute_dummy_batch()
(EngineCore_1 pid=3384) ERROR 09-09 23:19:59 [core.py:588] File "/workspace/vllm/vllm/v1/engine/core.py", line 322, in execute_dummy_batch
(EngineCore_1 pid=3384) ERROR 09-09 23:19:59 [core.py:588] self.model_executor.collective_rpc("execute_dummy_batch")
(EngineCore_1 pid=3384) ERROR 09-09 23:19:59 [core.py:588] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 215, in collective_rpc
(EngineCore_1 pid=3384) ERROR 09-09 23:19:59 [core.py:588] result = get_response(w, dequeue_timeout)
(EngineCore_1 pid=3384) ERROR 09-09 23:19:59 [core.py:588] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_1 pid=3384) ERROR 09-09 23:19:59 [core.py:588] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 202, in get_response
(EngineCore_1 pid=3384) ERROR 09-09 23:19:59 [core.py:588] raise RuntimeError(
(EngineCore_1 pid=3384) ERROR 09-09 23:19:59 [core.py:588] RuntimeError: Worker failed with error '[/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [10.151.18.93]:29989', please check the stack trace above for the root cause
(EngineCore_1 pid=3384) /workspace/vllm/vllm/v1/engine/core.py:766: ResourceWarning: Destroying context with unclosed socket <zmq.Socket(zmq.PUSH) at 0xfffca4546200>
(EngineCore_1 pid=3384) with ExitStack() as stack, zmq.Context() as ctx:
(EngineCore_1 pid=3384) /workspace/vllm/vllm/v1/engine/core.py:766: ResourceWarning: Destroying context with unclosed socket <zmq.Socket(zmq.PUSH) at 0xfffca4546120>
(EngineCore_1 pid=3384) with ExitStack() as stack, zmq.Context() as ctx:
(EngineCore_1 pid=3384) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffde558ecf0>
(EngineCore_1 pid=3384) w.worker_response_mq = None
(EngineCore_1 pid=3384) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed context <zmq.Context() at 0xfffde70b3830>
(EngineCore_1 pid=3384) w.worker_response_mq = None
(EngineCore_1 pid=3384) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffde8bea350>
(EngineCore_1 pid=3384) w.worker_response_mq = None
(EngineCore_1 pid=3384) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed context <zmq.Context() at 0xfffde70b3950>
(EngineCore_1 pid=3384) w.worker_response_mq = None
(EngineCore_1 pid=3384) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffde558edd0>
(EngineCore_1 pid=3384) w.worker_response_mq = None
(EngineCore_1 pid=3384) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed context <zmq.Context() at 0xfffde70b37d0>
(EngineCore_1 pid=3384) w.worker_response_mq = None
(EngineCore_1 pid=3384) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffde558ec10>
(EngineCore_1 pid=3384) w.worker_response_mq = None
(EngineCore_1 pid=3384) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed context <zmq.Context() at 0xfffde70b3a70>
(EngineCore_1 pid=3384) w.worker_response_mq = None
(VllmWorker rank=0 pid=3478) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed socket <zmq.Socket(zmq.XPUB) at 0xfffdbfb10fa0>
(VllmWorker rank=0 pid=3478) self.worker_response_mq = None
(VllmWorker rank=0 pid=3478) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdbfb0e1b0>
(VllmWorker rank=0 pid=3478) self.worker_response_mq = None
(VllmWorker rank=1 pid=3617) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:431: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffdc413d550>
(VllmWorker rank=1 pid=3617) self.rpc_broadcast_mq = None
(VllmWorker rank=1 pid=3617) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:431: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdc4135f10>
(VllmWorker rank=1 pid=3617) self.rpc_broadcast_mq = None
(VllmWorker rank=1 pid=3617) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed socket <zmq.Socket(zmq.XPUB) at 0xfffdc413d5c0>
(VllmWorker rank=1 pid=3617) self.worker_response_mq = None
(VllmWorker rank=1 pid=3617) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdc41360f0>
(VllmWorker rank=1 pid=3617) self.worker_response_mq = None
(VllmWorker rank=0 pid=3478) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed socket <zmq.Socket(zmq.XPUB) at 0xfffd62939cc0>
(VllmWorker rank=0 pid=3478) self.mq_broadcaster = None
(VllmWorker rank=0 pid=3478) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed context <zmq.Context() at 0xfffd629a1cd0>
(VllmWorker rank=0 pid=3478) self.mq_broadcaster = None
(VllmWorker rank=2 pid=3852) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:431: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffde5c752b0>
(VllmWorker rank=3 pid=4393) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:431: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffdd14d5780>
(VllmWorker rank=2 pid=3852) self.rpc_broadcast_mq = None
(VllmWorker rank=3 pid=4393) self.rpc_broadcast_mq = None
(VllmWorker rank=3 pid=4393) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:431: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdd14d1eb0>
(VllmWorker rank=2 pid=3852) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:431: ResourceWarning: Unclosed context <zmq.Context() at 0xfffde5c71fd0>
(VllmWorker rank=3 pid=4393) self.rpc_broadcast_mq = None
(VllmWorker rank=2 pid=3852) self.rpc_broadcast_mq = None
(VllmWorker rank=1 pid=3617) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffd53e42200>
(VllmWorker rank=1 pid=3617) self.mq_broadcaster = None
(VllmWorker rank=1 pid=3617) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed context <zmq.Context() at 0xfffd53ea9d90>
(VllmWorker rank=1 pid=3617) self.mq_broadcaster = None
(VllmWorker rank=3 pid=4393) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed socket <zmq.Socket(zmq.XPUB) at 0xfffdd14d57f0>
(VllmWorker rank=3 pid=4393) self.worker_response_mq = None
(VllmWorker rank=2 pid=3852) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed socket <zmq.Socket(zmq.XPUB) at 0xfffde5c75320>
(VllmWorker rank=2 pid=3852) self.worker_response_mq = None
(VllmWorker rank=3 pid=4393) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdd14d2090>
(VllmWorker rank=3 pid=4393) self.worker_response_mq = None
(VllmWorker rank=2 pid=3852) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed context <zmq.Context() at 0xfffde5c721b0>
(VllmWorker rank=2 pid=3852) self.worker_response_mq = None
(VllmWorker rank=3 pid=4393) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffd7433a200>
(VllmWorker rank=3 pid=4393) self.mq_broadcaster = None
(VllmWorker rank=3 pid=4393) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed context <zmq.Context() at 0xfffd743435f0>
(VllmWorker rank=3 pid=4393) self.mq_broadcaster = None
(VllmWorker rank=2 pid=3852) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffd79a2dda0>
(VllmWorker rank=2 pid=3852) self.mq_broadcaster = None
(VllmWorker rank=2 pid=3852) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed context <zmq.Context() at 0xfffd79a919d0>
(VllmWorker rank=2 pid=3852) self.mq_broadcaster = None
(VllmWorker rank=0 pid=3479) /usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py:-1: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffdd3515240>
(VllmWorker rank=0 pid=3479) /usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py:-1: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdd3511eb0>
(VllmWorker rank=1 pid=3616) /usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py:-1: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffdd6d79710>
(VllmWorker rank=1 pid=3616) /usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py:-1: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdd6d75eb0>
(VllmWorker rank=3 pid=4387) /usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py:-1: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffdf2e332a0>
(VllmWorker rank=3 pid=4387) /usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py:-1: ResourceWarning: Unclosed context <zmq.Context() at 0xfffde6bddeb0>
(VllmWorker rank=2 pid=3786) /usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py:-1: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffdd7597bd0>
(VllmWorker rank=2 pid=3786) /usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py:-1: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdccb61eb0>
(VllmWorker rank=0 pid=3478) /usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py:-1: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffdbfb10f30>
(VllmWorker rank=0 pid=3478) /usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py:-1: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdbfb0dfd0>
(EngineCore_0 pid=3249) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:266: ResourceWarning: Unclosed socket <zmq.Socket(zmq.XPUB) at 0xfffdeb9b27b0>
(EngineCore_0 pid=3249) self.rpc_broadcast_mq = None
(EngineCore_0 pid=3249) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:266: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdef022d50>
(EngineCore_0 pid=3249) self.rpc_broadcast_mq = None
(EngineCore_1 pid=3384) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:266: ResourceWarning: Unclosed socket <zmq.Socket(zmq.XPUB) at 0xfffde563a970>
(EngineCore_1 pid=3384) self.rpc_broadcast_mq = None
(EngineCore_1 pid=3384) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:266: ResourceWarning: Unclosed context <zmq.Context() at 0xfffde8c06d50>
(EngineCore_1 pid=3384) self.rpc_broadcast_mq = None
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [3104]
`
Beg for some help, Thanks !
@matthewygf

[matthewygf]

@StevenBrown008 Thanks again for the detail trace, can you double check whether disabling the AscendScheduler would work, i.e. use the default vLLM V1 scheduler instead.

This could be due to the check where AscendScheduler introduce in v0.9.2 https://github.com/vllm-project/vllm-ascend/blob/b5b7e0ecc765adfec4ec4efdd14bfc7cce715e23/vllm_ascend/core/scheduler.py#L200

Where vLLM Scheduler should always update connector state to avoid request not being found in unfinished.

[StevenBrown008]

Thanks for your guidance. Yeah, I have tried. If I continue remove the confige item "--additional-config '{"ascend_scheduler_config":{"enabled": true}}" ,there will be another different error like this:

INFO: Started server process [9828]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO 09-09 23:55:45 [logger.py:43] Received request cmpl-c17b24bdcec745cf9acfb46dc47e3014-0: prompt: 'The future of AI is', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.0, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=50, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: [0, 671, 3988, 294, 7703, 344], prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None.
INFO 09-09 23:55:45 [async_llm.py:270] Added request cmpl-c17b24bdcec745cf9acfb46dc47e3014-0.
(EngineCore_0 pid=9973) [2025-09-09 23:55:45,832] LMCache INFO: Reqid: cmpl-c17b24bdcec745cf9acfb46dc47e3014-0, Total tokens 6, LMCache hit tokens: 0, need to load: 0 (vllm_v1_adapter.py:803:lmcache.integration.vllm.vllm_v1_adapter)
(EngineCore_0 pid=9973) ('Warning: torch.save with "_use_new_zipfile_serialization = False" is not recommended for npu tensor, which may bring unexpected errors and hopefully set "_use_new_zipfile_serialization = True"', 'if it is necessary to use this, please convert the npu tensor to cpu tensor for saving')
(VllmWorker rank=1 pid=10338) [2025-09-09 23:55:45,999] LMCache INFO: Post-initializing LMCacheEngine (cache_engine.py:148:lmcache.v1.cache_engine)
(VllmWorker rank=2 pid=10513) [2025-09-09 23:55:46,126] LMCache INFO: Post-initializing LMCacheEngine (cache_engine.py:148:lmcache.v1.cache_engine)
(VllmWorker rank=0 pid=10203) [2025-09-09 23:55:46,307] LMCache INFO: Post-initializing LMCacheEngine (cache_engine.py:148:lmcache.v1.cache_engine)
(VllmWorker rank=3 pid=11117) [2025-09-09 23:55:46,315] LMCache INFO: Post-initializing LMCacheEngine (cache_engine.py:148:lmcache.v1.cache_engine)
(VllmWorker rank=2 pid=10513) [2025-09-09 23:55:51,106] LMCache INFO: Storing KV cache for 6 out of 6 tokens (skip_leading_tokens=0) for request cmpl-c17b24bdcec745cf9acfb46dc47e3014-0 (vllm_v1_adapter.py:709:lmcache.integration.vllm.vllm_v1_adapter)
(VllmWorker rank=0 pid=10203) [2025-09-09 23:55:51,106] LMCache INFO: Storing KV cache for 6 out of 6 tokens (skip_leading_tokens=0) for request cmpl-c17b24bdcec745cf9acfb46dc47e3014-0 (vllm_v1_adapter.py:709:lmcache.integration.vllm.vllm_v1_adapter)
(VllmWorker rank=1 pid=10338) [2025-09-09 23:55:51,106] LMCache INFO: Storing KV cache for 6 out of 6 tokens (skip_leading_tokens=0) for request cmpl-c17b24bdcec745cf9acfb46dc47e3014-0 (vllm_v1_adapter.py:709:lmcache.integration.vllm.vllm_v1_adapter)
(VllmWorker rank=3 pid=11117) [2025-09-09 23:55:51,106] LMCache INFO: Storing KV cache for 6 out of 6 tokens (skip_leading_tokens=0) for request cmpl-c17b24bdcec745cf9acfb46dc47e3014-0 (vllm_v1_adapter.py:709:lmcache.integration.vllm.vllm_v1_adapter)
(VllmWorker rank=1 pid=10338) [2025-09-09 23:55:51,108] LMCache INFO: Stored 6 out of total 6 tokens. size: 0.0004 gb, cost 1.2762 ms, throughput: 0.3077 GB/s; offload_time: 1.2293 ms, put_time: 0.0469 ms (cache_engine.py:251:lmcache.v1.cache_engine)
(VllmWorker rank=0 pid=10203) [2025-09-09 23:55:51,108] LMCache INFO: Stored 6 out of total 6 tokens. size: 0.0004 gb, cost 1.2792 ms, throughput: 0.3070 GB/s; offload_time: 1.2269 ms, put_time: 0.0522 ms (cache_engine.py:251:lmcache.v1.cache_engine)
(VllmWorker rank=2 pid=10513) [2025-09-09 23:55:51,108] LMCache INFO: Stored 6 out of total 6 tokens. size: 0.0004 gb, cost 1.4293 ms, throughput: 0.2747 GB/s; offload_time: 1.3604 ms, put_time: 0.0690 ms (cache_engine.py:251:lmcache.v1.cache_engine)
(VllmWorker rank=3 pid=11117) [2025-09-09 23:55:51,108] LMCache INFO: Stored 6 out of total 6 tokens. size: 0.0004 gb, cost 1.4451 ms, throughput: 0.2717 GB/s; offload_time: 1.3920 ms, put_time: 0.0532 ms (cache_engine.py:251:lmcache.v1.cache_engine)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] WorkerProc hit an exception.
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] output = func(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 217, in execute_model
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] output = self.model_runner.execute_model(scheduler_output,
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1421, in execute_model
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] num_scheduled_tokens_np, finished_sending, finished_recving) = (self._process_reqs(
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1139, in _process_reqs
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] hidden_states = self.model(
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/models/deepseek_v2.py", line 982, in forward
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] hidden_states = self.model(input_ids, positions, kv_caches,
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/models/deepseek_v2.py", line 832, in forward
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] hidden_states, residual = layer(
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/models/deepseek_v2.py", line 738, in forward
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] hidden_states = self.mlp(hidden_states,
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/models/deepseek_v2.py", line 397, in forward
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] experts_hidden_states = self.experts(
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/ops/fused_moe.py", line 1361, in forward
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] e_hidden_states = self.quant_method.apply(
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/quantization/quant_config.py", line 312, in apply
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self.quant_method.apply(
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/quantization/w8a8_dynamic.py", line 769, in apply
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return fused_experts_with_mc2(
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/quantization/w8a8_dynamic.py", line 161, in fused_experts_with_mc2
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] output = torch_npu.npu_moe_distribute_dispatch(**kwargs_mc2)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/_ops.py", line 1116, in call
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._op(*args, **(kwargs or {}))
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] RuntimeError: call aclnnMoeDistributeDispatchV2 failed, detail:
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] output = func(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 217, in execute_model
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] output = self.model_runner.execute_model(scheduler_output,
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1421, in execute_model
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] num_scheduled_tokens_np, finished_sending, finished_recving) = (self._process_reqs(
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1139, in _process_reqs
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] hidden_states = self.model(
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/models/deepseek_v2.py", line 982, in forward
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] hidden_states = self.model(input_ids, positions, kv_caches,
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] WorkerProc hit an exception.
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] output = func(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 217, in execute_model
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] output = self.model_runner.execute_model(scheduler_output,
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/models/deepseek_v2.py", line 832, in forward
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] hidden_states, residual = layer(
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1421, in execute_model
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] num_scheduled_tokens_np, finished_sending, finished_recving) = (self._process_reqs(
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1139, in _process_reqs
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] hidden_states = self.model(
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/models/deepseek_v2.py", line 738, in forward
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] hidden_states = self.mlp(hidden_states,
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/models/deepseek_v2.py", line 982, in forward
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] hidden_states = self.model(input_ids, positions, kv_caches,
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/models/deepseek_v2.py", line 397, in forward
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] experts_hidden_states = self.experts(
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/models/deepseek_v2.py", line 832, in forward
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] hidden_states, residual = layer(
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/ops/fused_moe.py", line 1361, in forward
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] e_hidden_states = self.quant_method.apply(
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/models/deepseek_v2.py", line 738, in forward
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/quantization/quant_config.py", line 312, in apply
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] hidden_states = self.mlp(hidden_states,
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self.quant_method.apply(
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/quantization/w8a8_dynamic.py", line 769, in apply
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return fused_experts_with_mc2(
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/quantization/w8a8_dynamic.py", line 161, in fused_experts_with_mc2
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] output = torch_npu.npu_moe_distribute_dispatch(**kwargs_mc2)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/models/deepseek_v2.py", line 397, in forward
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/_ops.py", line 1116, in call
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] experts_hidden_states = self.experts(
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._op(*args, **(kwargs or {}))
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522] RuntimeError: call aclnnMoeDistributeDispatchV2 failed, detail:
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=10203) ERROR 09-09 23:55:51 [multiproc_executor.py:522]
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/ops/fused_moe.py", line 1361, in forward
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] e_hidden_states = self.quant_method.apply(
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/quantization/quant_config.py", line 312, in apply
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self.quant_method.apply(
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/quantization/w8a8_dynamic.py", line 769, in apply
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return fused_experts_with_mc2(
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/quantization/w8a8_dynamic.py", line 161, in fused_experts_with_mc2
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] output = torch_npu.npu_moe_distribute_dispatch(**kwargs_mc2)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/_ops.py", line 1116, in call
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._op(*args, **(kwargs or {}))
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] RuntimeError: call aclnnMoeDistributeDispatchV2 failed, detail:
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] output = func(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 217, in execute_model
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] output = self.model_runner.execute_model(scheduler_output,
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1421, in execute_model
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] num_scheduled_tokens_np, finished_sending, finished_recving) = (self._process_reqs(
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1139, in _process_reqs
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] hidden_states = self.model(
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/models/deepseek_v2.py", line 982, in forward
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] hidden_states = self.model(input_ids, positions, kv_caches,
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/models/deepseek_v2.py", line 832, in forward
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] hidden_states, residual = layer(
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/models/deepseek_v2.py", line 738, in forward
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] hidden_states = self.mlp(hidden_states,
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/models/deepseek_v2.py", line 397, in forward
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] experts_hidden_states = self.experts(
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/ops/fused_moe.py", line 1361, in forward
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] e_hidden_states = self.quant_method.apply(
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/quantization/quant_config.py", line 312, in apply
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self.quant_method.apply(
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/quantization/w8a8_dynamic.py", line 769, in apply
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return fused_experts_with_mc2(
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/workspace/vllm-ascend/vllm_ascend/quantization/w8a8_dynamic.py", line 161, in fused_experts_with_mc2
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] output = torch_npu.npu_moe_distribute_dispatch(**kwargs_mc2)
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/_ops.py", line 1116, in call
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] return self._op(*args, **(kwargs or {}))
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522] RuntimeError: call aclnnMoeDistributeDispatchV2 failed, detail:
(VllmWorker rank=2 pid=10513) ERROR 09-09 23:55:51 [multiproc_executor.py:522]
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.9.2) with config: model='/data/models/DeepSeek-V3-W8A8', speculative_config=None, tokenizer='/data/models/DeepSeek-V3-W8A8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=ascend, enforce_eager=True, kv_cache_dtype=auto, device_config=npu, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=1024, served_model_name=deepseek_v3, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":["all"],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":0,"local_cache_dir":null},
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[], scheduled_cached_reqs=CachedRequestData(req_ids=['cmpl-c17b24bdcec745cf9acfb46dc47e3014-0'], resumed_from_preemption=[false], new_token_ids=[], new_block_ids=[[[]]], num_computed_tokens=[6]), num_scheduled_tokens={cmpl-c17b24bdcec745cf9acfb46dc47e3014-0: 1}, total_num_scheduled_tokens=1, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[1], finished_req_ids=[], free_encoder_input_ids=[], structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=LMCacheConnectorMetadata(requests=[], lookup_requests_in_step=[]))
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, kv_cache_usage=0.006329113924050667, prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0), spec_decoding_stats=None, num_corrupted_reqs=0)
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] EngineCore encountered a fatal error.
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] Traceback (most recent call last):
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] File "/workspace/vllm/vllm/v1/engine/core.py", line 579, in run_engine_core
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] engine_core.run_busy_loop()
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] File "/workspace/vllm/vllm/v1/engine/core.py", line 938, in run_busy_loop
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] executed = self._process_engine_step()
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] File "/workspace/vllm/vllm/v1/engine/core.py", line 631, in _process_engine_step
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] outputs, model_executed = self.step_fn()
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] ^^^^^^^^^^^^^^
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] File "/workspace/vllm/vllm/v1/engine/core.py", line 235, in step
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] model_output = self.execute_model(scheduler_output)
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] File "/workspace/vllm/vllm/v1/engine/core.py", line 221, in execute_model
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] raise err
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] File "/workspace/vllm/vllm/v1/engine/core.py", line 212, in execute_model
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] return self.model_executor.execute_model(scheduler_output)
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 158, in execute_model
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] (output, ) = self.collective_rpc(
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] ^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 215, in collective_rpc
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] result = get_response(w, dequeue_timeout)
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 202, in get_response
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] raise RuntimeError(
(EngineCore_0 pid=9973) ERROR 09-09 23:55:51 [core.py:588] RuntimeError: Worker failed with error 'call aclnnMoeDistributeDispatchV2 failed, detail:', please check the stack trace above for the root cause
(EngineCore_0 pid=9973) /workspace/vllm/vllm/v1/engine/core.py:766: ResourceWarning: Destroying context with unclosed socket <zmq.Socket(zmq.PUSH) at 0xfffc90d018d0>
(EngineCore_0 pid=9973) with ExitStack() as stack, zmq.Context() as ctx:
(EngineCore_0 pid=9973) /workspace/vllm/vllm/v1/engine/core.py:766: ResourceWarning: Destroying context with unclosed socket <zmq.Socket(zmq.PUSH) at 0xfffc90d017f0>
(EngineCore_0 pid=9973) with ExitStack() as stack, zmq.Context() as ctx:
(EngineCore_0 pid=9973) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffdd81db460>
(EngineCore_0 pid=9973) w.worker_response_mq = None
(EngineCore_0 pid=9973) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdd7fb7650>
(EngineCore_0 pid=9973) w.worker_response_mq = None
(EngineCore_0 pid=9973) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffdd7cd2120>
(EngineCore_0 pid=9973) w.worker_response_mq = None
(EngineCore_0 pid=9973) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdd7fb7410>
(EngineCore_0 pid=9973) w.worker_response_mq = None
ERROR 09-09 23:55:51 [async_llm.py:419] AsyncLLM output_handler failed.
ERROR 09-09 23:55:51 [async_llm.py:419] Traceback (most recent call last):
ERROR 09-09 23:55:51 [async_llm.py:419] File "/workspace/vllm/vllm/v1/engine/async_llm.py", line 378, in output_handler
ERROR 09-09 23:55:51 [async_llm.py:419] outputs = await engine_core.get_output_async()
ERROR 09-09 23:55:51 [async_llm.py:419] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 09-09 23:55:51 [async_llm.py:419] File "/workspace/vllm/vllm/v1/engine/core_client.py", line 740, in get_output_async
ERROR 09-09 23:55:51 [async_llm.py:419] raise self._format_exception(outputs) from None
ERROR 09-09 23:55:51 [async_llm.py:419] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
INFO 09-09 23:55:51 [async_llm.py:345] Request cmpl-c17b24bdcec745cf9acfb46dc47e3014-0 failed (engine dead).
(EngineCore_0 pid=9973) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffdd6536430>
(EngineCore_0 pid=9973) w.worker_response_mq = None
(EngineCore_0 pid=9973) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdd7fb73b0>
(EngineCore_0 pid=9973) w.worker_response_mq = None
INFO: 127.0.0.1:41718 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error
(EngineCore_0 pid=9973) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffdd9aea350>
(EngineCore_0 pid=9973) w.worker_response_mq = None
(EngineCore_0 pid=9973) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:263: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdd7fb7530>
(EngineCore_0 pid=9973) w.worker_response_mq = None
(VllmWorker rank=0 pid=10203) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed socket <zmq.Socket(zmq.XPUB) at 0xfffdca130ad0>
(VllmWorker rank=0 pid=10203) self.worker_response_mq = None
(VllmWorker rank=0 pid=10203) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdca12dd90>
(VllmWorker rank=0 pid=10203) self.worker_response_mq = None
(VllmWorker rank=1 pid=10338) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:431: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffdd94d10f0>
(VllmWorker rank=1 pid=10338) self.rpc_broadcast_mq = None
(VllmWorker rank=1 pid=10338) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:431: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdd94c9a90>
(VllmWorker rank=1 pid=10338) self.rpc_broadcast_mq = None
(VllmWorker rank=1 pid=10338) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed socket <zmq.Socket(zmq.XPUB) at 0xfffdd94d1160>
(VllmWorker rank=1 pid=10338) self.worker_response_mq = None
(VllmWorker rank=1 pid=10338) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdd94c9c70>
(VllmWorker rank=1 pid=10338) self.worker_response_mq = None
(VllmWorker rank=0 pid=10203) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed socket <zmq.Socket(zmq.XPUB) at 0xfffd6cfdd780>
(VllmWorker rank=0 pid=10203) self.mq_broadcaster = None
(VllmWorker rank=0 pid=10203) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed context <zmq.Context() at 0xfffd6d03d790>
(VllmWorker rank=0 pid=10203) self.mq_broadcaster = None
(VllmWorker rank=2 pid=10513) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed socket <zmq.Socket(zmq.XPUB) at 0xfffdb49dca60>
(VllmWorker rank=2 pid=10513) self.worker_response_mq = None
(VllmWorker rank=3 pid=11117) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:431: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffde9fd5390>
(VllmWorker rank=3 pid=11117) self.rpc_broadcast_mq = None
(VllmWorker rank=2 pid=10513) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed context <zmq.Context() at 0xfffdb49cddf0>
(VllmWorker rank=1 pid=10338) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffd6d265d30>
(VllmWorker rank=2 pid=10513) self.worker_response_mq = None
(VllmWorker rank=1 pid=10338) self.mq_broadcaster = None
(VllmWorker rank=3 pid=11117) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:431: ResourceWarning: Unclosed context <zmq.Context() at 0xfffde9fcda90>
(VllmWorker rank=3 pid=11117) self.rpc_broadcast_mq = None
(VllmWorker rank=1 pid=10338) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed context <zmq.Context() at 0xfffd6d2dd7f0>
(VllmWorker rank=1 pid=10338) self.mq_broadcaster = None
(VllmWorker rank=3 pid=11117) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed socket <zmq.Socket(zmq.XPUB) at 0xfffde9fd5400>
(VllmWorker rank=3 pid=11117) self.worker_response_mq = None
(VllmWorker rank=3 pid=11117) /workspace/vllm/vllm/v1/executor/multiproc_executor.py:432: ResourceWarning: Unclosed context <zmq.Context() at 0xfffde9fcdc70>
(VllmWorker rank=3 pid=11117) self.worker_response_mq = None
(VllmWorker rank=2 pid=10513) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffd487898d0>
(VllmWorker rank=2 pid=10513) self.mq_broadcaster = None
(VllmWorker rank=2 pid=10513) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed context <zmq.Context() at 0xfffd487ed310>
(VllmWorker rank=2 pid=10513) self.mq_broadcaster = None
(VllmWorker rank=3 pid=11117) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed socket <zmq.Socket(zmq.SUB) at 0xfffd7dd6dd30>
(VllmWorker rank=3 pid=11117) self.mq_broadcaster = None
(VllmWorker rank=3 pid=11117) /workspace/vllm/vllm/distributed/parallel_state.py:781: ResourceWarning: Unclosed context <zmq.Context() at 0xfffd7dd73290>
(VllmWorker rank=3 pid=11117) self.mq_broadcaster = None
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [9828]

@matthewygf

[chloroethylene]

it seems to be an error from "call aclnnMoeDistributeDispatchV2 failed", cc @wangxiyuan , @Yikun

[matthewygf]

@StevenBrown008 Is it possible to check whether there are important error messages in your $HOME/ascend/log/debug/plog/ as described here https://www.hiascend.com/document/detail/zh/canncommercial/82RC1/maintenref/logreference/logreference_0002.html .

Furthermore, our colleagues mentioned it is possible that vllm-ascend v0.9.1-dev branch could run the model. We have not ran it with LMCache-Ascend, but it is possible that this could help immediately , otherwise we will test it internally.

[StevenBrown008]

OK, Thanks for your suggestion, I 'll try it.

[matthewygf]

@StevenBrown008 just checking, any updates?