[Feature][1/2] Impl the connector based on the llmdatadist for v1

[jianzs]

This PR implements the connector functionality for NPU based on LLMDataDist, building upon the connector API merged in vLLM v1. (vllm-project/vllm#15960) We've successfully tested various scenarios in offline environments:

• Single-machine: Verified 2P2D testing with dense models (Llama) and MoE models (DeepSeek v2 Lite)
• Two-machine: Completed 1P1D testing with DeepSeek R1 W8A8

Key implementation aspects include:

Cross-machine PD: LLMDataDist requires NPU device IP for connection establishment. Our approach utilizes a global rank table (JSON) on each machine containing:

• Unique server IDs
• IP addresses and device IDs for each card
• Server ID specification in connector extra config at startup for instance information retrieval

nPmD: Given that the community's nPmD design, particularly the router component API, is still evolving, we've implemented a solution using a meta server component (to be provided separately) that:

• Records prefill completion details (device and dp rank information)
• Responds to decode node queries with prefill node locations
• Enables decode nodes to retrieve data from appropriate prefill nodes

We propose initially merging the 1P1D implementation, where the global rank table contains information for two nodes, allowing direct prefill node identification. The nPmD implementation can be refined and merged following community discussion.

Todo:

• Implement 1P1D (one prefill, one decode) configuration support
• Add sample script for automatic global rank table generation
• Document global rank table format specifications
• Provide user guide for PD (Prefill-Decode) functionality
• Add test cases

re #448

---

Note:
A minor modification to vLLM's codebase is required to run this example successfully. The patch enables the scheduler process to locate the appropriate connector class by importing the necessary module.

The change should be made in vllm/v1/core/sched/scheduler.py, adding an import statement for vllm_ascend.distributed.

This is a temporary solution, and we need to implement a more elegant module discovery mechanism.

diff --git a/vllm/v1/core/sched/scheduler.py b/vllm/v1/core/sched/scheduler.py
index 69e7cc8ee..b15525971 100644
--- a/vllm/v1/core/sched/scheduler.py
+++ b/vllm/v1/core/sched/scheduler.py
@@ -31,6 +31,8 @@ from vllm.v1.structured_output import StructuredOutputManager
 
 logger = init_logger(__name__)
 
+# TODO(jianzs): Find the suitable place to put this.
+import vllm_ascend.distributed
 
 class Scheduler(SchedulerInterface):
 

---

Limits:

1. We use a hash function (string_to_int64_hash) to convert request IDs to datadist request IDs. This conversion is lossy, potentially creating duplicate IDs, leading to duplicate CacheKeys and allocate_cache failures.

[jianzs]

Measure the time it takes for KV transfers at different sequence lengths.

Environment:

1. Two machines
2. 1P1D
3. DeepSeek R1 W8A8
4. TP=16，DP=1

Stacked charts show higher times than overall charts because each stage of measurement has NPU synchronization before and after, especially extract kv, scatter update, and inject kv, all of which synchronize on each layer, resulting in significant host overhead.

• load kv = pull cache + inject kv
• save kv = extract kv + scatter update

[Kevin-XiongC]

Hi, I tried this PR. But it seems like there exists a precision issue here. The prompt is "What is the largest animal in the world?" with temperature == 0 using qwen2.5 0.5b.

using PD disaggregated

{"id":"cmpl-request2005","object":"text_completion","created":1746675979,"model":"/deepseek/Qwen2.5-0.5B-Instruct","choices":[{"index":0,"text":" T of the\nTheorem's Conclusion.\nIn this case, we have to find the value that will make the equation true. We can do so by substituting the given values into the original equation","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":10,"total_tokens":50,"completion_tokens":40,"prompt_tokens_details":null}}

normal

{"id":"cmpl-request16718","object":"text_completion","created":1746676100,"model":"/deepseek/Qwen2.5-0.5B-Instruct","choices":[{"index":0,"text":" The largest animal in the world, as of 2021, was the blue whale. Blue whales can grow up to weigh over 150 tons and be up to 30","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":9,"total_tokens":49,"completion_tokens":40,"prompt_tokens_details":null}}

[jianzs]

Hi, I tried this PR. But it seems like there exists a precision issue here. The prompt is "What is the largest animal in the world?" with temperature == 0 using qwen2.5 0.5b.

using PD disaggregated

{"id":"cmpl-request2005","object":"text_completion","created":1746675979,"model":"/deepseek/Qwen2.5-0.5B-Instruct","choices":[{"index":0,"text":" T of the\nTheorem's Conclusion.\nIn this case, we have to find the value that will make the equation true. We can do so by substituting the given values into the original equation","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":10,"total_tokens":50,"completion_tokens":40,"prompt_tokens_details":null}}

normal

{"id":"cmpl-request16718","object":"text_completion","created":1746676100,"model":"/deepseek/Qwen2.5-0.5B-Instruct","choices":[{"index":0,"text":" The largest animal in the world, as of 2021, was the blue whale. Blue whales can grow up to weigh over 150 tons and be up to 30","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":9,"total_tokens":49,"completion_tokens":40,"prompt_tokens_details":null}}

Thank you for reporting this issue. I've tested with DeepSeek v2 Lite and Llama2 7B, and observed that:

• The first response is incorrect
• Second and subsequent responses are as expected

Could you confirm if you're seeing incorrect responses consistently in your tests? And are your configurations, including parallelism, identical in both the disaggregated and singlestone environments?

[Kevin-XiongC]

Yes, I see this consistently.

I used the same sh script 'disaggregated_prefill_multi_prefill.sh' but changed its tp to 1 and model to qwen2.5 0.5b.

and as the aggregated one. I've used the default settings below.

 python -m vllm.entrypoints.openai.api_server --model Qwen2.5-0.5B-Instruct

as for the ranktable

{
    "server_group_list":[
        {
            "group_id": "0",
            "version": "1.0",
            "server_count": "1",
            "server_list": [
                {
                    "server_ip": "10.172.116.166",
                    "container_ip": "10.172.116.166"
                }
            ],
            
            "status": "completed"
        },
        {
            "group_id": "1",
            "version": "1.0",
            "server_count": "1",
            "server_list": [
                {
                    "server_ip": "10.172.116.166",
                    "server_id": "server-0",
                    "device": [
                        {
                            "device_id": "0",
                            "device_ip": "172.22.17.1",
                            "rank_id": "0"
                        }
                    ],
                    "container_ip": "10.172.116.166"
                }
            ],
            
            "status": "completed"
        },
        {
            "group_id": "2",
            "version": "1.0",
            "server_count": "1",
            "server_list": [
                {
                    "server_ip": "10.172.116.166",
                    "server_id": "server-1",
                    "device": [
                        {
                            "device_id": "4",
                            "device_ip": "172.22.17.5",
                            "rank_id": "0"
                        }
                    ],
                    "container_ip": "10.172.116.166"
                }
            ],
            
            "status": "completed"
        }
    ]
}

[jianzs]

Yes, I see this consistently.

I used the same sh script 'disaggregated_prefill_multi_prefill.sh' but changed its tp to 1 and model to qwen2.5 0.5b.

and as the aggregated one. I've used the default settings below.

 python -m vllm.entrypoints.openai.api_server --model Qwen2.5-0.5B-Instruct

as for the ranktable

{
    "server_group_list":[
        {
            "group_id": "0",
            "version": "1.0",
            "server_count": "1",
            "server_list": [
                {
                    "server_ip": "10.172.116.166",
                    "container_ip": "10.172.116.166"
                }
            ],
            
            "status": "completed"
        },
        {
            "group_id": "1",
            "version": "1.0",
            "server_count": "1",
            "server_list": [
                {
                    "server_ip": "10.172.116.166",
                    "server_id": "server-0",
                    "device": [
                        {
                            "device_id": "0",
                            "device_ip": "172.22.17.1",
                            "rank_id": "0"
                        }
                    ],
                    "container_ip": "10.172.116.166"
                }
            ],
            
            "status": "completed"
        },
        {
            "group_id": "2",
            "version": "1.0",
            "server_count": "1",
            "server_list": [
                {
                    "server_ip": "10.172.116.166",
                    "server_id": "server-1",
                    "device": [
                        {
                            "device_id": "4",
                            "device_ip": "172.22.17.5",
                            "rank_id": "0"
                        }
                    ],
                    "container_ip": "10.172.116.166"
                }
            ],
            
            "status": "completed"
        }
    ]
}

I fixed an accuracy issue. Please try again.

[Kevin-XiongC]

Yes, I see this consistently.
I used the same sh script 'disaggregated_prefill_multi_prefill.sh' but changed its tp to 1 and model to qwen2.5 0.5b.
and as the aggregated one. I've used the default settings below.

 python -m vllm.entrypoints.openai.api_server --model Qwen2.5-0.5B-Instruct

as for the ranktable

{
    "server_group_list":[
        {
            "group_id": "0",
            "version": "1.0",
            "server_count": "1",
            "server_list": [
                {
                    "server_ip": "10.172.116.166",
                    "container_ip": "10.172.116.166"
                }
            ],
            
            "status": "completed"
        },
        {
            "group_id": "1",
            "version": "1.0",
            "server_count": "1",
            "server_list": [
                {
                    "server_ip": "10.172.116.166",
                    "server_id": "server-0",
                    "device": [
                        {
                            "device_id": "0",
                            "device_ip": "172.22.17.1",
                            "rank_id": "0"
                        }
                    ],
                    "container_ip": "10.172.116.166"
                }
            ],
            
            "status": "completed"
        },
        {
            "group_id": "2",
            "version": "1.0",
            "server_count": "1",
            "server_list": [
                {
                    "server_ip": "10.172.116.166",
                    "server_id": "server-1",
                    "device": [
                        {
                            "device_id": "4",
                            "device_ip": "172.22.17.5",
                            "rank_id": "0"
                        }
                    ],
                    "container_ip": "10.172.116.166"
                }
            ],
            
            "status": "completed"
        }
    ]
}

I fixed an accuracy issue. Please try again.

Hi, thx for your work. Unfortunately, it still produces inconsistent results with the 0.5B model, but when I switch to 1.5B model, the disaggregated version produces the correct one. I hope it helps

[jianzs]

Yes, I see this consistently.
I used the same sh script 'disaggregated_prefill_multi_prefill.sh' but changed its tp to 1 and model to qwen2.5 0.5b.
and as the aggregated one. I've used the default settings below.

 python -m vllm.entrypoints.openai.api_server --model Qwen2.5-0.5B-Instruct

as for the ranktable

{
    "server_group_list":[
        {
            "group_id": "0",
            "version": "1.0",
            "server_count": "1",
            "server_list": [
                {
                    "server_ip": "10.172.116.166",
                    "container_ip": "10.172.116.166"
                }
            ],
            
            "status": "completed"
        },
        {
            "group_id": "1",
            "version": "1.0",
            "server_count": "1",
            "server_list": [
                {
                    "server_ip": "10.172.116.166",
                    "server_id": "server-0",
                    "device": [
                        {
                            "device_id": "0",
                            "device_ip": "172.22.17.1",
                            "rank_id": "0"
                        }
                    ],
                    "container_ip": "10.172.116.166"
                }
            ],
            
            "status": "completed"
        },
        {
            "group_id": "2",
            "version": "1.0",
            "server_count": "1",
            "server_list": [
                {
                    "server_ip": "10.172.116.166",
                    "server_id": "server-1",
                    "device": [
                        {
                            "device_id": "4",
                            "device_ip": "172.22.17.5",
                            "rank_id": "0"
                        }
                    ],
                    "container_ip": "10.172.116.166"
                }
            ],
            
            "status": "completed"
        }
    ]
}

I fixed an accuracy issue. Please try again.

Hi, thx for your work. Unfortunately, it still produces inconsistent results with the 0.5B model, but when I switch to 1.5B model, the disaggregated version produces the correct one. I hope it helps

Thx. Fix a bug, please try again...

[Kevin-XiongC]

Yes, I see this consistently.
I used the same sh script 'disaggregated_prefill_multi_prefill.sh' but changed its tp to 1 and model to qwen2.5 0.5b.
and as the aggregated one. I've used the default settings below.

 python -m vllm.entrypoints.openai.api_server --model Qwen2.5-0.5B-Instruct

as for the ranktable

{
    "server_group_list":[
        {
            "group_id": "0",
            "version": "1.0",
            "server_count": "1",
            "server_list": [
                {
                    "server_ip": "10.172.116.166",
                    "container_ip": "10.172.116.166"
                }
            ],
            
            "status": "completed"
        },
        {
            "group_id": "1",
            "version": "1.0",
            "server_count": "1",
            "server_list": [
                {
                    "server_ip": "10.172.116.166",
                    "server_id": "server-0",
                    "device": [
                        {
                            "device_id": "0",
                            "device_ip": "172.22.17.1",
                            "rank_id": "0"
                        }
                    ],
                    "container_ip": "10.172.116.166"
                }
            ],
            
            "status": "completed"
        },
        {
            "group_id": "2",
            "version": "1.0",
            "server_count": "1",
            "server_list": [
                {
                    "server_ip": "10.172.116.166",
                    "server_id": "server-1",
                    "device": [
                        {
                            "device_id": "4",
                            "device_ip": "172.22.17.5",
                            "rank_id": "0"
                        }
                    ],
                    "container_ip": "10.172.116.166"
                }
            ],
            
            "status": "completed"
        }
    ]
}

I fixed an accuracy issue. Please try again.

Hi, thx for your work. Unfortunately, it still produces inconsistent results with the 0.5B model, but when I switch to 1.5B model, the disaggregated version produces the correct one. I hope it helps

Thx. Fix a bug, please try again...

Great! It works for me now.

[ganyi1996ppo]

Why mod the min size of the tp size of pd? Can't it just adopt the tp rank.

[jianzs]

This design originally aimed to support heterogeneous parallelism between prefill and decode phases. For scenarios where prefill TP size < decode TP size, each rank could determine its connection count using the modulo method.

However, due to current LLMDataDist constraints, decode TP size must be ≤ prefill TP size. Consequently, using either modulo operation or direct TP rank assignment achieves identical results.

[ganyi1996ppo]

What dose this kv_parallel_size do?

[jianzs]

The v0 implementation needed this, but I'm unsure if it's still necessary.

[ganyi1996ppo]

The code looks good to me in general, but I'm not very familiar with the llmdatadist. Can @whx-sjtu review this PR for some of its detail?

[whx-sjtu]

You may need to add a new member super_device_id if you want to run disaggregated-prefill on A3 super node.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.