Skip to content

Commit cef43b5

Browse files
LCAIZJfems14DreamerLeaderPz1116lizy124
authored
[Feat] A Connector that supports Mooncake store (#2913)
### What this PR does / why we need it? Added a new connector for Mooncake store integration to enable kvcache reuse in scenarios with system prompts or multi-turn dialogues. ### How was this patch tested? - vLLM version: v0.10.2 - vLLM main: vllm-project/vllm@5963b98 --------- Signed-off-by: LCAIZJ <leichao139636@163.com> Signed-off-by: fems14 <1804143737@qq.com> Co-authored-by: fems14 <1804143737@qq.com> Co-authored-by: Dreamerleader <2270923832@qq.com> Co-authored-by: Pz1116 <zpbzpb123123@gmail.com> Co-authored-by: lizy124 <1950471827@qq.com> Co-authored-by: zouyida2052 <zouyida2002@gmail.com>
1 parent 723d460 commit cef43b5

File tree

9 files changed

+2033
-9
lines changed

9 files changed

+2033
-9
lines changed

‎examples/disaggregated_prefill_v1/mooncake_connector_store_deployment_guide.md‎

Lines changed: 266 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,266 @@
1+
# Mooncacke Store Deployment Guide
2+
3+
## Environmental Dependencies
4+
5+
* Software:
6+
* Python >= 3.9, < 3.12
7+
* CANN >= 8.2.rc1
8+
* PyTorch >= 2.7.1, torch-npu >= 2.7.1.dev20250724
9+
* vLLM：main branch
10+
* vLLM-Ascend：main branch
11+
* Mooncake：[AscendTransport/Mooncake at pooling-async-memcpy](https://github.com/AscendTransport/Mooncake/tree/pooling-async-memcpy)(Currently available branch code, continuously updated.)
12+
Installation and Compilation Guide：https://github.com/AscendTransport/Mooncake/tree/pooling-async-memcpy?tab=readme-ov-file#build-and-use-binaries
13+
14+
## run mooncake master
15+
16+
### 1.Configure mooncake.json
17+
18+
The environment variable **MOONCAKE_CONFIG_PATH** is configured to the full path where mooncake.json is located.
19+
20+
```
21+
{
22+
"local_hostname": "xx.xx.xx.xx",
23+
"metadata_server": "P2PHANDSHAKE",
24+
"protocol": "ascend",
25+
"device_name": "",
26+
"master_server_address": "xx.xx.xx.xx:50088",
27+
"global_segment_size": 30000000000
28+
}
29+
```
30+
31+
**local_hostname**: Configured as the IP address of the current master node,
32+
**metadata_server**: Configured as **P2PHANDSHAKE**,
33+
**protocol:** Configured for Ascend to use Mooncake's HCCL communication,
34+
**device_name**: ""
35+
**master_server_address**: Configured with the IP and port of the master service
36+
**global_segment_size**: Expands the kvcache size registered by the PD node to the master
37+
38+
### 2. Start mooncake_master
39+
40+
Under the mooncake folder:
41+
42+
```
43+
mooncake_master --port 50088
44+
```
45+
46+
## Pooling and Prefill Decode Disaggregate Scenario
47+
48+
### 1.Run `prefill` Node and `decode` Node
49+
50+
Using MultiConnector to simultaneously utilize both p2p connectors and pooled connectors. P2P performs kv_transfer, while pooling creates a larger prefix-cache.
51+
52+
`prefill` Node：
53+
54+
```
55+
bash multi_producer.sh
56+
```
57+
58+
The content of the multi_producer.sh script:
59+
60+
```
61+
export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:$LD_LIBRARY_PATH
62+
export PYTHONPATH=$PYTHONPATH:/xxxxx/vllm
63+
export MOONCAKE_CONFIG_PATH="/xxxxxx/mooncake.json"
64+
export VLLM_USE_V1=1
65+
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3
66+
export ASCEND_TRANSPORT_PRINT=1
67+
# The upper boundary environment variable for memory swap logging is set to mooncake, where 1 indicates enabled and 0 indicates disabled.
68+
export ASCEND_AGGREGATE_ENABLE=1
69+
# The upper-level environment variable is the switch for enabling the mooncake aggregation function, where 1 means on and 0 means off.
70+
71+
python3 -m vllm.entrypoints.openai.api_server \
72+
--model /xxxxx/Qwen2.5-7B-Instruct \
73+
--port 8100 \
74+
--trust-remote-code \
75+
--enforce-eager \
76+
--no_enable_prefix_caching \
77+
--tensor-parallel-size 1 \
78+
--data-parallel-size 1 \
79+
--max-model-len 10000 \
80+
--block-size 128 \
81+
--max-num-batched-tokens 4096 \
82+
--kv-transfer-config \
83+
'{
84+
"kv_connector": "MultiConnector",
85+
"kv_role": "kv_producer",
86+
"kv_connector_extra_config": {
87+
"use_layerwise": false,
88+
"connectors": [
89+
{
90+
"kv_connector": "MooncakeConnectorV1",
91+
"kv_role": "kv_producer",
92+
"kv_port": "20001",
93+
"kv_connector_extra_config": {
94+
"prefill": {
95+
"dp_size": 1,
96+
"tp_size": 1
97+
},
98+
"decode": {
99+
"dp_size": 1,
100+
"tp_size": 1
101+
}
102+
}
103+
},
104+
{
105+
"kv_connector": "MooncakeConnectorStoreV1",
106+
"kv_role": "kv_producer",
107+
}
108+
]
109+
}
110+
}' > p.log 2>&1
111+
```
112+
113+
`decode` Node：
114+
115+
```
116+
bash multi_consumer.sh
117+
```
118+
119+
The content of multi_consumer.sh:
120+
121+
```
122+
export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:$LD_LIBRARY_PATH
123+
export PYTHONPATH=$PYTHONPATH:/xxxxx/vllm
124+
export MOONCAKE_CONFIG_PATH="/xxxxx/mooncake.json"
125+
export VLLM_USE_V1=1
126+
export ASCEND_RT_VISIBLE_DEVICES=4,5,6,7
127+
export ASCEND_TRANSPORT_PRINT=1
128+
# The upper boundary environment variable for memory swap logging is set to mooncake, where 1 indicates enabled and 0 indicates disabled.
129+
export ASCEND_AGGREGATE_ENABLE=1
130+
# The upper-level environment variable is the switch for enabling the mooncake aggregation function, where 1 means on and 0 means off.
131+
132+
python3 -m vllm.entrypoints.openai.api_server \
133+
--model /xxxxx/Qwen2.5-7B-Instruct \
134+
--port 8200 \
135+
--trust-remote-code \
136+
--enforce-eager \
137+
--no_enable_prefix_caching \
138+
--tensor-parallel-size 1 \
139+
--data-parallel-size 1 \
140+
--max-model-len 10000 \
141+
--block-size 128 \
142+
--max-num-batched-tokens 4096 \
143+
--kv-transfer-config \
144+
'{
145+
"kv_connector": "MultiConnector",
146+
"kv_role": "kv_consumer",
147+
"kv_connector_extra_config": {
148+
"use_layerwise": false,
149+
"connectors": [
150+
{
151+
"kv_connector": "MooncakeConnectorV1",
152+
"kv_role": "kv_consumer",
153+
"kv_port": "20002",
154+
"kv_connector_extra_config": {
155+
"prefill": {
156+
"dp_size": 1,
157+
"tp_size": 1
158+
},
159+
"decode": {
160+
"dp_size": 1,
161+
"tp_size": 1
162+
}
163+
}
164+
},
165+
{
166+
"kv_connector": "MooncakeConnectorStoreV1",
167+
"kv_role": "kv_consumer",
168+
}
169+
]
170+
}
171+
}' > d.log 2>&1
172+
```
173+
174+
### 2、Start proxy_server.
175+
176+
```
177+
bash proxy.sh
178+
```
179+
180+
proxy.sh content:
181+
Change localhost to your actual IP address.
182+
183+
```
184+
python vllm-ascend/examples/disaggregated_prefill_v1/load_balance_proxy_server_example.py \
185+
--host localhost\
186+
--prefiller-hosts localhost \
187+
--prefiller-ports 8100 \
188+
--decoder-hosts localhost\
189+
--decoder-ports 8200 \
190+
```
191+
192+
### 3. Run Inference
193+
194+
Configure the localhost, port, and model weight path in the command to your own settings.
195+
196+
Short question:
197+
198+
```
199+
curl -s http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{ "model": "/xxxxx/Qwen2.5-7B-Instruct", "prompt": "Hello. I have a question. The president of the United States is", "max_tokens": 200, "temperature":0.0 }'
200+
```
201+
202+
Long question:
203+
204+
```
205+
curl -s http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{ "model": "/xxxxx/Qwen2.5-7B-Instruct", "prompt": "Given the accelerating impacts of climate change—including rising sea levels, increasing frequency of extreme weather events, loss of biodiversity, and adverse effects on agriculture and human health—there is an urgent need for a robust, globally coordinated response. However, international efforts are complicated by a range of factors: economic disparities between high-income and low-income countries, differing levels of industrialization, varying access to clean energy technologies, and divergent political systems that influence climate policy implementation. In this context, how can global agreements like the Paris Accord be redesigned or strengthened to not only encourage but effectively enforce emission reduction targets? Furthermore, what mechanisms can be introduced to promote fair and transparent technology transfer, provide adequate financial support for climate adaptation in vulnerable regions, and hold nations accountable without exacerbating existing geopolitical tensions or disproportionately burdening those with historically lower emissions?", "max_tokens": 256, "temperature":0.0 }'
206+
```
207+
208+
## Pooling and Mixed Deployment Scenario
209+
210+
### 1、Run Mixed Department Script
211+
212+
The mixed script is essentially a pure pooling scenario for the P node.
213+
214+
```
215+
bash mixed_department.sh
216+
```
217+
218+
Content of mixed_department.sh:
219+
220+
```
221+
export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:$LD_LIBRARY_PATH
222+
export PYTHONPATH=$PYTHONPATH:/xxxxx/vllm
223+
export MOONCAKE_CONFIG_PATH="/xxxxxx/mooncake.json"
224+
export VLLM_USE_V1=1
225+
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3
226+
export ASCEND_TRANSPORT_PRINT=1
227+
# The upper boundary environment variable for memory swap logging is set to mooncake, where 1 indicates enabled and 0 indicates disabled.
228+
export ASCEND_AGGREGATE_ENABLE=1
229+
# The upper-level environment variable is the switch for enabling the mooncake aggregation function, where 1 means on and 0 means off.
230+
231+
python3 -m vllm.entrypoints.openai.api_server \
232+
--model /xxxxx/Qwen2.5-7B-Instruct \
233+
--port 8100 \
234+
--trust-remote-code \
235+
--enforce-eager \
236+
--no_enable_prefix_caching \
237+
--tensor-parallel-size 1 \
238+
--data-parallel-size 1 \
239+
--max-model-len 10000 \
240+
--block-size 128 \
241+
--max-num-batched-tokens 4096 \
242+
--kv-transfer-config \
243+
'{
244+
"kv_connector": "MooncakeConnectorStoreV1",
245+
"kv_role": "kv_producer",
246+
"kv_connector_extra_config": {
247+
"use_layerwise": false
248+
}
249+
}' > mix.log 2>&1
250+
```
251+
252+
### 2. Run Inference
253+
254+
Configure the localhost, port, and model weight path in the command to your own settings. The requests sent will only go to the port where the mixed deployment script is located, and there is no need to start a separate proxy.
255+
256+
Short question:
257+
258+
```
259+
curl -s http://localhost:8100/v1/completions -H "Content-Type: application/json" -d '{ "model": "/xxxxx/Qwen2.5-7B-Instruct", "prompt": "Hello. I have a question. The president of the United States is", "max_tokens": 200, "temperature":0.0 }'
260+
```
261+
262+
Long question:
263+
264+
```
265+
curl -s http://localhost:8100/v1/completions -H "Content-Type: application/json" -d '{ "model": "/xxxxx/Qwen2.5-7B-Instruct", "prompt": "Given the accelerating impacts of climate change—including rising sea levels, increasing frequency of extreme weather events, loss of biodiversity, and adverse effects on agriculture and human health—there is an urgent need for a robust, globally coordinated response. However, international efforts are complicated by a range of factors: economic disparities between high-income and low-income countries, differing levels of industrialization, varying access to clean energy technologies, and divergent political systems that influence climate policy implementation. In this context, how can global agreements like the Paris Accord be redesigned or strengthened to not only encourage but effectively enforce emission reduction targets? Furthermore, what mechanisms can be introduced to promote fair and transparent technology transfer, provide adequate financial support for climate adaptation in vulnerable regions, and hold nations accountable without exacerbating existing geopolitical tensions or disproportionately burdening those with historically lower emissions?", "max_tokens": 256, "temperature":0.0 }'
266+
```

‎vllm_ascend/distributed/__init__.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,3 +26,8 @@
2626
KVConnectorFactory.register_connector(
2727
"MooncakeConnectorV1", "vllm_ascend.distributed.mooncake_connector",
2828
"MooncakeConnector")
29+
30+
KVConnectorFactory.register_connector(
31+
"MooncakeConnectorStoreV1",
32+
"vllm_ascend.distributed.mooncake.mooncake_store_connector_v1",
33+
"MooncakeConnectorV1")

‎vllm_ascend/distributed/mooncake/__init__.py‎

Whitespace-only changes.

0 commit comments

Comments
 (0)