Skip to content

Unify server-side and model-side Config #2862

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 15 commits into
base: develop
Choose a base branch
from
Open

Unify server-side and model-side Config #2862

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
6ab62f1
unify config
YuanRisheng Jul 16, 2025
d7fbd67
resolve conflict
YuanRisheng Jul 16, 2025
13a1604
modify parallel_config
YuanRisheng Jul 16, 2025
914e9b7
resolve conflicgt
YuanRisheng Jul 16, 2025
0d153b8
delete code
YuanRisheng Jul 16, 2025
277929e
delete code
YuanRisheng Jul 16, 2025
b7781d3
fix bug
YuanRisheng Jul 21, 2025
449f025
resolve conflict
YuanRisheng Jul 21, 2025
c788f2e
resolve conflict
YuanRisheng Jul 21, 2025
61a3525
fix vl
YuanRisheng Jul 21, 2025
20a12c5
fix rl
YuanRisheng Jul 22, 2025
8f62cbf
resolve conflict
YuanRisheng Jul 22, 2025
2805b71
fix qwen2
YuanRisheng Jul 23, 2025
68624e7
resolve conflict
YuanRisheng Jul 23, 2025
0e2d220
fix mm
YuanRisheng Jul 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion fastdeploy/cache_manager/cache_transfer_manager.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
import paddle

from fastdeploy.cache_manager.cache_data import CacheStatus
from fastdeploy.engine.config import SpeculativeConfig
from fastdeploy.config import SpeculativeConfig
from fastdeploy.inter_communicator import EngineCacheQueue, IPCSignal
from fastdeploy.model_executor.ops.gpu import (
cuda_host_alloc,
Expand Down
2 changes: 1 addition & 1 deletion fastdeploy/cache_manager/prefix_cache_manager.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ def launch_cache_manager(
+ f" --device_id {int(device_ids[i])}"
+ f" --rank {i}"
+ f" --splitwise_role {self.splitwise_role}"
+ f" --num_layers {cache_config.model_cfg.num_layers}"
+ f" --num_layers {cache_config.model_cfg.num_hidden_layers}"
+ f" --head_dim {cache_config.model_cfg.head_dim}"
+ f" --kv_num_head {kv_num_head}"
+ f" --mp_num {tensor_parallel_size}"
Expand Down
Loading
Loading