[CI]Update accuracy report test #1288

zhangxinyuehfad · 2025-06-18T15:01:42Z

What this PR does / why we need it?

Update accuracy report test

Add Record commit hashes and GitHub links for both vllm and vllm-ascend in accuracy reports
Add accuracy result verification checks to ensure output correctness
Creat PR via forked repository workflow

Does this PR introduce any user-facing change?

How was this patch tested?

dense-accuracy-test: https://github.com/vllm-project/vllm-ascend/actions/runs/15745619485
create pr via forked repository workflow: https://github.com/zhangxinyuehfad/vllm-ascend/actions/runs/15747013719/job/44385134080
accuracy report pr: #1292

Currently, the accuracy report used is old and needs to be merged into pr, retest, update new report, then close #1292 .

Yikun · 2025-06-19T03:49:34Z

.github/workflows/accuracy_test.yaml

        type: choice
        options:
          - main
+          - v0.9.0-dev


Suggested change

- v0.9.0-dev

- v0.9.1-dev

Yikun · 2025-06-19T04:14:38Z

.github/workflows/accuracy_report.yaml

        type: choice
        options:
          - main
+          - v0.9.0-dev


Please remove accuracy_report.yaml

Please append accuracy_report after accuracy test

Report only when dispatch

Using bot

Yikun · 2025-06-19T04:21:09Z

benchmarks/scripts/run_accuracy.py

                                                datasets=datasets)
    model = model_name.split("/")[1]
-    preamble = f"""# 🎯 {model} Accuracy Test
+    preamble = f"""# {ACCURACY_FLAG}🎯 {model}


I curious why this is not a markdown format to reduce maintainence cost

Please update hash as follow:

# 🎯 Qwen2.5-7B-Instruct **vLLM Version**: vLLM: 0.9.1 ([b6553be](https://github.com/vllm-project/vllm/commit/b6553be1bc75f046b00046a4ad7576364d03c835)) , **vLLM Ascend**: refs/pull/1288/merge([c59e3895](https://github.com/vllm-project/vllm-ascend/commit/c59e3895e227dfc07cef71f0163b6ca8ad3649a6))

Preview:

🎯 Qwen2.5-7B-Instruct

vLLM Version: vLLM: 0.9.1 (b6553be) , vLLM Ascend: refs/pull/1288/merge(c59e3895)

Yikun · 2025-06-19T04:27:20Z

benchmarks/scripts/run_accuracy.py

                   f"| {n_shot:6} "
                   f"| {metric:<6} "
-                   f"| ↑ {value:>5.4f} "
+                   f"| {value:>5.4f} "


Add ✅ ❌ here

Yikun · 2025-06-19T04:27:45Z

benchmarks/scripts/run_accuracy.py



 def main(args):
+    global ACCURACY_FLAG


Remove this.

Yikun · 2025-06-19T04:29:23Z

benchmarks/scripts/run_accuracy.py

    parser.add_argument("--torch_npu_version", type=str, required=False)
    parser.add_argument("--vllm_version", type=str, required=False)
    parser.add_argument("--cann_version", type=str, required=False)
+    parser.add_argument("--vllm_commit", type=lambda s: s[:7], required=False)


only vllm_commit and vllm_ascend_commit are needed, other var can be generate from them

Yikun · 2025-06-19T04:30:06Z

benchmarks/tests/accuracy.json

@@ -0,0 +1,28 @@
+{


Let's move this to run_accuracy to keep it simple

codecov · 2025-06-23T01:55:50Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 27.21%. Comparing base (c30ddb8) to head (9a0fc54).
⚠️ Report is 548 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1288      +/-   ##
==========================================
- Coverage   27.39%   27.21%   -0.19%     
==========================================
  Files          56       56              
  Lines        6191     6214      +23     
==========================================
- Hits         1696     1691       -5     
- Misses       4495     4523      +28

Flag	Coverage Δ
unittests	`27.21% <ø> (-0.19%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: hfadzxy <starmoon_zhang@163.com>

Yikun · 2025-06-25T03:03:14Z

.github/workflows/accuracy_test.yaml

+      - name: Get vLLM-Ascend commit hash and URL
+        working-directory: ./vllm-ascend
+        run: |
+          VLLM_ASCEND_COMMIT=$(git rev-parse HEAD)


Yikun

Thanks, let's merge this to see it work or not. some nits could address in separate PR.

Yikun · 2025-06-25T03:03:23Z

.github/workflows/accuracy_test.yaml

+        run: |
+          VLLM_ASCEND_COMMIT=$(git rev-parse HEAD)
+          echo "VLLM_ASCEND_COMMIT=$VLLM_ASCEND_COMMIT" >> $GITHUB_ENV
+          echo "VLLM_ASCEND_COMMIT_URL=https://github.com/vllm-project/vllm-ascend/commit/$VLLM_ASCEND_COMMIT" >> $GITHUB_ENV


Yikun · 2025-06-25T06:14:40Z

post test here: https://github.com/vllm-project/vllm-ascend/actions/workflows/accuracy_test.yaml

### What this PR does / why we need it? fix accuracy test: 1. fix accuracy report like:https://vllm-ascend--1429.org.readthedocs.build/en/1429/developer_guide/evaluation/accuracy_report/Qwen2.5-7B-Instruct-V0.html 2. fix create pr for report Signed-off-by: hfadzxy <starmoon_zhang@163.com>

### What this PR does / why we need it? Update accuracy report test 1. Add Record commit hashes and GitHub links for both vllm and vllm-ascend in accuracy reports 2. Add accuracy result verification checks to ensure output correctness 3. Creat PR via forked repository workflow ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? dense-accuracy-test: https://github.com/vllm-project/vllm-ascend/actions/runs/15745619485 create pr via forked repository workflow: https://github.com/zhangxinyuehfad/vllm-ascend/actions/runs/15747013719/job/44385134080 accuracy report pr: vllm-project#1292 Currently, the accuracy report used is old and needs to be merged into pr, retest, update new report, then close vllm-project#1292 . Signed-off-by: hfadzxy <starmoon_zhang@163.com>

…llm-project#1435) ### What this PR does / why we need it? fix accuracy test: 1. fix accuracy report like:https://vllm-ascend--1429.org.readthedocs.build/en/1429/developer_guide/evaluation/accuracy_report/Qwen2.5-7B-Instruct-V0.html 2. fix create pr for report Signed-off-by: hfadzxy <starmoon_zhang@163.com>

zhangxinyuehfad force-pushed the zxy_accuracy_report branch from cc85eba to d32a860 Compare June 18, 2025 15:16

vllm-ascend-ci added dense-accuracy-test accuracy-test enable all accuracy test for PR ready-for-test start test by label for PR and removed dense-accuracy-test accuracy-test enable all accuracy test for PR labels Jun 18, 2025

zhangxinyuehfad force-pushed the zxy_accuracy_report branch 2 times, most recently from 4832fa4 to 7052731 Compare June 18, 2025 17:50

vllm-ascend-ci added ready-for-test start test by label for PR and removed ready-for-test start test by label for PR labels Jun 18, 2025

zhangxinyuehfad force-pushed the zxy_accuracy_report branch from 7052731 to ba0fb30 Compare June 19, 2025 00:29

zhangxinyuehfad mentioned this pull request Jun 19, 2025

[Doc] Update accuracy reports for main #1292

Closed

zhangxinyuehfad force-pushed the zxy_accuracy_report branch from ba0fb30 to 580b14f Compare June 19, 2025 01:44

Yikun reviewed Jun 19, 2025

View reviewed changes

zhangxinyuehfad force-pushed the zxy_accuracy_report branch 4 times, most recently from fe8dd57 to b0b31fc Compare June 23, 2025 01:41

vllm-ascend-ci added dense-accuracy-test and removed dense-accuracy-test labels Jun 23, 2025

zhangxinyuehfad force-pushed the zxy_accuracy_report branch 2 times, most recently from 10af3a6 to ae04ba2 Compare June 23, 2025 11:41

zhangxinyuehfad force-pushed the zxy_accuracy_report branch from ae04ba2 to b0f64e8 Compare June 23, 2025 12:01

vllm-ascend-ci added dense-accuracy-test and removed dense-accuracy-test labels Jun 24, 2025

zhangxinyuehfad force-pushed the zxy_accuracy_report branch 3 times, most recently from 477f4eb to 5c5d704 Compare June 24, 2025 08:04

[CI] Update accuracy report test

9a0fc54

Signed-off-by: hfadzxy <starmoon_zhang@163.com>

zhangxinyuehfad force-pushed the zxy_accuracy_report branch from 5c5d704 to 9a0fc54 Compare June 24, 2025 15:21

vllm-ascend-ci added dense-accuracy-test and removed dense-accuracy-test labels Jun 25, 2025

Yikun reviewed Jun 25, 2025

View reviewed changes

Yikun approved these changes Jun 25, 2025

View reviewed changes

Yikun merged commit 0060886 into vllm-project:main Jun 25, 2025
20 of 22 checks passed

[CI]Update accuracy report test #1288

[CI]Update accuracy report test #1288

Uh oh!

Conversation

zhangxinyuehfad commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

🎯 Qwen2.5-7B-Instruct

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Yikun left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Yikun commented Jun 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zhangxinyuehfad commented Jun 18, 2025 •

edited

Loading

codecov bot commented Jun 23, 2025 •

edited

Loading