Skip to content

Add testing for latest stable version (JDK 24) #8875

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

sarahchen6
Copy link
Contributor

@sarahchen6 sarahchen6 commented May 23, 2025

What Does This Do

Add testing for stable image (JDK 24)

Motivation

Add Java 24 tests to CI

Additional Notes

Contributor Checklist

Jira ticket: https://datadoghq.atlassian.net/browse/LANGPLAT-421

@pr-commenter
Copy link

pr-commenter bot commented May 23, 2025

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master sarahchen6/add-jdk-24-testing
git_commit_date 1749044153 1749404317
git_commit_sha 7787af7 c7ee38f
release_version 1.50.0-SNAPSHOT~7787af738f 1.50.0-SNAPSHOT~c7ee38f9a8
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1749406679 1749406679
ci_job_id 973506922 973506922
ci_pipeline_id 67293364 67293364
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-pz8nwqvr-project-304-concurrent-0-anfl7pwg 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-pz8nwqvr-project-304-concurrent-0-anfl7pwg 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 58 metrics, 13 unstable metrics.

Startup time reports for insecure-bank
gantt
    title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~c7ee38f9a8, baseline=1.50.0-SNAPSHOT~7787af738f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.021 s) : 0, 1020579
Total [baseline] (8.501 s) : 0, 8500823
Agent [candidate] (1.025 s) : 0, 1025036
Total [candidate] (8.541 s) : 0, 8540846
section iast
Agent [baseline] (1.149 s) : 0, 1148962
Total [baseline] (9.217 s) : 0, 9216732
Agent [candidate] (1.149 s) : 0, 1148898
Total [candidate] (9.191 s) : 0, 9191169
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.153 s) : 0, 1153018
Total [baseline] (9.132 s) : 0, 9132162
Agent [candidate] (1.15 s) : 0, 1150171
Total [candidate] (9.145 s) : 0, 9144767
section iast_TELEMETRY_OFF
Agent [baseline] (1.153 s) : 0, 1153076
Total [baseline] (9.204 s) : 0, 9203620
Agent [candidate] (1.156 s) : 0, 1155994
Total [candidate] (9.235 s) : 0, 9235463
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.021 s -
Agent iast 1.149 s 128.384 ms (12.6%)
Agent iast_HARDCODED_SECRET_DISABLED 1.153 s 132.439 ms (13.0%)
Agent iast_TELEMETRY_OFF 1.153 s 132.497 ms (13.0%)
Total tracing 8.501 s -
Total iast 9.217 s 715.909 ms (8.4%)
Total iast_HARDCODED_SECRET_DISABLED 9.132 s 631.339 ms (7.4%)
Total iast_TELEMETRY_OFF 9.204 s 702.798 ms (8.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.025 s -
Agent iast 1.149 s 123.863 ms (12.1%)
Agent iast_HARDCODED_SECRET_DISABLED 1.15 s 125.136 ms (12.2%)
Agent iast_TELEMETRY_OFF 1.156 s 130.959 ms (12.8%)
Total tracing 8.541 s -
Total iast 9.191 s 650.322 ms (7.6%)
Total iast_HARDCODED_SECRET_DISABLED 9.145 s 603.921 ms (7.1%)
Total iast_TELEMETRY_OFF 9.235 s 694.617 ms (8.1%)
gantt
    title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~c7ee38f9a8, baseline=1.50.0-SNAPSHOT~7787af738f

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (681.663 ms) : 0, 681663
BytebuddyAgent [candidate] (683.738 ms) : 0, 683738
GlobalTracer [baseline] (239.553 ms) : 0, 239553
GlobalTracer [candidate] (241.038 ms) : 0, 241038
AppSec [baseline] (57.809 ms) : 0, 57809
AppSec [candidate] (56.432 ms) : 0, 56432
Debugger [baseline] (6.149 ms) : 0, 6149
Debugger [candidate] (6.235 ms) : 0, 6235
Remote Config [baseline] (721.143 µs) : 0, 721
Remote Config [candidate] (738.674 µs) : 0, 739
Telemetry [baseline] (11.204 ms) : 0, 11204
Telemetry [candidate] (13.359 ms) : 0, 13359
section iast
BytebuddyAgent [baseline] (800.915 ms) : 0, 800915
BytebuddyAgent [candidate] (801.278 ms) : 0, 801278
GlobalTracer [baseline] (230.319 ms) : 0, 230319
GlobalTracer [candidate] (230.941 ms) : 0, 230941
IAST [baseline] (27.674 ms) : 0, 27674
IAST [candidate] (26.787 ms) : 0, 26787
AppSec [baseline] (51.982 ms) : 0, 51982
AppSec [candidate] (52.062 ms) : 0, 52062
Debugger [baseline] (5.982 ms) : 0, 5982
Debugger [candidate] (5.947 ms) : 0, 5947
Remote Config [baseline] (604.375 µs) : 0, 604
Remote Config [candidate] (593.437 µs) : 0, 593
Telemetry [baseline] (7.993 ms) : 0, 7993
Telemetry [candidate] (7.87 ms) : 0, 7870
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (804.243 ms) : 0, 804243
BytebuddyAgent [candidate] (801.485 ms) : 0, 801485
GlobalTracer [baseline] (231.095 ms) : 0, 231095
GlobalTracer [candidate] (230.542 ms) : 0, 230542
IAST [baseline] (29.377 ms) : 0, 29377
IAST [candidate] (26.301 ms) : 0, 26301
AppSec [baseline] (50.234 ms) : 0, 50234
AppSec [candidate] (53.768 ms) : 0, 53768
Debugger [baseline] (5.982 ms) : 0, 5982
Debugger [candidate] (6.011 ms) : 0, 6011
Remote Config [baseline] (602.188 µs) : 0, 602
Remote Config [candidate] (590.079 µs) : 0, 590
Telemetry [baseline] (7.948 ms) : 0, 7948
Telemetry [candidate] (7.952 ms) : 0, 7952
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (803.538 ms) : 0, 803538
BytebuddyAgent [candidate] (808.142 ms) : 0, 808142
GlobalTracer [baseline] (231.722 ms) : 0, 231722
GlobalTracer [candidate] (230.907 ms) : 0, 230907
IAST [baseline] (28.96 ms) : 0, 28960
IAST [candidate] (28.737 ms) : 0, 28737
AppSec [baseline] (50.725 ms) : 0, 50725
AppSec [candidate] (50.13 ms) : 0, 50130
Debugger [baseline] (6.062 ms) : 0, 6062
Debugger [candidate] (5.971 ms) : 0, 5971
Remote Config [baseline] (607.847 µs) : 0, 608
Remote Config [candidate] (596.41 µs) : 0, 596
Telemetry [baseline] (7.954 ms) : 0, 7954
Telemetry [candidate] (7.872 ms) : 0, 7872
Loading
Startup time reports for petclinic
gantt
    title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~c7ee38f9a8, baseline=1.50.0-SNAPSHOT~7787af738f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.019 s) : 0, 1019267
Total [baseline] (11.093 s) : 0, 11092587
Agent [candidate] (1.022 s) : 0, 1021876
Total [candidate] (11.144 s) : 0, 11143836
section appsec
Agent [baseline] (1.166 s) : 0, 1165748
Total [baseline] (11.206 s) : 0, 11205942
Agent [candidate] (1.163 s) : 0, 1162894
Total [candidate] (11.224 s) : 0, 11223941
section iast
Agent [baseline] (1.172 s) : 0, 1172318
Total [baseline] (11.396 s) : 0, 11395542
Agent [candidate] (1.149 s) : 0, 1148695
Total [candidate] (11.472 s) : 0, 11472187
section profiling
Agent [baseline] (1.268 s) : 0, 1268212
Total [baseline] (11.437 s) : 0, 11437267
Agent [candidate] (1.263 s) : 0, 1263078
Total [candidate] (11.432 s) : 0, 11431639
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.019 s -
Agent appsec 1.166 s 146.482 ms (14.4%)
Agent iast 1.172 s 153.051 ms (15.0%)
Agent profiling 1.268 s 248.945 ms (24.4%)
Total tracing 11.093 s -
Total appsec 11.206 s 113.355 ms (1.0%)
Total iast 11.396 s 302.955 ms (2.7%)
Total profiling 11.437 s 344.681 ms (3.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.022 s -
Agent appsec 1.163 s 141.018 ms (13.8%)
Agent iast 1.149 s 126.819 ms (12.4%)
Agent profiling 1.263 s 241.202 ms (23.6%)
Total tracing 11.144 s -
Total appsec 11.224 s 80.105 ms (0.7%)
Total iast 11.472 s 328.351 ms (2.9%)
Total profiling 11.432 s 287.803 ms (2.6%)
gantt
    title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~c7ee38f9a8, baseline=1.50.0-SNAPSHOT~7787af738f

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (681.228 ms) : 0, 681228
BytebuddyAgent [candidate] (682.501 ms) : 0, 682501
GlobalTracer [baseline] (239.554 ms) : 0, 239554
GlobalTracer [candidate] (240.618 ms) : 0, 240618
AppSec [baseline] (57.684 ms) : 0, 57684
AppSec [candidate] (56.314 ms) : 0, 56314
Debugger [baseline] (6.124 ms) : 0, 6124
Debugger [candidate] (6.157 ms) : 0, 6157
Remote Config [baseline] (745.169 µs) : 0, 745
Remote Config [candidate] (731.489 µs) : 0, 731
Telemetry [baseline] (10.424 ms) : 0, 10424
Telemetry [candidate] (11.921 ms) : 0, 11921
section appsec
BytebuddyAgent [baseline] (702.621 ms) : 0, 702621
BytebuddyAgent [candidate] (700.655 ms) : 0, 700655
GlobalTracer [baseline] (238.206 ms) : 0, 238206
GlobalTracer [candidate] (237.757 ms) : 0, 237757
IAST [baseline] (21.964 ms) : 0, 21964
IAST [candidate] (22.001 ms) : 0, 22001
AppSec [baseline] (176.332 ms) : 0, 176332
AppSec [candidate] (175.95 ms) : 0, 175950
Debugger [baseline] (6.004 ms) : 0, 6004
Debugger [candidate] (5.986 ms) : 0, 5986
Remote Config [baseline] (621.783 µs) : 0, 622
Remote Config [candidate] (624.226 µs) : 0, 624
Telemetry [baseline] (7.368 ms) : 0, 7368
Telemetry [candidate] (7.364 ms) : 0, 7364
section iast
BytebuddyAgent [baseline] (817.417 ms) : 0, 817417
BytebuddyAgent [candidate] (800.727 ms) : 0, 800727
GlobalTracer [baseline] (234.975 ms) : 0, 234975
GlobalTracer [candidate] (230.629 ms) : 0, 230629
IAST [baseline] (26.929 ms) : 0, 26929
IAST [candidate] (25.143 ms) : 0, 25143
AppSec [baseline] (54.356 ms) : 0, 54356
AppSec [candidate] (54.344 ms) : 0, 54344
Debugger [baseline] (6.015 ms) : 0, 6015
Debugger [candidate] (5.877 ms) : 0, 5877
Remote Config [baseline] (609.42 µs) : 0, 609
Remote Config [candidate] (588.745 µs) : 0, 589
Telemetry [baseline] (8.027 ms) : 0, 8027
Telemetry [candidate] (7.849 ms) : 0, 7849
section profiling
BytebuddyAgent [baseline] (676.783 ms) : 0, 676783
BytebuddyAgent [candidate] (674.451 ms) : 0, 674451
GlobalTracer [baseline] (360.768 ms) : 0, 360768
GlobalTracer [candidate] (359.225 ms) : 0, 359225
AppSec [baseline] (61.766 ms) : 0, 61766
AppSec [candidate] (61.395 ms) : 0, 61395
Debugger [baseline] (6.102 ms) : 0, 6102
Debugger [candidate] (6.112 ms) : 0, 6112
Remote Config [baseline] (648.508 µs) : 0, 649
Remote Config [candidate] (644.568 µs) : 0, 645
Telemetry [baseline] (8.152 ms) : 0, 8152
Telemetry [candidate] (8.118 ms) : 0, 8118
ProfilingAgent [baseline] (103.108 ms) : 0, 103108
ProfilingAgent [candidate] (102.396 ms) : 0, 102396
Profiling [baseline] (103.132 ms) : 0, 103132
Profiling [candidate] (102.42 ms) : 0, 102420
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2025-06-08T17:47:27 2025-06-08T18:00:36
git_branch master sarahchen6/add-jdk-24-testing
git_commit_date 1749044153 1749404317
git_commit_sha 7787af7 c7ee38f
release_version 1.50.0-SNAPSHOT~7787af738f 1.50.0-SNAPSHOT~c7ee38f9a8
start_time 2025-06-08T17:47:12 2025-06-08T18:00:21
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1749406262 1749406262
ci_job_id 973506923 973506923
ci_pipeline_id 67293364 67293364
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-pz8nwqvr-project-304-concurrent-1-catj7513 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-pz8nwqvr-project-304-concurrent-1-catj7513 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
thresholds_or_results results results
variant iast iast

Summary

Found 2 performance improvements and 4 performance regressions! Performance is the same for 5 metrics, 17 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:petclinic:code_origins unstable
[+50.064ms; +51.430ms] or [+634.792%; +652.107%]
worse
[-569.925op/s; -528.342op/s] or [-91.188%; -84.535%]
58.633ms 75.867op/s 7.887ms 625.000op/s
scenario:load:petclinic:iast unstable
[+43.473ms; +44.355ms] or [+533.715%; +544.539%]
worse
[-531.213op/s; -501.293op/s] or [-87.650%; -82.713%]
52.059ms 89.808op/s 8.145ms 606.061op/s
scenario:load:petclinic:no_agent worse
[+950.232µs; +998.522µs] or [+12.402%; +13.032%]
worse
[-88.038op/s; -49.104op/s] or [-13.756%; -7.673%]
8.636ms 571.429op/s 7.662ms 640.000op/s
scenario:load:petclinic:profiling better
[-687.061µs; -588.783µs] or [-58.757%; -50.353%]
unstable
[-3271.993op/s; +9648.804op/s] or [-94.070%; +277.403%]
0.531ms 6666.667op/s 1.169ms 3478.261op/s
scenario:load:petclinic:tracing better
[-699.274µs; -607.422µs] or [-7.163%; -6.222%]
unstable
[+1.957op/s; +66.466op/s] or [+0.387%; +13.127%]
9.109ms 540.541op/s 9.762ms 506.329op/s

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master sarahchen6/add-jdk-24-testing
git_commit_date 1749044153 1749404317
git_commit_sha 7787af7 c7ee38f
release_version 1.50.0-SNAPSHOT~7787af738f 1.50.0-SNAPSHOT~c7ee38f9a8
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1749406205 1749406205
ci_job_id 973506924 973506924
ci_pipeline_id 67293364 67293364
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-i2yz3wjx-project-304-concurrent-0-ahvi3lvc 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-i2yz3wjx-project-304-concurrent-0-ahvi3lvc 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~c7ee38f9a8, baseline=1.50.0-SNAPSHOT~7787af738f
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.476 ms) : 1464, 1487
.   : milestone, 1476,
appsec (2.392 ms) : 2344, 2440
.   : milestone, 2392,
iast (2.163 ms) : 2102, 2224
.   : milestone, 2163,
iast_GLOBAL (2.218 ms) : 2157, 2280
.   : milestone, 2218,
profiling (2.014 ms) : 1965, 2063
.   : milestone, 2014,
tracing (1.998 ms) : 1951, 2045
.   : milestone, 1998,
section candidate
no_agent (1.475 ms) : 1464, 1487
.   : milestone, 1475,
appsec (2.385 ms) : 2337, 2433
.   : milestone, 2385,
iast (2.178 ms) : 2117, 2239
.   : milestone, 2178,
iast_GLOBAL (2.224 ms) : 2162, 2285
.   : milestone, 2224,
profiling (2.016 ms) : 1967, 2064
.   : milestone, 2016,
tracing (2.011 ms) : 1963, 2059
.   : milestone, 2011,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.476 ms [1.464 ms, 1.487 ms] -
appsec 2.392 ms [2.344 ms, 2.44 ms] 916.362 µs (62.1%)
iast 2.163 ms [2.102 ms, 2.224 ms] 687.58 µs (46.6%)
iast_GLOBAL 2.218 ms [2.157 ms, 2.28 ms] 742.659 µs (50.3%)
profiling 2.014 ms [1.965 ms, 2.063 ms] 538.659 µs (36.5%)
tracing 1.998 ms [1.951 ms, 2.045 ms] 522.252 µs (35.4%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.475 ms [1.464 ms, 1.487 ms] -
appsec 2.385 ms [2.337 ms, 2.433 ms] 909.841 µs (61.7%)
iast 2.178 ms [2.117 ms, 2.239 ms] 702.407 µs (47.6%)
iast_GLOBAL 2.224 ms [2.162 ms, 2.285 ms] 748.36 µs (50.7%)
profiling 2.016 ms [1.967 ms, 2.064 ms] 540.493 µs (36.6%)
tracing 2.011 ms [1.963 ms, 2.059 ms] 535.573 µs (36.3%)
Execution time for biojava
gantt
    title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~c7ee38f9a8, baseline=1.50.0-SNAPSHOT~7787af738f
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.477 s) : 15477000, 15477000
.   : milestone, 15477000,
appsec (15.103 s) : 15103000, 15103000
.   : milestone, 15103000,
iast (18.849 s) : 18849000, 18849000
.   : milestone, 18849000,
iast_GLOBAL (17.877 s) : 17877000, 17877000
.   : milestone, 17877000,
profiling (15.292 s) : 15292000, 15292000
.   : milestone, 15292000,
tracing (14.799 s) : 14799000, 14799000
.   : milestone, 14799000,
section candidate
no_agent (15.766 s) : 15766000, 15766000
.   : milestone, 15766000,
appsec (14.759 s) : 14759000, 14759000
.   : milestone, 14759000,
iast (18.712 s) : 18712000, 18712000
.   : milestone, 18712000,
iast_GLOBAL (17.853 s) : 17853000, 17853000
.   : milestone, 17853000,
profiling (15.307 s) : 15307000, 15307000
.   : milestone, 15307000,
tracing (14.832 s) : 14832000, 14832000
.   : milestone, 14832000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.477 s [15.477 s, 15.477 s] -
appsec 15.103 s [15.103 s, 15.103 s] -374.0 ms (-2.4%)
iast 18.849 s [18.849 s, 18.849 s] 3.372 s (21.8%)
iast_GLOBAL 17.877 s [17.877 s, 17.877 s] 2.4 s (15.5%)
profiling 15.292 s [15.292 s, 15.292 s] -185.0 ms (-1.2%)
tracing 14.799 s [14.799 s, 14.799 s] -678.0 ms (-4.4%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.766 s [15.766 s, 15.766 s] -
appsec 14.759 s [14.759 s, 14.759 s] -1.007 s (-6.4%)
iast 18.712 s [18.712 s, 18.712 s] 2.946 s (18.7%)
iast_GLOBAL 17.853 s [17.853 s, 17.853 s] 2.087 s (13.2%)
profiling 15.307 s [15.307 s, 15.307 s] -459.0 ms (-2.9%)
tracing 14.832 s [14.832 s, 14.832 s] -934.0 ms (-5.9%)

@sarahchen6 sarahchen6 force-pushed the sarahchen6/add-jdk-24-testing branch 2 times, most recently from 97de10c to bfd5082 Compare June 8, 2025 16:58
@sarahchen6 sarahchen6 force-pushed the sarahchen6/add-jdk-24-testing branch from bfd5082 to c7ee38f Compare June 8, 2025 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant