Skip to content

WIP: instrumenter index experiment #9031

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 1 commit into
base: mcculls/cleanup-instrumenter-enablement
Choose a base branch
from
Draft

WIP: instrumenter index experiment #9031

wants to merge 1 commit into from

Conversation

mcculls
Copy link
Contributor

@mcculls mcculls commented Jun 24, 2025

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@mcculls mcculls added the tag: do not merge Do not merge changes label Jun 24, 2025
@mcculls mcculls changed the base branch from master to mcculls/cleanup-instrumenter-enablement June 24, 2025 23:05
@pr-commenter
Copy link

pr-commenter bot commented Jun 24, 2025 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mcculls/instrumenter-index-experiment
git_commit_date 1750839907 1750806252
git_commit_sha c5581ea 22f46ba
release_version 1.51.0-SNAPSHOT~c5581eae59 1.51.0-SNAPSHOT~22f46ba16f
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1750844293 1750844293
ci_job_id 997462979 997462979
ci_pipeline_id 68681310 68681310
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-bziamzy-project-304-concurrent-0-agd5m9xz 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-bziamzy-project-304-concurrent-0-agd5m9xz 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 3 performance improvements and 3 performance regressions! Performance is the same for 37 metrics, 10 unstable metrics.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:startup:insecure-bank:iast:BytebuddyAgent better
[-24.683ms; -19.034ms] or [-3.065%; -2.364%]		 783.337ms 805.195ms
scenario:startup:insecure-bank:iast:GlobalTracer worse
[+6.014ms; +8.812ms] or [+2.595%; +3.803%]		 239.139ms 231.727ms
scenario:startup:insecure-bank:tracing:BytebuddyAgent better
[-19.179ms; -13.943ms] or [-2.793%; -2.031%]		 670.069ms 686.630ms
scenario:startup:petclinic:appsec:BytebuddyAgent better
[-26.633ms; -14.932ms] or [-3.756%; -2.106%]		 688.280ms 709.063ms
scenario:startup:petclinic:appsec:GlobalTracer worse
[+6.093ms; +12.601ms] or [+2.590%; +5.356%]		 244.608ms 235.261ms
scenario:startup:petclinic:iast:GlobalTracer worse
[+5.764ms; +13.519ms] or [+2.484%; +5.826%]		 241.692ms 232.050ms
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.51.0-SNAPSHOT~22f46ba16f, baseline=1.51.0-SNAPSHOT~c5581eae59

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (993.939 ms) : 0, 993939
Total [baseline] (8.535 s) : 0, 8534971
Agent [candidate] (977.466 ms) : 0, 977466
Total [candidate] (8.523 s) : 0, 8523050
section iast
Agent [baseline] (1.127 s) : 0, 1126634
Total [baseline] (9.23 s) : 0, 9230270
Agent [candidate] (1.113 s) : 0, 1112623
Total [candidate] (9.255 s) : 0, 9254605
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 993.939 ms -
Agent iast 1.127 s 132.695 ms (13.4%)
Total tracing 8.535 s -
Total iast 9.23 s 695.299 ms (8.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 977.466 ms -
Agent iast 1.113 s 135.157 ms (13.8%)
Total tracing 8.523 s -
Total iast 9.255 s 731.555 ms (8.6%) 
gantt
    title insecure-bank - break down per module: candidate=1.51.0-SNAPSHOT~22f46ba16f, baseline=1.51.0-SNAPSHOT~c5581eae59

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (686.63 ms) : 0, 686630
BytebuddyAgent [candidate] (670.069 ms) : 0, 670069
GlobalTracer [baseline] (241.645 ms) : 0, 241645
GlobalTracer [candidate] (241.58 ms) : 0, 241580
AppSec [baseline] (30.029 ms) : 0, 30029
AppSec [candidate] (30.01 ms) : 0, 30010
Debugger [baseline] (6.054 ms) : 0, 6054
Debugger [candidate] (6.015 ms) : 0, 6015
Remote Config [baseline] (658.682 µs) : 0, 659
Remote Config [candidate] (655.444 µs) : 0, 655
Telemetry [baseline] (8.106 ms) : 0, 8106
Telemetry [candidate] (8.267 ms) : 0, 8267
section iast
BytebuddyAgent [baseline] (805.195 ms) : 0, 805195
BytebuddyAgent [candidate] (783.337 ms) : 0, 783337
GlobalTracer [baseline] (231.727 ms) : 0, 231727
GlobalTracer [candidate] (239.139 ms) : 0, 239139
AppSec [baseline] (24.858 ms) : 0, 24858
AppSec [candidate] (31.176 ms) : 0, 31176
Debugger [baseline] (5.781 ms) : 0, 5781
Debugger [candidate] (5.706 ms) : 0, 5706
Remote Config [baseline] (585.039 µs) : 0, 585
Remote Config [candidate] (557.426 µs) : 0, 557
Telemetry [baseline] (7.903 ms) : 0, 7903
Telemetry [candidate] (8.712 ms) : 0, 8712
IAST [baseline] (29.97 ms) : 0, 29970
IAST [candidate] (23.087 ms) : 0, 23087
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.51.0-SNAPSHOT~22f46ba16f, baseline=1.51.0-SNAPSHOT~c5581eae59

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (998.751 ms) : 0, 998751
Total [baseline] (10.705 s) : 0, 10705191
Agent [candidate] (981.507 ms) : 0, 981507
Total [candidate] (10.654 s) : 0, 10653802
section appsec
Agent [baseline] (1.171 s) : 0, 1170512
Total [baseline] (10.699 s) : 0, 10698515
Agent [candidate] (1.161 s) : 0, 1160925
Total [candidate] (10.784 s) : 0, 10783520
section iast
Agent [baseline] (1.129 s) : 0, 1129178
Total [baseline] (10.814 s) : 0, 10813742
Agent [candidate] (1.128 s) : 0, 1127566
Total [candidate] (11.029 s) : 0, 11029151
section profiling
Agent [baseline] (1.24 s) : 0, 1240424
Total [baseline] (10.957 s) : 0, 10956680
Agent [candidate] (1.231 s) : 0, 1230830
Total [candidate] (10.968 s) : 0, 10967575
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 998.751 ms -
Agent appsec 1.171 s 171.761 ms (17.2%)
Agent iast 1.129 s 130.427 ms (13.1%)
Agent profiling 1.24 s 241.672 ms (24.2%)
Total tracing 10.705 s -
Total appsec 10.699 s -6.676 ms (-0.1%)
Total iast 10.814 s 108.551 ms (1.0%)
Total profiling 10.957 s 251.488 ms (2.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 981.507 ms -
Agent appsec 1.161 s 179.418 ms (18.3%)
Agent iast 1.128 s 146.058 ms (14.9%)
Agent profiling 1.231 s 249.322 ms (25.4%)
Total tracing 10.654 s -
Total appsec 10.784 s 129.717 ms (1.2%)
Total iast 11.029 s 375.349 ms (3.5%)
Total profiling 10.968 s 313.773 ms (2.9%) 
gantt
    title petclinic - break down per module: candidate=1.51.0-SNAPSHOT~22f46ba16f, baseline=1.51.0-SNAPSHOT~c5581eae59

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (689.608 ms) : 0, 689608
BytebuddyAgent [candidate] (674.549 ms) : 0, 674549
GlobalTracer [baseline] (243.037 ms) : 0, 243037
GlobalTracer [candidate] (241.145 ms) : 0, 241145
AppSec [baseline] (30.236 ms) : 0, 30236
AppSec [candidate] (29.952 ms) : 0, 29952
Debugger [baseline] (6.085 ms) : 0, 6085
Debugger [candidate] (6.001 ms) : 0, 6001
Remote Config [baseline] (661.75 µs) : 0, 662
Remote Config [candidate] (640.932 µs) : 0, 641
Telemetry [baseline] (8.182 ms) : 0, 8182
Telemetry [candidate] (8.198 ms) : 0, 8198
section appsec
BytebuddyAgent [baseline] (709.063 ms) : 0, 709063
BytebuddyAgent [candidate] (688.28 ms) : 0, 688280
GlobalTracer [baseline] (235.261 ms) : 0, 235261
GlobalTracer [candidate] (244.608 ms) : 0, 244608
AppSec [baseline] (168.974 ms) : 0, 168974
AppSec [candidate] (170.092 ms) : 0, 170092
Debugger [baseline] (5.804 ms) : 0, 5804
Debugger [candidate] (5.804 ms) : 0, 5804
Remote Config [baseline] (610.149 µs) : 0, 610
Remote Config [candidate] (610.003 µs) : 0, 610
Telemetry [baseline] (8.188 ms) : 0, 8188
Telemetry [candidate] (8.366 ms) : 0, 8366
IAST [baseline] (21.855 ms) : 0, 21855
IAST [candidate] (22.329 ms) : 0, 22329
section iast
BytebuddyAgent [baseline] (806.59 ms) : 0, 806590
BytebuddyAgent [candidate] (793.986 ms) : 0, 793986
GlobalTracer [baseline] (232.05 ms) : 0, 232050
GlobalTracer [candidate] (241.692 ms) : 0, 241692
AppSec [baseline] (26.096 ms) : 0, 26096
AppSec [candidate] (32.771 ms) : 0, 32771
Debugger [baseline] (5.789 ms) : 0, 5789
Debugger [candidate] (5.865 ms) : 0, 5865
Remote Config [baseline] (580.906 µs) : 0, 581
Remote Config [candidate] (580.16 µs) : 0, 580
Telemetry [baseline] (7.924 ms) : 0, 7924
Telemetry [candidate] (7.993 ms) : 0, 7993
IAST [baseline] (29.368 ms) : 0, 29368
IAST [candidate] (23.522 ms) : 0, 23522
section profiling
BytebuddyAgent [baseline] (676.487 ms) : 0, 676487
BytebuddyAgent [candidate] (665.179 ms) : 0, 665179
GlobalTracer [baseline] (360.677 ms) : 0, 360677
GlobalTracer [candidate] (361.767 ms) : 0, 361767
AppSec [baseline] (32.081 ms) : 0, 32081
AppSec [candidate] (30.586 ms) : 0, 30586
Debugger [baseline] (7.514 ms) : 0, 7514
Debugger [candidate] (11.163 ms) : 0, 11163
Remote Config [baseline] (678.241 µs) : 0, 678
Remote Config [candidate] (677.715 µs) : 0, 678
Telemetry [baseline] (10.953 ms) : 0, 10953
Telemetry [candidate] (8.827 ms) : 0, 8827
ProfilingAgent [baseline] (102.732 ms) : 0, 102732
ProfilingAgent [candidate] (104.054 ms) : 0, 104054
Profiling [baseline] (102.758 ms) : 0, 102758
Profiling [candidate] (104.078 ms) : 0, 104078
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mcculls/instrumenter-index-experiment
git_commit_date 1750788935 1750806252
git_commit_sha faeb62c 22f46ba
release_version 1.51.0-SNAPSHOT~faeb62cfb1 1.51.0-SNAPSHOT~22f46ba16f
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1750807926 1750807926
ci_job_id 997130998 997130998
ci_pipeline_id 68681310 68681310
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-vaym2f8s-project-304-concurrent-0-73h7z8i5 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-vaym2f8s-project-304-concurrent-0-73h7z8i5 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 2 performance regressions! Performance is the same for 9 metrics, 12 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:insecure-bank:profiling:high_load better
[-923.042µs; -617.688µs] or [-9.865%; -6.602%]		 unstable
[-20.180op/s; +108.305op/s] or [-4.068%; +21.834%]		 8.586ms 540.094op/s 9.357ms 496.031op/s
scenario:load:petclinic:code_origins:high_load worse
[+2.003ms; +2.816ms] or [+4.592%; +6.457%]		 unstable
[-13.280op/s; +2.055op/s] or [-12.383%; +1.916%]		 46.025ms 101.638op/s 43.616ms 107.250op/s
scenario:load:petclinic:iast:high_load worse
[+2.206ms; +3.102ms] or [+4.954%; +6.967%]		 unstable
[-13.815op/s; +1.965op/s] or [-13.143%; +1.870%]		 47.182ms 99.188op/s 44.528ms 105.112op/s
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~22f46ba16f, baseline=1.51.0-SNAPSHOT~faeb62cfb1
    dateFormat X
    axisFormat %s
section baseline
no_agent (35.892 ms) : 35604, 36181
.   : milestone, 35892,
appsec (48.025 ms) : 47601, 48450
.   : milestone, 48025,
code_origins (43.616 ms) : 43249, 43982
.   : milestone, 43616,
iast (44.528 ms) : 44131, 44925
.   : milestone, 44528,
profiling (50.022 ms) : 49528, 50517
.   : milestone, 50022,
tracing (44.908 ms) : 44548, 45269
.   : milestone, 44908,
section candidate
no_agent (36.639 ms) : 36342, 36936
.   : milestone, 36639,
appsec (47.064 ms) : 46646, 47481
.   : milestone, 47064,
code_origins (46.025 ms) : 45636, 46414
.   : milestone, 46025,
iast (47.182 ms) : 46747, 47617
.   : milestone, 47182,
profiling (51.075 ms) : 50554, 51596
.   : milestone, 51075,
tracing (44.247 ms) : 43868, 44626
.   : milestone, 44247,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 35.892 ms [35.604 ms, 36.181 ms] -
appsec 48.025 ms [47.601 ms, 48.45 ms] 12.133 ms (33.8%)
code_origins 43.616 ms [43.249 ms, 43.982 ms] 7.723 ms (21.5%)
iast 44.528 ms [44.131 ms, 44.925 ms] 8.636 ms (24.1%)
profiling 50.022 ms [49.528 ms, 50.517 ms] 14.13 ms (39.4%)
tracing 44.908 ms [44.548 ms, 45.269 ms] 9.016 ms (25.1%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 36.639 ms [36.342 ms, 36.936 ms] -
appsec 47.064 ms [46.646 ms, 47.481 ms] 10.425 ms (28.5%)
code_origins 46.025 ms [45.636 ms, 46.414 ms] 9.386 ms (25.6%)
iast 47.182 ms [46.747 ms, 47.617 ms] 10.543 ms (28.8%)
profiling 51.075 ms [50.554 ms, 51.596 ms] 14.437 ms (39.4%)
tracing 44.247 ms [43.868 ms, 44.626 ms] 7.608 ms (20.8%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~22f46ba16f, baseline=1.51.0-SNAPSHOT~faeb62cfb1
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.468 ms) : 4410, 4526
.   : milestone, 4468,
iast (9.094 ms) : 8948, 9240
.   : milestone, 9094,
iast_FULL (13.839 ms) : 13570, 14109
.   : milestone, 13839,
iast_GLOBAL (9.967 ms) : 9777, 10157
.   : milestone, 9967,
profiling (9.357 ms) : 9204, 9509
.   : milestone, 9357,
tracing (7.749 ms) : 7635, 7863
.   : milestone, 7749,
section candidate
no_agent (4.492 ms) : 4439, 4544
.   : milestone, 4492,
iast (9.377 ms) : 9225, 9528
.   : milestone, 9377,
iast_FULL (13.575 ms) : 13302, 13847
.   : milestone, 13575,
iast_GLOBAL (10.084 ms) : 9910, 10259
.   : milestone, 10084,
profiling (8.586 ms) : 8456, 8716
.   : milestone, 8586,
tracing (7.756 ms) : 7645, 7868
.   : milestone, 7756,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 4.468 ms [4.41 ms, 4.526 ms] -
iast 9.094 ms [8.948 ms, 9.24 ms] 4.626 ms (103.5%)
iast_FULL 13.839 ms [13.57 ms, 14.109 ms] 9.371 ms (209.7%)
iast_GLOBAL 9.967 ms [9.777 ms, 10.157 ms] 5.499 ms (123.1%)
profiling 9.357 ms [9.204 ms, 9.509 ms] 4.888 ms (109.4%)
tracing 7.749 ms [7.635 ms, 7.863 ms] 3.281 ms (73.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 4.492 ms [4.439 ms, 4.544 ms] -
iast 9.377 ms [9.225 ms, 9.528 ms] 4.885 ms (108.8%)
iast_FULL 13.575 ms [13.302 ms, 13.847 ms] 9.083 ms (202.2%)
iast_GLOBAL 10.084 ms [9.91 ms, 10.259 ms] 5.593 ms (124.5%)
profiling 8.586 ms [8.456 ms, 8.716 ms] 4.095 ms (91.2%)
tracing 7.756 ms [7.645 ms, 7.868 ms] 3.265 ms (72.7%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mcculls/instrumenter-index-experiment
git_commit_date 1750788935 1750806252
git_commit_sha faeb62c 22f46ba
release_version 1.51.0-SNAPSHOT~faeb62cfb1 1.51.0-SNAPSHOT~22f46ba16f
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1750808414 1750808414
ci_job_id 997131000 997131000
ci_pipeline_id 68681310 68681310
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-6z7wq9aq-project-304-concurrent-0-09ta9ift 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-6z7wq9aq-project-304-concurrent-0-09ta9ift 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~22f46ba16f, baseline=1.51.0-SNAPSHOT~faeb62cfb1
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.606 s) : 15606000, 15606000
.   : milestone, 15606000,
appsec (14.84 s) : 14840000, 14840000
.   : milestone, 14840000,
iast (18.382 s) : 18382000, 18382000
.   : milestone, 18382000,
iast_GLOBAL (18.036 s) : 18036000, 18036000
.   : milestone, 18036000,
profiling (15.2 s) : 15200000, 15200000
.   : milestone, 15200000,
tracing (14.56 s) : 14560000, 14560000
.   : milestone, 14560000,
section candidate
no_agent (15.552 s) : 15552000, 15552000
.   : milestone, 15552000,
appsec (14.965 s) : 14965000, 14965000
.   : milestone, 14965000,
iast (18.307 s) : 18307000, 18307000
.   : milestone, 18307000,
iast_GLOBAL (18.052 s) : 18052000, 18052000
.   : milestone, 18052000,
profiling (15.278 s) : 15278000, 15278000
.   : milestone, 15278000,
tracing (14.812 s) : 14812000, 14812000
.   : milestone, 14812000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.606 s [15.606 s, 15.606 s] -
appsec 14.84 s [14.84 s, 14.84 s] -766.0 ms (-4.9%)
iast 18.382 s [18.382 s, 18.382 s] 2.776 s (17.8%)
iast_GLOBAL 18.036 s [18.036 s, 18.036 s] 2.43 s (15.6%)
profiling 15.2 s [15.2 s, 15.2 s] -406.0 ms (-2.6%)
tracing 14.56 s [14.56 s, 14.56 s] -1.046 s (-6.7%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.552 s [15.552 s, 15.552 s] -
appsec 14.965 s [14.965 s, 14.965 s] -587.0 ms (-3.8%)
iast 18.307 s [18.307 s, 18.307 s] 2.755 s (17.7%)
iast_GLOBAL 18.052 s [18.052 s, 18.052 s] 2.5 s (16.1%)
profiling 15.278 s [15.278 s, 15.278 s] -274.0 ms (-1.8%)
tracing 14.812 s [14.812 s, 14.812 s] -740.0 ms (-4.8%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~22f46ba16f, baseline=1.51.0-SNAPSHOT~faeb62cfb1
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.48 ms) : 1468, 1492
.   : milestone, 1480,
appsec (2.412 ms) : 2363, 2461
.   : milestone, 2412,
iast (2.192 ms) : 2131, 2253
.   : milestone, 2192,
iast_GLOBAL (2.242 ms) : 2180, 2304
.   : milestone, 2242,
profiling (2.043 ms) : 1992, 2093
.   : milestone, 2043,
tracing (2.017 ms) : 1970, 2065
.   : milestone, 2017,
section candidate
no_agent (1.488 ms) : 1476, 1499
.   : milestone, 1488,
appsec (2.408 ms) : 2359, 2456
.   : milestone, 2408,
iast (2.194 ms) : 2133, 2255
.   : milestone, 2194,
iast_GLOBAL (2.225 ms) : 2164, 2286
.   : milestone, 2225,
profiling (2.03 ms) : 1981, 2078
.   : milestone, 2030,
tracing (2.011 ms) : 1964, 2059
.   : milestone, 2011,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.48 ms [1.468 ms, 1.492 ms] -
appsec 2.412 ms [2.363 ms, 2.461 ms] 931.777 µs (63.0%)
iast 2.192 ms [2.131 ms, 2.253 ms] 711.899 µs (48.1%)
iast_GLOBAL 2.242 ms [2.18 ms, 2.304 ms] 762.004 µs (51.5%)
profiling 2.043 ms [1.992 ms, 2.093 ms] 562.716 µs (38.0%)
tracing 2.017 ms [1.97 ms, 2.065 ms] 537.308 µs (36.3%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.488 ms [1.476 ms, 1.499 ms] -
appsec 2.408 ms [2.359 ms, 2.456 ms] 919.884 µs (61.8%)
iast 2.194 ms [2.133 ms, 2.255 ms] 706.183 µs (47.5%)
iast_GLOBAL 2.225 ms [2.164 ms, 2.286 ms] 737.346 µs (49.6%)
profiling 2.03 ms [1.981 ms, 2.078 ms] 541.796 µs (36.4%)
tracing 2.011 ms [1.964 ms, 2.059 ms] 523.487 µs (35.2%)

@pr-commenter
Copy link

pr-commenter bot commented Jun 24, 2025

Kafka / producer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch mcculls/cleanup-instrumenter-enablement mcculls/instrumenter-index-experiment
git_commit_date 1750792614 1750806252
git_commit_sha 0e4a616 22f46ba
See matching parameters
Baseline Candidate
ci_job_date 1750807497 1750807497
ci_job_id 997131004 997131004
ci_pipeline_id 68681310 68681310
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.25 11.0.25
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.25+9-post-Ubuntu-1ubuntu122.04 11.0.25+9-post-Ubuntu-1ubuntu122.04

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaProduceBenchmark.benchProduce same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce same

@pr-commenter
Copy link

pr-commenter bot commented Jun 24, 2025

Kafka / consumer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch mcculls/cleanup-instrumenter-enablement mcculls/instrumenter-index-experiment
git_commit_date 1750792614 1750806252
git_commit_sha 0e4a616 22f46ba
See matching parameters
Baseline Candidate
ci_job_date 1750807541 1750807541
ci_job_id 997131006 997131006
ci_pipeline_id 68681310 68681310
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.25 11.0.25
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.25+9-post-Ubuntu-1ubuntu122.04 11.0.25+9-post-Ubuntu-1ubuntu122.04

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume unsure
[+1539.435op/s; +15558.679op/s] or [+0.499%; +5.040%]
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume same

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
tag: do not merge Do not merge changes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@mcculls