[CI] AddIndexBlockRollingUpgradeIT testAddBlock {upgradedNodes=3} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #25446 / 8.18.4_bwc-snapshots
• elasticsearch-pull-request #80577 / 8.18.4_part_6_bwc-snapshots
• elasticsearch-intake #25399 / 8.18.4_bwc-snapshots

Reproduction Line:

./gradlew ":qa:rolling-upgrade:v8.18.4#bwcTest" -Dtests.class="org.elasticsearch.upgrades.AddIndexBlockRollingUpgradeIT" -Dtests.method="testAddBlock {upgradedNodes=3}" -Dtests.seed=875A9395504001AA -Dtests.bwc=true -Dtests.locale=fr-WF -Dtests.timezone=Universal -Druntime.java=24

Applicable branches:
8.19

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.Exception: Test abandoned because suite timeout was reached.

Issue Reasons:

• [8.19] 2 consecutive failures in step 8.18.4_bwc-snapshots
• [8.19] 3 failures in test testAddBlock {upgradedNodes=3} (0.7% fail rate in 440 executions)
• [8.19] 2 failures in step 8.18.4_bwc-snapshots (100.0% fail rate in 2 executions)
• [8.19] 2 failures in pipeline elasticsearch-intake (50.0% fail rate in 4 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[kingherc]

From one of the artifact I see:

./qa/mixed-cluster/build/testclusters/v8.18.4-3/logs/v8.18.4.log:java.lang.AssertionError: soft deletes must be enabled in version 8525000

This seems worrisome as it relates to the recent N-2 effort (e.g., #120526 and #119743 ). I've marked it as medium risk, but I wonder if it might be worth of higher risk. @tlrx , @henningandersen could you comment?