[CI] FileSettingsServiceIT testNewErrorOnRestartReprocessing failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #25388 / part1
• elasticsearch-periodic #9182 / openjdk17_checkpart1_java-fips-matrix

Reproduction Line:

./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.reservedstate.service.FileSettingsServiceIT.testNewErrorOnRestartReprocessing" -Dtests.seed=D455BF6D184B5757 -Dtests.locale=vi -Dtests.timezone=Iran -Druntime.java=24

Applicable branches:
8.18

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: 
Expected: iterable with items [a string containing "not_cluster_settings"] in any order
     but: not matched: "java.lang.IllegalStateException: Error processing state change request for file_settings, errors: org.elasticsearch.xcontent.XContentParseException: [7:10] [reserved_state_chunk] failed to parse field [state]\n\tat org.elasticsearch.xcontent.ObjectParser.throwFailedToParse(ObjectParser.java:620)\n\tat org.elasticsearch.xcontent.ObjectParser.parseValue(ObjectParser.java:615)\n\tat org.elasticsearch.xcontent.ObjectParser.parseSub(ObjectParser.java:633)\n\tat org.elasticsearch.xcontent.ObjectParser.parse(ObjectParser.java:316)\n\tat org.elasticsearch.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:167)\n\tat org.elasticsearch.xcontent.ConstructingObjectParser.apply(ConstructingObjectParser.java:159)\n\tat org.elasticsearch.reservedstate.service.ReservedClusterStateService.parse(ReservedClusterStateService.java:111)\n\tat org.elast
[truncated]

Issue Reasons:

• [8.18] 2 failures in test testNewErrorOnRestartReprocessing (0.4% fail rate in 546 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch 8.18

Mute Reasons:

• [8.18] 2 failures in test testNewErrorOnRestartReprocessing (0.4% fail rate in 546 executions)

Build Scans:

• elasticsearch-intake #25388 / part1
• elasticsearch-periodic #9182 / openjdk17_checkpart1_java-fips-matrix

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[pxsalehi]

Seems more related to reserved cluster state.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[mark-vieira]

I'm unable to reproduce this. I've unmuted for now to see if we can get any more failures.