Fleet search using wait_for_checkpoints can fail if the node executing the search is recovering

[fcofdez]

Currently searches can potentially be executed in an INITIALIZING shard:

elasticsearch/server/src/main/java/org/elasticsearch/cluster/routing/OperationRouting.java

Lines 249 to 259 in b90f374

	private ShardIterator shardRoutings(
	IndexShardRoutingTable indexShard,
	@Nullable ResponseCollectorService collectorService,
	@Nullable Map<String, Long> nodeCounts
	) {
	if (useAdaptiveReplicaSelection) {
	return indexShard.activeInitializingShardsRankedIt(collectorService, nodeCounts);
	} else {
	return indexShard.activeInitializingShardsRandomIt();
	}
	}

This means that the shard could be going through recovery and the requested checkpoint through the wait_for_checkpoint might be greater than the current max seq no throwing the following exception:

Cannot wait for unissued seqNo checkpoint [wait_for_checkpoint=1299, max_issued_seqNo=0]

If the shard where the search request gets executed is INITIALIZING we should wait until it moves to STARTED or even consider if we should just avoid executing search requests with wait_for_checkpoints in such shards.

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[fcofdez]

Tagging both teams as it affects mostly the search side of things but the issue comes from the Fleet API.