[ML] Prevent the trained model deployment memory estimation from double-counting allocations.

[valeriy42]

We observed that update of the number of deployment allocation lead to double-accounting of the function StartTrainedModelDeploymentAction.estimateMemoryUsageBytes. Further investigation showed that this bug was introduced in #104260.

This PR reverts changes from #104260 and refactors memory calculation in ML inference assignment planning to improve testability and prevent memory accounting issues.

Main Changes

• Refactored memory calculation in AssignmentPlan.Builder:
  • Added assignModelToNodeAndAccountForCurrentAllocations() in AssignmentPlan.java that handles both new and current allocations in a single call
  • Moved memory accounting into the Builder class to reduce code duplication and potential bugs
  • Used by ZoneAwareAssignmentPlanner and TrainedModelAssignmentRebalancer for consistent memory handling
• Added dependency injection in Deployment for memory estimation functional
• Created test to verify correct memory accounting when scaling allocations from 3 to 4 and confirmed no double-counting of memory during allocation scaling

[elasticsearchmachine]

Hi @valeriy42, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[jan-elastic]

I don't understand this. Why call:

• assignModelToNode for the new allocations; and
• accountMemory for the old ones?

I guess I also don't really understand what the state of AssignmentPlan exactly contains.

Isn't the old already accounted for? And what about the cores?

[jan-elastic]

Furthermore, it feels like this shouldn't be doing that much, so I don't get why it's 500+ lines of similar confusing methods...