Time framed thread-pool utilization

[mhl-b]

This PR changes Thread Pool utilization reporting intervals from dynamic (based on caller's frequency) into static (time frame based).

Originally there was a single consumer of utilization metric that runs on specific interval, so we would calculate utilization on the fly comparing previous invocation time and current. And it worked, kind of... With growing demand on utilization metrics, from shards balancing and allocation, we have new consumers, so this poll mechanism does not scale well. Thread pool needs to create a separate tracker for each caller.

This PR introduces total execution time measurements for thread-pool per time frame. Default frame duration is 30 seconds, but can configured per thread-pool via TaskTrackingConfig. All past time frames have accurate data, because we only update current frame. That means utilization metric can be stale up to duration of the time-frame in worst case. It's a tradeoff between liveliness and accuracy/efficiency. Utilization is not a "real-time" metric and there are no immediate actions on a single utilization metric.

A new FramedTimeTracker incapsulates frame tracking logic, comes with own set of tests. Also added JMH benchmark, advised by Nick, due to use of synchronized methods and risk of high contention. So far I cannot see slowness from the bench, but would like to hear feedback on it. Those numbers are funny looking, but consistent through multiple runs, tracking thread-pool somehow faster with larger number of threads.

Latest result from my machine

Benchmark                           (poolSize)  (tasksNum)  (trackUtilization)  (utilizationIntervalMs)  Mode  Cnt     Score   Error  Units
ThreadPoolUtilizationBenchmark.run           4     1000000               false                       10  avgt        108.888          ms/op
ThreadPoolUtilizationBenchmark.run           4     1000000                true                       10  avgt        420.603          ms/op
ThreadPoolUtilizationBenchmark.run           8     1000000               false                       10  avgt        780.527          ms/op
ThreadPoolUtilizationBenchmark.run           8     1000000                true                       10  avgt        478.278          ms/op
ThreadPoolUtilizationBenchmark.run          16     1000000               false                       10  avgt       1123.840          ms/op
ThreadPoolUtilizationBenchmark.run          16     1000000                true                       10  avgt        506.619          ms/op

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[elasticsearchmachine]

Hi @mhl-b, I've created a changelog YAML for you.