Scaling on Waiting has unexpected behavior

[baderbuddy]

Because every job behind a block is considered waiting, if you turn on scaling based on waiting it could cause a lot of unexpected scaling. In our organization we do a lot of block steps for optional things like rollbacks, that most of hte tiem are never executed, so we have plenty of jobs that are stuck in a waiting stage that never get cleaned up.

Having a little more logic in there to exclude older blocked steps, or jobs behind a block would make this option much more useful.

[DrJosh9000]

Thanks @baderbuddy, that's a good point.

Currently the metrics endpoint only exposes a single "waiting" metric which includes all kinds of waiting jobs. But I think that even if we did break the metric down into different categories (such as behind a block vs not), there might not be a common rule that we could apply to all kinds of pipeline.

[joemiller]

Just got bit by this too. Disabled Scale out for waiting for now.