Skip to content

Support multiple queues in BUILDKITE_QUEUE #205

@joemiller

Description

@joemiller

It would be a great addition if the autoscaler could be configured to listen on multiple queue names.

For instance, I have a queue named private that has been in place for a long time and a lot of repos target this queue. Now, I have introduced private-arm64 as we expand into multi-arch builds.

I have used the BuildkiteAgentTags input on the elastic-ci stack to add additional queue tags to private: queue: private-amd64 and queue: private-x86_64. This mostly works. If a job is created and asks for private-amd64 it will be assigned. But if there are no private queue workers running none will be created because the autoscaler is only watching for BUILDKITE_AGENT: private and not private-amd64.

Also, being able to target queue: private-{{matrix.arch}} is VERY useful to avoid duplication in pipelines.

This means I either need to:

  1. Switch over to queue names such as private-amd64 and private-arm64, which would require updating a lot of repos from private -> private-amd64
  2. Introduce an entirely new private-amd64 queue. Slightly inefficient on resource usage with extra agents hanging around.

I think the ideal would be for the autoscaler to support multiple queue names to trigger a scale event, eg: BUILDKITE_QUEUE: private,private-amd64,private-x86_64. This would also match the behavior of the buildkite-agent which is able to listen for and accept jobs on multiple queue tags.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions