Start Publishing Staging Images to Artifact Registry

[upodroid]

Part of: #1343

Notes: https://gist.github.com/upodroid/a33723a7e1abc5e9c6fabc6b07e7aac0

When images are built after a PR is merged, they need to be pushed to Artifact Registry(AR).

Unlike Google Container Registry(GCR), you don't need a separate GCP project per staging project. The permission boundary is at the repository level instead of the project. This allows multiple registries to be created per project. Therefore, we will create a new project, start publishing images to it and delete the other projects after the transition.

We will need to do the following:
- create a new GCP project (k8s-artifacts-staging)
- create a docker image repository per project
- modify the jobs to push images to prow.
- Provision the infra with terraform. There is a separate issue open to deploy changes to terraform via Prow on merges.

Prod changes need to be done via the shell scripts which makes deploying staging via terraform kind of pointless.

Open questions:

• Do we need to backfill the staging repository via gcrane?

@puerco

/area artifacts
/priority important-soon
/area release-eng

[upodroid]

After #3968 is merged, we need to do a few things:

• repos that use ko to build images(k-sigs/bom,etc), modify KO_DOCKER_REPO variable to us-docker.pkg.dev/k8s-staging-bom/images/bom as an example
• Modify image-builder to tweak the cloudbuild.yaml file

Bit unsure about other things.

[upodroid]

Spoke about this at the sig-testing meeting. Will push it out to the future when GCR deprecation is actually announced by Google.

May want to standardize on Aritfact Registry eventually, but would require tracking down a number of staging registries and changing team push processes (with unclear payoff right now)

/priority backlog
/sig testing

[ameukam]

/milestone v1.26