add test scripts and periodic prow job for testing two steps upgrade

[ynfx8]

• add test scripts to perform two steps upgrade - binary version upgrade and emulated version upgrade.
• config a n-minus-1 prow job for the new test. plan to configure n-minus-2 when seeing n-minus-1 tests are stable.

[k8s-ci-robot]

Hi @ynfx8. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ynfx8]

/assign @BenTheElder

[ynfx8]

/assign @siyuanfoundation

[BenTheElder]

/ok-to-test

[BenTheElder]

drop this, we're using a stable tag based on the $date-$commit-hash of test-infra so pulling it every time is pointless and slow (it's a large shared image with e2e tools)

if the other job has it, was should also drop this.

[ynfx8]

https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml#L88

I am seeing this image been used in the job everywhere. Do you mean we don't need the image filed?

[BenTheElder]

I meant the imagePullPolicy: Always, this says "every time you run this pod, resolve and pull the tag" but this tag isn't mutated so pulling it every time even if the image is already available on the host is a waste.

https://kubernetes.io/docs/concepts/containers/images/#image-pull-policy

[ynfx8]

removed imagePullPolicy from all jobs in compatibility-versions-e2e.yaml

[BenTheElder]

this is a lot of bash to review, can we summarize the changes from the file it was modified from?

(or create it in two commits, one to copy the original file, the second to make changes so the diff is reviewable)

[BenTheElder]

[either describing in the pull request discussion or doing the two commit approach would be great, describing inside the file won't make sense, it can only be a point-in-time comparison before they diverge further]

[ynfx8]

I updated

1. the following 3 lines

EMULATED_VERSION_UPGRADE_SCRIPT="${EMULATED_VERSION_UPGRADE_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/emulated-version-upgrade.sh}"
echo "Upgrading cluster with ${EMULATED_VERSION_UPGRADE_SCRIPT}"
bash -x "${EMULATED_VERSION_UPGRADE_SCRIPT}" | tee "${ARTIFACTS}/emulated-upgrade-output.txt"

2. rename upgrade_cluster_components to upgrade_cluster_components_binary_version

[BenTheElder]

OK, it might be easier to share the code instead? we now need to patch both of these and the scripts are getting fairly complicated.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ynfx8
Once this PR has been reviewed and has the lgtm label, please ask for approval from bentheelder. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• config/OWNERS
• experiment/compatibility-versions/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ynfx8]

/test pull-test-infra-verify-lint