Introduce kata containerized testing tool

[Ankita13-code]

Merge Checklist

• [] Followed patch format from upstream recommendation: https://github.com/kata-containers/community/blob/main/CONTRIBUTING.md#patch-format
  • Included a single commit in a given PR - at least unless there are related commits and each makes sense as a change on its own.
• Aware about the PR to be merged using "create a merge commit" rather than "squash and merge" (or similar)
• The upstream/missing label (or upstream/not-needed) has been set on the PR.

Summary

Test Methodology

[Redent0r]

Could you add an option to format the results in junit format? In the pipelines, that allows us to to upload the results to ADO and visualize the results like this . May look into https://github.com/gotestyourself/gotestsum?tab=readme-ov-file#junit-xml-output

[sprt]

Agreed - is JSON ever helpful?

[Ankita13-code]

@Redent0r @sprt I've added a method to save the results in Junit format instead of Json. Initially I added Json format for results since I was testing manually. But yes, from onboarding this to a pipeline point of view, Junit is definitely better

[ms-mahuber]

Should we not fail here?

[ms-mahuber]

Imagine we may have a test eg. for memory that may expect different values based on what runtime handler the container runs on, and imagine we wanted to only author a single pod YAML that would work for both kata and CC runtime handler? can we explain or think how this can be best implemented? or would it be better to author different pod manifests?

[sprt]

@ms-mahuber Seems like we can have the test infra pass different params to the container based on the test infra environment?

[Ankita13-code]

@ms-mahuber I was thinking of using a single config file which can be mounted as a volume for solving this problem instead of just passing the env variables or cmd line arguments.

We could have a single config.yaml with different environment configurations for the Host VM and the Guest VM -

environments:
  host:
    tests:
      - name: "cpu"
        expected_values:
          expected_vcpu_count: 4
      - name: "memory"
        expected_values:
          expected_memory_mb: 8192
  
  uvm:
    tests:
      - name: "cpu"
        expected_values:
          expected_vcpu_count: 2
      - name: "memory"
        expected_values:
          expected_memory_mb: 4096

Then based on the runtime handler, we can load the appropriate environment config. This would help us have a single pod manifest for both the scenarios and add/modify the test parameters in a cleaner way.

[sprt]

I wouldn't tightly couple the test infra and code this way - this seems more appropriate #312 (comment)

[ms-mahuber]

@sprt , @Ankita13-code - I would either trend towards the use of CLI as suggested above, or think of defining and adding a single config.yaml in the form of a container image layer. Assuming the container gets dynamically built in a test framework, and is thus "volatile", we can bake the configuration into a container image layer itself.

[sprt]

Gotcha - if we plan to upstream this, my vote goes to CLI (otherwise it seems much harder to justify/generalize that config file)

[ms-mahuber]

If the container would implement about 30 tests, would this still scale?

[sprt]

I see two aspects to this:

1. Passing the test parameters that correspond to a specific environment (e.g. footprint == X)
2. Modifying the parameters based on the environment (e.g. for CoCo, footprint = X+1, for Kata == X).

(2) can be done with a file but I think that should be handled by the consumer of the container, otherwise we introduce coupling between the environments and the test container. For example, if this file is in the container image instead, downstreams will have conflicts.

Now (1) should be simpler to implement in the container. We could still use a file here but at a smaller scale it seems CLI args are more appropriate. Regardless, the consumer has to specify these args somehow.

[sprt]

This target should require the binary

[Ankita13-code]

Updated!

[sprt]

Better to put a separate .PHONY next to each target (otherwise easy to miss this)

[Ankita13-code]

Updated!

[sprt]

UVM is MSFT lingo

Suggested change

	#### UVM
	#### Guest VM

[Ankita13-code]

Updated!

[sprt]

Are these parameters to the container because we expect the test infra to specify different parameters depending on the environment?

Regardless, I strongly suggest passing actual CLI parameters rather than environment variables. This will:

• Force us to pass parameters around explicitly in the Go code and make it more readable
• Make it much cleaner and easier to parse the parameters based on type and such - Go has builtin facilities for this

This should look something like:

Suggested change

	env:
	- name: ENABLED_TESTS
	value: "cpu,memory"
	- name: TEST_CPU_EXPECTED_VCPU_COUNT
	value: "2"
	- name: TEST_MEMORY_EXPECTED_MEMORY_MB
	value: "4096"
	args: [
	"--test", "cpu",
	"--test", "memory",
	"--test.cpu.expected_vcpus", "2",
	"--test.memory.expected_memory_mb", "4096",
	]

[ms-mahuber]

Good suggestion +1

[sprt]

Any reason not to do this in the Makefile?

[sprt]

I strongly suggest that the pipeline should build the test container every time it is run, rather than having a pre-built container that we upload manually. This way:

• We reduce the number of manual steps we have to do to maintain/publish the container
• It becomes much easier to add a new test and validate it

[ms-mahuber]

Can do as follow-up?

[sprt]

Isn't this already done in main()?

[Ankita13-code]

Removed from the Dockerfile

[sprt]

@ms-mahuber Seems like we can have the test infra pass different params to the container based on the test infra environment?

[sprt]

Agreed - is JSON ever helpful?

[sprt]

I would extract this list of tests to its own file, this way it's trivial to modify.

[sprt]

In Go all doc comments are supposed to start with the name you're documenting, e.g. // NewFramework creates a new testing framework.

[Ankita13-code]

Updated the docs!

[sprt]

nit: This should be a variable passed to the Dockerfile

[sprt]

We shouldn't automatically clean, in case we can just reuse the binary and save time - go will check that for us.

Suggested change

	all: clean build docker
	all: build docker

[sprt]

I wouldn't tightly couple the test infra and code this way - this seems more appropriate #312 (comment)