Fix TestKubeletConfigRemediation unstable issue by Anna-Koudelkova · Pull Request #1080 · ComplianceAsCode/compliance-operator

Anna-Koudelkova · 2026-02-06T15:21:36Z

Note:
As disscussed on Friday with the dev team and today with Xiaojie, we do not need the specific image the test case was using because we now ship these rules as part of the ocp4 profile bundle. Also, based on Xiaojie's suggestion I have replaced the overly complicated defer logic for unapplying the remediation with an implementation of already existing functions. Thanks for the feedback!

Changes in this PR:

Add cleanup to test case TestKubeletConfigRemediation - the test case itself is not failing much in our serial job run, but the subsequent test, TestSuspendScanSetting, is failing quite often. I think the root cause lies in TestKubeletConfigRemediation, that applied remediation and did not unapply it as well as it did not wait for the scan and suite to be deleted.

To fix that, I have added func WaitForComplianceSuiteDeletion() to common.go and make minor modification in the test itself.

I hope this works, TestKubeletConfigRemediation is problematic, Xiaojie is trying a different approach in PR #1055 and in some of the PRs any few of my local tries, the test case fails on the result of the suite being NOT-APPLICABLE, which is a whole different story.

When running test case in my local cluster, often after PASSing runs my environment ended up with one worker node degraded. This is apparently caused by removing the e2e label from the worker node and immediately deleting the e2e machineconfigpool before the node fully reboots as part of worker node pool. I have added a func WaitForMachineConfigPoolUpdated() into common.go and call for this function into the main_entry.go file to the teardown phase.
This is not important for our CI runs, but it greatly improves quality of my experience when I run these tests locally and saves my sanity.

At least I did not harm to the TestKubeletConfigRemediation:

2026/02/06 15:55:25 All scans in ComplianceSuite have finished (kubelet-remediation-test-suite-node)
2026/02/06 15:55:25 scan re-run has finished
--- PASS: TestKubeletConfigRemediation (373.79s)
PASS
...
ok  	github.com/ComplianceAsCode/compliance-operator/tests/e2e/serial	658.249s

openshift-ci · 2026-02-06T15:21:42Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Anna-Koudelkova

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [Anna-Koudelkova]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

github-actions · 2026-02-06T15:29:11Z

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1080-00de17c89de971cd81bec1ccef51093501797691

github-actions · 2026-02-09T14:43:53Z

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1080-3beaa2c3dc1b014fd6da7e9f74db1c28d829a6e7

rhmdnd

One comment about using t.Fatal versus t.Log (primarily noting an inconsistency and wanting to make sure I understand it).

Otherwise, this looks like a much more stable approach to the serial tests.

rhmdnd · 2026-03-05T14:31:03Z

tests/e2e/framework/common.go

+			}
+		}
+
+		log.Printf("Machine Config Pool %s has not finished updating yet... retrying\n", poolName)


Super minor nit: could add the c.Status here to see that in the log as it rolls through the update (kinda like we do when waiting for scan statuses).

rhmdnd · 2026-03-05T14:34:02Z

tests/e2e/framework/common.go

+		return false, nil
+	})
+	if err != nil {
+		return fmt.Errorf("ComplianceSuite %s may not have been fully cleaned up: %w", name, err)


Does this output the length of time we waited for the suite to cleanup? That might be good information to log if we're not already through the err.

rhmdnd · 2026-03-05T14:37:43Z

tests/e2e/serial/main_test.go

+		// Finally clean up by removing the remediation and waiting for the nodes to reboot one more time
+		err = f.UnApplyRemediationAndCheck(f.OperatorNamespace, remName, "worker")
+		if err != nil {
+			t.Fatal(err)


Above - we're using t.Log instead of t.Fatal. Do we want those to be consistent? Or was there a reason why you wanted to use t.Fatal here and not above?

For context - I think t.Fatal is the correct usage, since it will fail the test if it cannot cleanup after itself.

There is no reason for this - it is caused by me touching an already existing testcase while being assisted by Cursor. I have fixed it to t.Fatal so it is consistent

rhmdnd · 2026-03-05T14:54:52Z

tests/e2e/serial/main_test.go

+	}()

 	// Ensure that all the scans in the suite have finished and are marked as Done
 	err = f.WaitForSuiteScansStatus(f.OperatorNamespace, suiteName, compv1alpha1.PhaseDone, compv1alpha1.ResultNonCompliant)


This actually came back as Compliant which is what's failing the serial CI. We might need to either use the variadic version of this wait function, or find some other rules that will fail (which might be necessary if we're testing remediations).

I have not found any kubelet rules that are not passing by default now (it is possible I have not tried all the options). The only way how can I get the test case to pass now in my local cluster is by reverting back to using the brokenContentImagePath and creating a profilebundle. That is why I have uploaded this version here, so I can see what is going to happen in the CI now..
I am frankly quite confused by this.

We might need to adjust this by changing a rule like https://github.com/ComplianceAsCode/content/blob/master/applications/openshift/kubelet/kubelet_eviction_thresholds_set_hard_imagefs_inodesfree/rule.yml#L77 and its remediation value variable default https://github.com/ComplianceAsCode/content/blob/master/applications/openshift/kubelet/var_kubelet_evictionhard_imagefs_inodesfree.var to a fixed value, making it fail by default and making the test work in that way.

github-actions · 2026-03-09T20:43:32Z

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1080-9ad1b1d737a8cbcb055877a2db32e094832b5fb9

github-actions · 2026-03-10T18:35:14Z

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1080-dfc0fb2c78500c9773bcf2a2ce801d134d18a584

github-actions · 2026-03-10T18:40:24Z

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1080-521e12770fc51e293ab24dd1c599c436ca543b3d

github-actions · 2026-03-10T19:52:03Z

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1080-88210258ec7fe770c60de3bed358d3c650ce2b4e

github-actions · 2026-03-12T14:44:47Z

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1080-c4d183458a50ef93c2a5cd43aad56fee4a27ac22

openshift-ci · 2026-03-12T17:07:54Z

@Anna-Koudelkova: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-rosa	`c4d1834`	link	true	`/test e2e-rosa`
ci/prow/e2e-aws-serial	`c4d1834`	link	true	`/test e2e-aws-serial`
ci/prow/e2e-aws-serial-arm	`c4d1834`	link	true	`/test e2e-aws-serial-arm`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci bot requested review from mrogers950 and taimurhafeez February 6, 2026 15:21

openshift-ci bot added the approved label Feb 6, 2026

Anna-Koudelkova force-pushed the ci_cleanup branch from 00de17c to 3beaa2c Compare February 9, 2026 14:35

rhmdnd requested changes Mar 5, 2026

View reviewed changes

openshift-ci bot assigned rhmdnd Mar 5, 2026

Anna-Koudelkova force-pushed the ci_cleanup branch from 3beaa2c to 9ad1b1d Compare March 9, 2026 20:35

Anna-Koudelkova force-pushed the ci_cleanup branch 2 times, most recently from dfc0fb2 to 521e127 Compare March 10, 2026 18:32

Anna-Koudelkova force-pushed the ci_cleanup branch from 521e127 to 8821025 Compare March 10, 2026 19:44

Fix TestKubeletConfigRemediation issue

c4d1834

Anna-Koudelkova force-pushed the ci_cleanup branch from 8821025 to c4d1834 Compare March 12, 2026 14:37

Conversation

Anna-Koudelkova commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Feb 6, 2026

Uh oh!

github-actions bot commented Feb 6, 2026

Uh oh!

github-actions bot commented Feb 9, 2026

Uh oh!

rhmdnd left a comment

Choose a reason for hiding this comment

Uh oh!

rhmdnd Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

rhmdnd Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

rhmdnd Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Anna-Koudelkova Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

rhmdnd Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Anna-Koudelkova Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Vincent056 Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 9, 2026

Uh oh!

github-actions bot commented Mar 10, 2026

Uh oh!

github-actions bot commented Mar 10, 2026

Uh oh!

github-actions bot commented Mar 10, 2026

Uh oh!

github-actions bot commented Mar 12, 2026

Uh oh!

openshift-ci bot commented Mar 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Anna-Koudelkova commented Feb 6, 2026 •

edited

Loading