Skip to content

CLOUDP-320704: Fix clean up flow in generator to acommodate for already deleted clusters and projects #3918

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 10 commits into from
May 30, 2025
Merged

CLOUDP-320704: Fix clean up flow in generator to acommodate for already deleted clusters and projects #3918

merged 10 commits into from
May 30, 2025

Conversation

@blva
Copy link
Collaborator

@blva blva commented May 28, 2025 •
edited
Loading

Proposed changes

Jira ticket: CLOUDP-320704

  • Snapshot tests don't work well with two generators being called
  • Patch test
  • To fix this and also keep backup tests working on waterfall, I'm fixing the original issue w/ NOT_FOUND errors, see the errors below
  • The tests were not catching the stderr when cleaning up resources, so it woudl continue to retry and fail at the end. I've updated the cluster deletion and project deletion to also get the stderr

after

[2025/05/29 12:18:42.310]     backup_restores_test.go:61: project "6838499c7ef77a3892ea21dd" successfully deleted
[2025/05/29 12:21:44.142]     backup_restores_test.go:63: project "68384b5d7ef77a3892ea2518" successfully deleted
[2025/05/29 12:21:44.142]     atlas_e2e_test_generator.go:191: Cluster cleanup for project "68384b5d7ef77a3892ea2518"
[2025/05/29 12:21:44.270]     atlas_e2e_test_generator.go:347: Cluster "backupRestores2-946-e2f" was already deleted
[2025/05/29 12:21:44.270]     atlas_e2e_test_generator.go:191: Project cleanup "68384b5d7ef77a3892ea2518"
[2025/05/29 12:21:44.386]     atlas_e2e_test_generator.go:302: project "68384b5d7ef77a3892ea2518" successfully deleted
[2025/05/29 12:21:44.386]     atlas_e2e_test_generator.go:191: Cluster cleanup for project "68384b5d7ef77a3892ea2518"
[2025/05/29 12:21:44.514]     atlas_e2e_test_generator.go:347: Cluster "backupRestores2-946-e2f" was already deleted
[2025/05/29 12:21:44.514]     atlas_e2e_test_generator.go:191: Project cleanup "68384b5d7ef77a3892ea2518"
[2025/05/29 12:21:44.642]     atlas_e2e_test_generator.go:302: project "68384b5d7ef77a3892ea2518" successfully deleted

before

 [2025/03/28 07:16:39.950]     atlas_e2e_test_generator_test.go:117: projectID=67e67a9d01342155824fb611
 [2025/03/28 07:16:39.950]     atlas_e2e_test_generator_test.go:117: projectName=backupRestores-729-4af1c514defbb8f867a656a8ef4b61a1c2e185d7
 [2025/03/28 07:16:39.950]     backup_restores_test.go:37: clusterName=backupRestores-374-4af1c514defbb8f867a656a8ef4b61a1c2e185d7
 [2025/03/28 07:16:39.950]     atlas_e2e_test_generator_test.go:117: projectID=67e67c71f54d434adcb5ce09
 [2025/03/28 07:16:39.950]     atlas_e2e_test_generator_test.go:117: projectName=backupRestores2-288-4af1c514defbb8f867a656a8ef4b61a1c2e185d7
 [2025/03/28 07:16:39.950]     backup_restores_test.go:46: clusterName=backupRestores2-481-4af1c514defbb8f867a656a8ef4b61a1c2e185d7
 [2025/03/28 07:16:39.950]     atlas_e2e_test_generator_test.go:117: Cluster cleanup "67e67c71f54d434adcb5ce09"
 [2025/03/28 07:16:39.950] Error: https://cloud-dev.mongodb.com/api/atlas/v2/groups/67e67c71f54d434adcb5ce09/clusters/backupRestores2-481-4af1c514defbb8f867a656a8ef4b61a1c2e185d7 GET: HTTP 404 Not Found (Error code: "CLUSTER_NOT_FOUND") Detail: No cluster named backupRestores2-481-4af1c514defbb8f867a656a8ef4b61a1c2e185d7 exists in group 67e67c71f54d434adcb5ce09. Reason: Not Found. Params: [backupRestores2-481-4af1c514defbb8f867a656a8ef4b61a1c2e185d7 67e67c71f54d434adcb5ce09]
 [2025/03/28 07:16:39.950]     atlas_e2e_test_generator_test.go:175: project "67e67c71f54d434adcb5ce09" successfully deleted
 [2025/03/28 07:16:39.950]     atlas_e2e_test_generator_test.go:117: Cluster cleanup "67e67c71f54d434adcb5ce09"
 [2025/03/28 07:16:39.950] Error: https://cloud-dev.mongodb.com/api/atlas/v2/groups/67e67c71f54d434adcb5ce09/clusters/backupRestores2-481-4af1c514defbb8f867a656a8ef4b61a1c2e185d7 DELETE: HTTP 404 Not Found (Error code: "GROUP_NOT_FOUND") Detail: No group with ID 67e67c71f54d434adcb5ce09 exists. Reason: Not Found. Params: [67e67c71f54d434adcb5ce09]
 [2025/03/28 07:16:39.950]     atlas_e2e_test_generator_test.go:238: unexpected error deleting cluster: error deleting cluster  (exit status 1): 
 [2025/03/28 07:16:39.950] Error: https://cloud-dev.mongodb.com/api/atlas/v2/groups/67e67c71f54d434adcb5ce09 DELETE: HTTP 404 Not Found (Error code: "GROUP_NOT_FOUND") Detail: No group with ID 67e67c71f54d434adcb5ce09 exists. Reason: Not Found. Params: [67e67c71f54d434adcb5ce09], BadRequestDetail: 
 [2025/03/28 07:16:39.950]     atlas_e2e_test_generator_test.go:175: 1/10 attempts - trying again in 1 seconds: unexpected error while deleting the project "67e67c71f54d434adcb5ce09":  ( (exit status 1))
 [2025/03/28 07:16:39.950] Error: https://cloud-dev.mongodb.com/api/atlas/v2/groups/67e67c71f54d434adcb5ce09 DELETE: HTTP 404 Not Found (Error code: "GROUP_NOT_FOUND") Detail: No group with ID 67e67c71f54d434adcb5ce09 exists. Reason: Not Found. Params: [67e67c71f54d434adcb5ce09], BadRequestDetail: 
 [2025/03/28 07:16:39.950]     atlas_e2e_test_generator_test.go:175: 2/10 attempts - trying again in 2 seconds: unexpected error while deleting the project "67e67c71f54d434adcb5ce09":  ( (exit status 1))
 [2025/03/28 07:16:39.950] Error: https://cloud-dev.mongodb.com/api/atlas/v2/groups/67e67c71f54d434adcb5ce09 DELETE: HTTP 404 Not Found (Error code: "GROUP_NOT_FOUND") Detail: No group with ID 67e67c71f54d434adcb5ce09 exists. Reason: Not Found. Params: [67e67c71f54d434adcb5ce09], BadRequestDetail: 
 [2025/03/28 07:16:39.950]     atlas_e2e_test_generator_test.go:175: 3/10 attempts - trying again in 4 seconds: unexpected error while deleting the project "67e67c71f54d434adcb5ce09":  ( (exit status 1))
 [2025/03/28 07:16:39.950] Error: https://cloud-dev.mongodb.com/api/atlas/v2/groups/67e67c71f54d434adcb5ce09 DELETE: HTTP 404 Not Found (Error code: "GROUP_NOT_FOUND") Detail: No group with ID 67e67c71f54d434adcb5ce09 exists. Reason: Not Found. Params: [67e67c71f54d434adcb5ce09], BadRequestDetail: 
 [2025/03/28 07:16:39.950]     atlas_e2e_test_generator_test.go:175: 4/10 attempts - trying again in 8 seconds: unexpected error while deleting the project "67e67c71f54d434adcb5ce09":  ( (exit status 1))
 [2025/03/28 07:16:39.950] Error: https://cloud-dev.mongodb.com/api/atlas/v2/groups/67e67c71f54d434adcb5ce09 DELETE: HTTP 404 Not Found (Error code: "GROUP_NOT_FOUND") Detail: No group with ID 67e67c71f54d434adcb5ce09 exists. Reason: Not Found. Params: [67e67c71f54d434adcb5ce09], BadRequestDetail: 
 [2025/03/28 07:16:39.950]     atlas_e2e_test_generator_test.go:175: 5/10 attempts - trying again in 16 seconds: unexpected error while deleting the project "67e67c71f54d434adcb5ce09":  ( (exit status 1))
 [2025/03/28 07:16:39.950] Error: https://cloud-dev.mongodb.com/api/atlas/v2/groups/67e67c71f54d434adcb5ce09 DELETE: HTTP 404 Not Found (Error code: "GROUP_NOT_FOUND") Detail: No group with ID 67e67c71f54d434adcb5ce09 exists. Reason: Not Found. Params: [67e67c71f54d434adcb5ce09], BadRequestDetail: 
 [2025/03/28 07:16:39.950]     atlas_e2e_test_generator_test.go:175: 6/10 attempts - trying again in 32 seconds: unexpected error while deleting the project "67e67c71f54d434adcb5ce09":  ( (exit status 1))
 [2025/03/28 07:16:39.950] Error: https://cloud-dev.mongodb.com/api/atlas/v2/groups/67e67c71f54d434adcb5ce09 DELETE: HTTP 404 Not Found (Error code: "GROUP_NOT_FOUND") Detail: No group with ID 67e67c71f54d434adcb5ce09 exists. Reason: Not Found. Params: [67e67c71f54d434adcb5ce09], BadRequestDetail: 
 [2025/03/28 07:16:39.950]     atlas_e2e_test_generator_test.go:175: 7/10 attempts - trying again in 64 seconds: unexpected error while deleting the project "67e67c71f54d434adcb5ce09":  ( (exit status 1))
 [2025/03/28 07:16:39.950] Error: https://cloud-dev.mongodb.com/api/atlas/v2/groups/67e67c71f54d434adcb5ce09 DELETE: HTTP 404 Not Found (Error code: "GROUP_NOT_FOUND") Detail: No group with ID 67e67c71f54d434adcb5ce09 exists. Reason: Not Found. Params: [67e67c71f54d434adcb5ce09], BadRequestDetail: 
 [2025/03/28 07:16:39.950]     atlas_e2e_test_generator_test.go:175: 8/10 attempts - trying again in 128 seconds: unexpected error while deleting the project "67e67c71f54d434adcb5ce09":  ( (exit status 1))
 [2025/03/28 07:16:39.950] Error: https://cloud-dev.mongodb.com/api/atlas/v2/groups/67e67c71f54d434adcb5ce09 DELETE: HTTP 404 Not Found (Error code: "GROUP_NOT_FOUND") Detail: No group with ID 67e67c71f54d434adcb5ce09 exists. Reason: Not Found. Params: [67e67c71f54d434adcb5ce09], BadRequestDetail: 
 [2025/03/28 07:16:39.950]     atlas_e2e_test_generator_test.go:175: 9/10 attempts - trying again in 256 seconds: unexpected error while deleting the project "67e67c71f54d434adcb5ce09":  ( (exit status 1))
 [2025/03/28 07:16:39.950] Error: https://cloud-dev.mongodb.com/api/atlas/v2/groups/67e67c71f54d434adcb5ce09 DELETE: HTTP 404 Not Found (Error code: "GROUP_NOT_FOUND") Detail: No group with ID 67e67c71f54d434adcb5ce09 exists. Reason: Not Found. Params: [67e67c71f54d434adcb5ce09], BadRequestDetail: 
 [2025/03/28 07:16:39.950]     atlas_e2e_test_generator_test.go:175: 10/10 attempts - trying again in 512 seconds: unexpected error while deleting the project "67e67c71f54d434adcb5ce09":  ( (exit status 1))
 [2025/03/28 07:16:39.950]     atlas_e2e_test_generator_test.go:175: we could not delete the project "67e67c71f54d434adcb5ce09"

Checklist

  • I have signed the MongoDB CLA
  • I have added tests that prove my fix is effective or that my feature works
  • I have added any necessary documentation in document requirements section listed in CONTRIBUTING.md (if appropriate)
  • I have addressed the @mongodb/docs-cloud-team comments (if appropriate)
  • I have updated test/README.md (if an e2e test has been added)
  • I have run make fmt and formatted my code

Further comments

blva added 2 commits May 28, 2025 13:58
@blva
Revert "CLOUDP-309403: Flaky Test: mongodb-atlas-cli-master-github.co…
2951dd7 
…m_mongodb_mongodb-atlas-cli_atlascli_test_e2e.TestRestores/atlas_backups_restores_e2e/e2e_atlas_backups (#3906)"

This reverts commit ad444c4.
@blva
fix errors when group is not found and add missing clean up
766ba3d
@blva blva force-pushed the CLOUDP-320704 branch from 7c8ac2c to 766ba3d Compare May 28, 2025 13:04
blva
blva commented May 28, 2025
View reviewed changes
test/e2e/backuprestores/backup_restores_test.go
g.GenerateProjectAndCluster("backupRestores")
require.NotEmpty(t, g.ClusterName)

g2 := internal.NewAtlasE2ETestGenerator(t, internal.WithSnapshot(), internal.WithBackup())
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reverts 2951dd7

@blva blva changed the title CLOUDP-320704: Remove snapshots from g2 in backup snapashot test CLOUDP-320704: Fix clean up flow in generator to acommodate for already deleted clusters and projects May 28, 2025
@blva blva marked this pull request as ready for review May 29, 2025 14:45
@blva blva requested a review from a team as a code owner May 29, 2025 14:45
gssbzn
gssbzn reviewed May 29, 2025
View reviewed changes
@blva blva requested a review from gssbzn May 29, 2025 14:58
@blva blva enabled auto-merge (squash) May 29, 2025 15:50
cveticm
cveticm reviewed May 30, 2025
View reviewed changes
@blva blva requested a review from cveticm May 30, 2025 11:02
@github-actions
Copy link
Contributor

Coverage Report 📈

Branch Commit Coverage
master e2f1531 25.1%
CLOUDP-320704 b383807 25.1%
Difference 0%

cveticm
cveticm approved these changes May 30, 2025
View reviewed changes
Copy link
Collaborator

@cveticm cveticm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@blva blva merged commit cb0d2aa into master May 30, 2025
21 of 22 checks passed
@blva blva deleted the CLOUDP-320704 branch May 30, 2025 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cveticm cveticm cveticm approved these changes

@gssbzn gssbzn Awaiting requested review from gssbzn

Assignees

No one assigned

Labels

atlascli

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@blva @gssbzn @cveticm