Skip to content

refactor: Refactor ci and multi cluster cicd #6505

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 44 commits into from
Apr 22, 2025
Merged

refactor: Refactor ci and multi cluster cicd #6505

merged 44 commits into from
Apr 22, 2025

Conversation

kartik-579
Copy link
Member

@kartik-579 kartik-579 commented Apr 8, 2025 •
edited by Ash-exp
Loading

Description

Fixes https://github.com/devtron-labs/sprint-tasks/issues/692

Checklist:

  • The title of the PR states what changed and the related issues number (used for the release note).
  • Does this PR requires documentation updates?
  • I've updated documentation as required by this PR.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have tested it for all user roles.
  • I have added all the required unit/api test cases.

Does this PR introduce a user-facing change?

kartik-579 and others added 30 commits April 1, 2025 19:38
@kartik-579
changes around ciHandler
237d8d0
@kartik-579
common constants create
7e7928d
@kartik-579
ci service and trigger service creation
d9dcd09
@kartik-579
wire file missed
3bf9cd7
@kartik-579
trigger service moved completely
3d23b35
@Ash-exp
feat: added ucid validation for ci/cd workflow status update event
eb5eeeb
@Ash-exp
telemetry svc refactoring
3237725
@Ash-exp
updated wire gen files
8525719
@Ash-exp
fix: CheckIfReTriggerRequired condition and CdHandlerImpl.UpdateWorkf…
44abd3f 
…low method
@Ash-exp
fix: workflow message update
30fd819
@Ash-exp
updated import alias
8ca9d53
@Ash-exp
fix: ci/ cd status update inconsistencies
4e73878
@Ash-exp
Merge remote-tracking branch 'origin/develop' into fix-multi-cluster-…
2e1d236 
…ci-cd-status
@Ash-exp
fix: checkForOptOut method error handling
b49821e
@Ash-exp
fix: panic handling
186394a
@Ash-exp
fix: updated DevtronAdministratorInstanceLabelKey for system executor
48773ff
@Ash-exp
fix: reverted base64 encoding
756be67
@Ash-exp
fix: ci/ cd workflow msg update
3c42cd7
@Ash-exp
fix: ci/ cd workflow msg update
23befaa
@Ash-exp
matrics for retrigger failed event
87a2f25
@Ash-exp
updated logger
2a76464
@kartik-579
ci service ent oss completed
e9ac531
@kartik-579
added app to subdirectory for categorising entity
f6391d7
@kartik-579
cd handler and ci handle extracted, executor workflow service executed
414712a
@kartik-579
workflow service ent change
1a2de64
@kartik-579
workflow service ent change
3120992
@kartik-579
Redundant import removed
7ae5b3d
@kartik-579
rename pkg/
5ab4dfb
@kartik-579
reverted pkg categorisation
6bd6815
@Ash-exp
chore: posthug refactoring and const renamed
b54bb9e
@kartik-579 kartik-579 changed the base branch from main to develop April 8, 2025 10:21
github-advanced-security[bot]
github-advanced-security bot found potential problems Apr 8, 2025
View reviewed changes
pkg/build/trigger/HandlerService.go
}
workflowRequest, err = impl.updateWorkflowRequestWithBuildCacheData(workflowRequest, scope)
if err != nil {
impl.Logger.Errorw("error, updateWorkflowRequestWithBuildCacheData", "workflowRequest", workflowRequest, "err", err)

Check failure

Code scanning / CodeQL

Clear-text logging of sensitive information

[Sensitive data returned by an access to Password](1) flows to a logging call. [Sensitive data returned by an access to Password](2) flows to a logging call. [Sensitive data returned by an access to DockerPassword](3) flows to a logging call. [Sensitive data returned by an access to SecretKey](4) flows to a logging call. [Sensitive data returned by an access to BlobStorageS3SecretKey](5) flows to a logging call. [Sensitive data returned by an access to Password](6) flows to a logging call.

Copilot Autofix

AI about 2 months ago

To fix the problem, we need to ensure that sensitive information such as passwords is not logged in clear text. Instead, we can either omit the sensitive information from the logs or obfuscate it. In this case, we will omit the password from the log message to prevent any potential security risks.

We will modify the logging statement on line 765 to exclude the Password field from the workflowRequest object. This change will be made in the file pkg/build/trigger/HandlerService.go.

Suggested changeset 1
pkg/build/trigger/HandlerService.go

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/pkg/build/trigger/HandlerService.go b/pkg/build/trigger/HandlerService.go
--- a/pkg/build/trigger/HandlerService.go
+++ b/pkg/build/trigger/HandlerService.go
@@ -71,2 +71,11 @@
 
+func obfuscateSensitiveData(workflowRequest *types.WorkflowRequest) *types.WorkflowRequest {
+	// Create a copy of the workflowRequest to avoid modifying the original object
+	obfuscatedRequest := *workflowRequest
+	for i := range obfuscatedRequest.CiProjectDetails {
+		obfuscatedRequest.CiProjectDetails[i].GitOptions.Password = "****"
+	}
+	return &obfuscatedRequest
+}
+
 type HandlerService interface {
@@ -764,3 +773,3 @@
 	if err != nil {
-		impl.Logger.Errorw("error, updateWorkflowRequestWithBuildCacheData", "workflowRequest", workflowRequest, "err", err)
+		impl.Logger.Errorw("error, updateWorkflowRequestWithBuildCacheData", "workflowRequest", obfuscateSensitiveData(workflowRequest), "err", err)
 		return nil, nil, nil, nil, err
EOF
@@ -71,2 +71,11 @@

func obfuscateSensitiveData(workflowRequest *types.WorkflowRequest) *types.WorkflowRequest {
// Create a copy of the workflowRequest to avoid modifying the original object
obfuscatedRequest := *workflowRequest
for i := range obfuscatedRequest.CiProjectDetails {
obfuscatedRequest.CiProjectDetails[i].GitOptions.Password = "****"
}
return &obfuscatedRequest
}

type HandlerService interface {
@@ -764,3 +773,3 @@
if err != nil {
impl.Logger.Errorw("error, updateWorkflowRequestWithBuildCacheData", "workflowRequest", workflowRequest, "err", err)
impl.Logger.Errorw("error, updateWorkflowRequestWithBuildCacheData", "workflowRequest", obfuscateSensitiveData(workflowRequest), "err", err)
return nil, nil, nil, nil, err
Copilot is powered by AI and may make mistakes. Always verify output.
Unable to commit as this autofix suggestion is now outdated
pkg/build/trigger/HandlerService.go
}
_, numBytes, err := blobStorageService.Get(request)
if err != nil {
impl.Logger.Errorw("error occurred while downloading file", "request", request, "error", err)

Check failure

Code scanning / CodeQL

Clear-text logging of sensitive information

[Sensitive data returned by an access to BlobStorageS3SecretKey](1) flows to a logging call.

Copilot Autofix

AI about 2 months ago

To fix the problem, we need to ensure that sensitive information such as BlobStorageS3SecretKey is not logged in clear text. The best way to fix this without changing existing functionality is to remove the sensitive information from the logging statement. We can log the request without including sensitive fields or obfuscate the sensitive information before logging.

We will modify the logging statement on line 1945 to exclude the sensitive information from the request object. This can be done by creating a copy of the request object with the sensitive fields removed or obfuscated before logging.

Suggested changeset 1
pkg/build/trigger/HandlerService.go

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/pkg/build/trigger/HandlerService.go b/pkg/build/trigger/HandlerService.go
--- a/pkg/build/trigger/HandlerService.go
+++ b/pkg/build/trigger/HandlerService.go
@@ -1944,3 +1944,5 @@
 	if err != nil {
-		impl.Logger.Errorw("error occurred while downloading file", "request", request, "error", err)
+		safeRequest := *request
+		safeRequest.AwsS3BaseConfig.Passkey = "REDACTED"
+		impl.Logger.Errorw("error occurred while downloading file", "request", safeRequest, "error", err)
 		return nil, errors.New("failed to download resource")
EOF
@@ -1944,3 +1944,5 @@
if err != nil {
impl.Logger.Errorw("error occurred while downloading file", "request", request, "error", err)
safeRequest := *request
safeRequest.AwsS3BaseConfig.Passkey = "REDACTED"
impl.Logger.Errorw("error occurred while downloading file", "request", safeRequest, "error", err)
return nil, errors.New("failed to download resource")
Copilot is powered by AI and may make mistakes. Always verify output.
Unable to commit as this autofix suggestion is now outdated
pkg/deployment/trigger/devtronApps/prePostWfAndLogsCode.go
}
_, numBytes, err := blobStorageService.Get(request)
if err != nil {
impl.logger.Errorw("error occurred while downloading file", "request", request, "error", err)

Check failure

Code scanning / CodeQL

Clear-text logging of sensitive information

[Sensitive data returned by an access to BlobStorageS3SecretKey](1) flows to a logging call.

Copilot Autofix

AI about 2 months ago

To fix the problem, we need to ensure that sensitive information such as BlobStorageS3SecretKey is not logged in clear text. The best way to fix this is to remove the sensitive information from the logging call or to obfuscate it before logging.

In this case, we will remove the sensitive information from the logging call. Specifically, we will modify the logging statement on line 213 to exclude the request object, which contains the sensitive information.

Suggested changeset 1
pkg/deployment/trigger/devtronApps/prePostWfAndLogsCode.go

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/pkg/deployment/trigger/devtronApps/prePostWfAndLogsCode.go b/pkg/deployment/trigger/devtronApps/prePostWfAndLogsCode.go
--- a/pkg/deployment/trigger/devtronApps/prePostWfAndLogsCode.go
+++ b/pkg/deployment/trigger/devtronApps/prePostWfAndLogsCode.go
@@ -212,3 +212,3 @@
 	if err != nil {
-		impl.logger.Errorw("error occurred while downloading file", "request", request, "error", err)
+		impl.logger.Errorw("error occurred while downloading file", "error", err)
 		return nil, errors.New("failed to download resource")
EOF
@@ -212,3 +212,3 @@
if err != nil {
impl.logger.Errorw("error occurred while downloading file", "request", request, "error", err)
impl.logger.Errorw("error occurred while downloading file", "error", err)
return nil, errors.New("failed to download resource")
Copilot is powered by AI and may make mistakes. Always verify output.
@kartik-579 kartik-579 mentioned this pull request Apr 9, 2025
Closed
7 tasks
@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
7.0% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

@github-actions GitHub Actions
Copy link

The migration files have successfully passed the criteria!!

@Ash-exp Ash-exp merged commit fc44e3c into develop Apr 22, 2025
8 of 12 checks passed
@Ash-exp Ash-exp deleted the refactor-ci-and-multi-cluster-cicd branch April 22, 2025 09:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vikramdevtron vikramdevtron vikramdevtron approved these changes

@prakarsh-dt prakarsh-dt Awaiting requested review from prakarsh-dt prakarsh-dt is a code owner

@nishant-d nishant-d Awaiting requested review from nishant-d nishant-d is a code owner

@vivek-devtron vivek-devtron Awaiting requested review from vivek-devtron vivek-devtron is a code owner

Assignees

@Ash-exp Ash-exp

@kartik-579 kartik-579

Labels
PR:Ready-to-Review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kartik-579 @vikramdevtron @Ash-exp