Skip to content

Commit 21b7d83

Browse files
Ash-expgireesh-naiduiamayushmkartik-579ayu-devtron
authored
fix: autoscale error handling (#5481)
* fix: autoscale error handling * minor updates * chore: renaming and logging improved * chore: logging * chore: k8s request adapters * chore: k8s adapter * chore: updated common-lib version * fix: 5xx fixes issues and Panic handling (#5402) * feat: 5xx issue for manual cd trigger API * fix: panic handling for case scantool not found * fix: switch ci fixes (#5358) * fix: switch ci fixes * chore: review refactoring * chore: refactoring * fix: add missing validations in switching to external ci * fix: ci pipeline id is not being updated in cd pipeline if switching from external ci to other ci types * fix: update ci pipeline id in all the cd workflows that exists in the given workflow * fix: checkIfNsExistsForEnvIds , empty envIds check fix * fix: ci patch rbac fixes (#5461) * fix: ci patch rbac fixes * fix: rbac optimisation * fix: refactoring --------- Co-authored-by: Asutosh Das <asutosh2000ad@gmail.com> * feat: deployment config migration (#5368) * inital commit - deployment config service * wip: migration changes for devtron apps * changes for helm apps * dev testing fixes * wire fix * bulk get function * wip: bulk logic fixes * wip: bulk fixes * feat: Cd pipeline migration sql and query usage update (#5416) * Added sql * removed unused mthod * wip * wip * wip * wip * wip * wip * QUERY : updated GetOfLatestCdWfrByCdPipelineId * QUERY : GetOfLatestInstalledAppVersionHistoryByInstalledAppVersionId * QUERY : fix GetOfLatestCdWfrByCdPipelineId * QUERY : fix GetOfLatestInstalledAppVersionHistoryByInstalledAppVersionId * QUERY : updated GetLatestTriggersOfHelmPipelinesStuckInNonTerminalStatuses * QUERY : updated GetArgoPipelineByArgoAppName * QUERY : updated GetArgoPipelinesHavingLatestTriggerStuckInNonTerminalStatuses * QUERY : updated GetArgoPipelinesHavingTriggersStuckInLastPossibleNonTerminalTimelines * QUERY : updated GetAppAndEnvDetailsForDeploymentAppTypePipeline * QUERY : updated FindActiveByEnvIdAndDeploymentType * QUERY : updated GetActiveInstalledAppByEnvIdAndDeploymentType * QUERY : updated GetArgoPipelinesHavingTriggersStuckInLastPossibleNonTerminalTimelinesForAppStore * QUERY : updated GetArgoPipelinesHavingLatestTriggerStuckInNonTerminalStatusesForAppStore * QUERY : updated GetInstalledAppByInstalledAppVersionId * QUERY : updated GetInstalledAppByAppIdAndDeploymentType * QUERY : updated GetAppAndEnvDetailsForDeploymentAppTypeInstalledApps * QUERY : updated FindLatestByAppIdAndEnvId * QUERY : updated GetInstalledAppByGitRepoUrl * QUERY : updated FindChartByGitRepoUrl * dev testing fixes * QUERY : UPDATED buildAppListingWhereCondition and FIXED deploymentDetailsByAppIdAndEnvId * function for bulk update config * query changes * remaining query changes * PR review changes and logical fixes * nil pointer fix * fix cd pipeline helm app create * dev testing fixes * updating app stages status check * query active check for deployment config * wip * adding flag for migration and removing bulk code * build fix * wip: modifying get function * handling custom gitops case for app clone * handling app clone with custom gitops for system generated url * fix: wrong call for helm app deployment app type change * fix migration function for helm app * wip: updating dc on template update * fix: replacing migrate with get call * fix: wrong call for helm app * minor fixes * renaming chart path * updating deprecated columns * updating dc.active * updating dc.active * fix redirection * removing chart path * updating migration * conditions in patch object * fix error in flag off-> custom-gitops-> app-clone-> deploy-> flag-on-> migrate * wire fix * audit log fix * migation update * reverted 107 sql newline add --------- Co-authored-by: kartik-579 <kartik@devtron.ai> Co-authored-by: kartik-579 <84493919+kartik-579@users.noreply.github.com> * handle nil check (#5497) * doc: Added FAQ no. 28 + GoLang-migrate Link + Code Block Fix (#5502) * Added FAQ no. 28 + GoLang-migrate Link + Code Block Fix * Removed extra spacing at EOF * fix: bitbucket commit race condition for concurrent requests (#5505) * fix: hotfix v0.14.0 fixes (#5500) * fix: patch ci rbac fix (#5498) * handle nil check (#5497) * fix: patch ci rbac fix --------- Co-authored-by: ayu-devtron <167413063+ayu-devtron@users.noreply.github.com> * fix: external ci creation fix (#5499) * handle nil check (#5497) * fix: external ci creation fix --------- Co-authored-by: ayu-devtron <167413063+ayu-devtron@users.noreply.github.com> * fix for deployment config - auto post cd not working (#5501) * fix: do not create deployment config for switch ci, if pipeline is already created. (#5506) * chore: main sync (#5510) * handle nil check (#5497) * doc: Added FAQ no. 28 + GoLang-migrate Link + Code Block Fix (#5502) * Added FAQ no. 28 + GoLang-migrate Link + Code Block Fix * Removed extra spacing at EOF * fix: bitbucket commit race condition for concurrent requests (#5505) --------- Co-authored-by: ayu-devtron <167413063+ayu-devtron@users.noreply.github.com> Co-authored-by: ashokdevtron <141001279+ashokdevtron@users.noreply.github.com> Co-authored-by: Asutosh Das <asutosh2000ad@gmail.com> --------- Co-authored-by: ayu-devtron <167413063+ayu-devtron@users.noreply.github.com> Co-authored-by: kartik-579 <84493919+kartik-579@users.noreply.github.com> Co-authored-by: ashokdevtron <141001279+ashokdevtron@users.noreply.github.com> Co-authored-by: Asutosh Das <asutosh2000ad@gmail.com> --------- Co-authored-by: Gireesh Naidu <111440205+gireesh-devtron@users.noreply.github.com> Co-authored-by: iamayushm <32041961+iamayushm@users.noreply.github.com> Co-authored-by: kartik-579 <kartik@devtron.ai> Co-authored-by: kartik-579 <84493919+kartik-579@users.noreply.github.com> Co-authored-by: ayu-devtron <167413063+ayu-devtron@users.noreply.github.com> Co-authored-by: ashokdevtron <141001279+ashokdevtron@users.noreply.github.com> * updated: common-lib version --------- Co-authored-by: Gireesh Naidu <111440205+gireesh-devtron@users.noreply.github.com> Co-authored-by: iamayushm <32041961+iamayushm@users.noreply.github.com> Co-authored-by: kartik-579 <kartik@devtron.ai> Co-authored-by: kartik-579 <84493919+kartik-579@users.noreply.github.com> Co-authored-by: ayu-devtron <167413063+ayu-devtron@users.noreply.github.com> Co-authored-by: ashokdevtron <141001279+ashokdevtron@users.noreply.github.com>
1 parent 9db1489 commit 21b7d83

File tree

13 files changed

+258
-89
lines changed

13 files changed

+258
-89
lines changed

‎client/argocdServer/k8sClient.go

Lines changed: 41 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -20,11 +20,14 @@ import (
2020
"bytes"
2121
"context"
2222
"encoding/json"
23+
"errors"
2324
"fmt"
2425
"github.com/devtron-labs/common-lib/utils/k8s"
26+
"github.com/devtron-labs/devtron/internal/util"
2527
"github.com/devtron-labs/devtron/pkg/cluster/repository"
2628
"go.uber.org/zap"
2729
"io/ioutil"
30+
k8sError "k8s.io/apimachinery/pkg/api/errors"
2831
"k8s.io/apimachinery/pkg/runtime"
2932
"k8s.io/apimachinery/pkg/runtime/schema"
3033
"k8s.io/apimachinery/pkg/runtime/serializer"
@@ -117,7 +120,7 @@ func (impl ArgoK8sClientImpl) CreateArgoApplication(ctx context.Context, namespa
117120
if err != nil {
118121
return fmt.Errorf("error creating argo cd app")
119122
}
120-
impl.logger.Infow("creating application", "req", application)
123+
impl.logger.Debugw("creating argo application resource", "application", application)
121124
res, err := client.
122125
Post().
123126
Resource("applications").
@@ -126,23 +129,48 @@ func (impl ArgoK8sClientImpl) CreateArgoApplication(ctx context.Context, namespa
126129
Do(ctx).Raw()
127130

128131
if err != nil {
129-
response := make(map[string]interface{})
130-
err := json.Unmarshal(res, &response)
131-
if err != nil {
132-
impl.logger.Errorw("unmarshal error on app update status", "err", err)
133-
return fmt.Errorf("error creating argo cd app")
134-
}
135-
message := "error creating argo cd app"
136-
if response != nil && response["message"] != nil {
137-
message = response["message"].(string)
138-
}
139-
return fmt.Errorf(message)
132+
impl.logger.Errorw("error in argo application resource creation", "namespace", namespace, "res", res, "err", err)
133+
return impl.handleArgoAppCreationError(res, err)
140134
}
141135

142-
impl.logger.Infow("argo app create res", "res", string(res), "err", err)
136+
impl.logger.Infow("argo app create successfully", "namespace", namespace, "res", string(res))
143137
return err
144138
}
145139

140+
func (impl ArgoK8sClientImpl) handleArgoAppCreationError(res []byte, err error) error {
141+
// default error set
142+
apiError := &util.ApiError{
143+
InternalMessage: "error creating argo cd app",
144+
UserMessage: "error creating argo cd app",
145+
}
146+
// error override for errors.StatusError
147+
if statusError := (&k8sError.StatusError{}); errors.As(err, &statusError) {
148+
apiError.HttpStatusCode = int(statusError.Status().Code)
149+
apiError.InternalMessage = statusError.Error()
150+
apiError.UserMessage = statusError.Error()
151+
}
152+
response := make(map[string]interface{})
153+
jsonErr := json.Unmarshal(res, &response)
154+
if jsonErr != nil {
155+
impl.logger.Errorw("unmarshal error on app update status", "err", jsonErr)
156+
return apiError
157+
}
158+
// error override if API response exists, as response errors are more readable
159+
if response != nil {
160+
if statusCode, ok := response["code"]; apiError.HttpStatusCode == 0 && ok {
161+
if statusCodeFloat, ok := statusCode.(float64); ok {
162+
apiError.HttpStatusCode = int(statusCodeFloat)
163+
}
164+
}
165+
if response["message"] != nil {
166+
errMsg := response["message"].(string)
167+
apiError.InternalMessage = errMsg
168+
apiError.UserMessage = errMsg
169+
}
170+
}
171+
return apiError
172+
}
173+
146174
func (impl ArgoK8sClientImpl) GetArgoApplication(namespace string, appName string, cluster *repository.Cluster) (map[string]interface{}, error) {
147175

148176
config, err := rest.InClusterConfig()

‎go.mod

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ require (
2222
github.com/davecgh/go-spew v1.1.1
2323
github.com/deckarep/golang-set v1.8.0
2424
github.com/devtron-labs/authenticator v0.4.35-0.20240607135426-c86e868ecee1
25-
github.com/devtron-labs/common-lib v0.0.23
25+
github.com/devtron-labs/common-lib v0.0.24
2626
github.com/devtron-labs/go-bitbucket v0.9.60-beta
2727
github.com/devtron-labs/protos v0.0.3-0.20240527113333-08a3be5ec6c1
2828
github.com/evanphx/json-patch v5.7.0+incompatible

‎go.sum

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -197,8 +197,8 @@ github.com/denisenkom/go-mssqldb v0.0.0-20200428022330-06a60b6afbbc h1:VRRKCwnzq
197197
github.com/denisenkom/go-mssqldb v0.0.0-20200428022330-06a60b6afbbc/go.mod h1:xbL0rPBG9cCiLr28tMa8zpbdarY27NDyej4t/EjAShU=
198198
github.com/devtron-labs/authenticator v0.4.35-0.20240607135426-c86e868ecee1 h1:qdkpTAo2Kr0ZicZIVXfNwsGSshpc9OB9j9RzmKYdIwY=
199199
github.com/devtron-labs/authenticator v0.4.35-0.20240607135426-c86e868ecee1/go.mod h1:IkKPPEfgLCMR29he5yv2OCC6iM2R7K5/0AA3k8b9XNc=
200-
github.com/devtron-labs/common-lib v0.0.23 h1:3n9VgJALanjHrb/t0nSaLbLJcJa7sdsdyFf2leE+cN0=
201-
github.com/devtron-labs/common-lib v0.0.23/go.mod h1:UZGPt1ep9Tnd9Ak2sibGSiLr7p3ijO2/JLT+h+pqBuU=
200+
github.com/devtron-labs/common-lib v0.0.24 h1:9Tb1diOnyFFrKJ+RSdzvnBUIQsUhPVaOcyQemU5q2oo=
201+
github.com/devtron-labs/common-lib v0.0.24/go.mod h1:UZGPt1ep9Tnd9Ak2sibGSiLr7p3ijO2/JLT+h+pqBuU=
202202
github.com/devtron-labs/go-bitbucket v0.9.60-beta h1:VEx1jvDgdtDPS6A1uUFoaEi0l1/oLhbr+90xOwr6sDU=
203203
github.com/devtron-labs/go-bitbucket v0.9.60-beta/go.mod h1:GnuiCesvh8xyHeMCb+twm8lBR/kQzJYSKL28ZfObp1Y=
204204
github.com/devtron-labs/protos v0.0.3-0.20240527113333-08a3be5ec6c1 h1:R6qVeFaayqstBSu4w+ipWQqJyMKDqBVV3a11qoA2IaM=

‎internal/errors/bean.go

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,10 @@ func (r *ClientStatusCode) IsDeadlineExceededCode() bool {
4242
return r.Code == codes.DeadlineExceeded
4343
}
4444

45+
func (r *ClientStatusCode) IsUnavailableCode() bool {
46+
return r.Code == codes.Unavailable
47+
}
48+
4549
func (r *ClientStatusCode) IsCanceledCode() bool {
4650
return r.Code == codes.Canceled
4751
}

‎internal/middleware/instrument.go

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -100,10 +100,6 @@ var currentRequestGauge = promauto.NewGaugeVec(prometheus.GaugeOpts{
100100
Help: "no of request being served currently",
101101
}, []string{"path", "method"})
102102

103-
var AcdGetResourceCounter = promauto.NewCounterVec(prometheus.CounterOpts{
104-
Name: "acd_get_resource_counter",
105-
}, []string{"appId", "envId", "acdAppName"})
106-
107103
var CdTriggerCounter = promauto.NewCounterVec(prometheus.CounterOpts{
108104
Name: "cd_trigger_counter",
109105
}, []string{"appName", "envName"})

‎pkg/clusterTerminalAccess/UserTerminalAccessService.go

Lines changed: 2 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -366,13 +366,6 @@ func (impl *UserTerminalAccessServiceImpl) DisconnectTerminalSession(ctx context
366366
return err
367367
}
368368

369-
func getErrorDetailedMessage(err error) string {
370-
if errStatus, ok := err.(*k8sErrors.StatusError); ok {
371-
return errStatus.Status().Message
372-
}
373-
return ""
374-
}
375-
376369
func (impl *UserTerminalAccessServiceImpl) StopTerminalSession(ctx context.Context, userTerminalAccessId int) {
377370
impl.Logger.Infow("terminal stop request received for user", "userTerminalAccessId", userTerminalAccessId)
378371
impl.TerminalAccessDataArrayMutex.Lock()
@@ -529,7 +522,7 @@ func (impl *UserTerminalAccessServiceImpl) SyncPodStatus() {
529522
err = impl.DeleteTerminalPod(context.Background(), terminalAccessData.ClusterId, terminalAccessData.PodName, namespace)
530523
if err != nil {
531524
if k8s.IsResourceNotFoundErr(err) {
532-
errorDetailedMessage := getErrorDetailedMessage(err)
525+
errorDetailedMessage := k8s.GetClientErrorMessage(err)
533526
terminalPodStatusString = fmt.Sprintf("%s/%s", string(models.TerminalPodTerminated), errorDetailedMessage)
534527
} else {
535528
continue
@@ -841,7 +834,7 @@ func (impl *UserTerminalAccessServiceImpl) getPodManifest(ctx context.Context, c
841834
response, err := impl.K8sCommonService.GetResource(ctx, request)
842835
if err != nil {
843836
if k8s.IsResourceNotFoundErr(err) {
844-
errorDetailedMessage := getErrorDetailedMessage(err)
837+
errorDetailedMessage := k8s.GetClientErrorMessage(err)
845838
terminalPodStatusString := fmt.Sprintf("%s/%s", string(models.TerminalPodTerminated), errorDetailedMessage)
846839
return nil, errors.New(terminalPodStatusString)
847840
} else {

0 commit comments

Comments
 (0)