From 5c8ef150e809c32102dea3fb8706a268d2d5fe79 Mon Sep 17 00:00:00 2001 From: ashokdevtron Date: Tue, 28 May 2024 21:17:23 +0530 Subject: [PATCH 1/5] Created Resource Watcher Doc --- docs/SUMMARY.md | 1 + docs/user-guide/resource-watcher.md | 116 ++++++++++++++++++++++++++++ 2 files changed, 117 insertions(+) create mode 100644 docs/user-guide/resource-watcher.md diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 7082ef1c30..7f1a928e24 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -104,6 +104,7 @@ * [Overview](user-guide/jobs/overview-job.md) * [Application Groups](user-guide/application-groups.md) * [Resource Browser](user-guide/resource-browser.md) +* [Resource Watcher](user-guide/resource-watcher.md) * [Charts](user-guide/deploy-chart/README.md) * [Charts Overview](user-guide/deploy-chart/overview-of-charts.md) * [Deploy & Observe](user-guide/deploy-chart/deployment-of-charts.md) diff --git a/docs/user-guide/resource-watcher.md b/docs/user-guide/resource-watcher.md new file mode 100644 index 0000000000..8478df2953 --- /dev/null +++ b/docs/user-guide/resource-watcher.md @@ -0,0 +1,116 @@ +# Resource Watcher + +## Introduction [![](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/elements/EnterpriseTag.svg)](https://devtron.ai/pricing) + +An incident response if delayed can impact businesses, revenue, and waste valuable engineering time. Devtron's Resource Watcher enables you to perform automated actions upon the occurrence of events: + +* **Create Event** - Occurs when a new Kubernetes resource is created, for e.g., a new pod spun up to handle increased traffic. +* **Update Event** - Occurs when an existing Kubernetes resource is modified, for e.g., deployment configuration tweaked to increase the replica count. +* **Delete Event** - Occurs when an existing Kubernetes resource is deleted, for e.g., deletion of an orphaned pod. + +You can make the Resource Watcher listen to the above events and accordingly run a job you wish to get done, for e.g., increasing memory, executing a script, raising Jira ticket, emailing your stakeholders, sending Slack notifications, and many more. Since manual intervention is absent, the timely response of this auto-remediation system improves your operational efficiency. + +--- + +## Creating a Watcher + +{% hint style="warning" %} +### Who Can Perform This Action? +Users need to have super-admin permission to create a watcher. +{% endhint %} + +This page allows you to create a watcher to track events and run a job. It also shows the existing list of watchers (if any). + +1. Click **+ Create Watcher**. + + ![Figure 1: Watchers - Page](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/resource-watcher/watchers-page.jpg) + +2. Creating a watcher consists of 4 parts, fill all the sections one by one: + * [Basic Details](#basic-details) + * [Namespaces to Watch](#namespaces-to-watch) + * [Intercept Change in Resources](#intercept-change-in-resources) + * [Execute Runbook](#execute-runbook) + + ![Figure 2: Create Watcher - Window](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/resource-watcher/create-watcher-window.jpg) + +### Basic Details + +Here, you can give a name and description to your watcher. + +![Figure 3: Adding Name and Description of Watcher](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/resource-watcher/basic-details.gif) + +### Namespaces to Watch + +Here, you can select the [namespaces](../reference/glossary.md#namespace) whose [Kubernetes resource](../reference/glossary.md#objects) you wish to monitor for changes. + +* You can watch the namespace(s) across **All Clusters** (existing and future). + + ![Figure 4: Choosing Namespaces of all Clusters](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/resource-watcher/all-cluster.gif) + +* Or you can watch namespace(s) of **Specific Clusters**. + + ![Figure 5: Choosing Namespaces of Specific Clusters](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/resource-watcher/specific-cluster.gif) + +{% hint style="warning" %} +In both the above options, if you choose 'Specific Namespaces', you can further decide whether to track the namespaces you enter (by clicking 'Include selections') or to track the namespaces except the ones you enter (by clicking 'Exclude selections'). +{% endhint %} + + +### Intercept Change in Resources + +Here, you can select the exact Kubernetes resource(s) you wish to track for changes (in the namespace(s) you selected in the previous step). + +![Figure 6: Picking Resources to Track](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/resource-watcher/intercept-changes.gif) + +* You can choose the resource from the **Resource kind(s) to watch** dropdown. Enter the link if it's a custom resource, for e.g., `install.istio.io/v1apha1/IstioOperator` + +* Choose the event type your watcher should listen to: `Created`, `Updated`, `Deleted`. + +* (Optional) You may also enter a [CEL expression](https://github.com/google/cel-spec/blob/master/doc/langdef.md) to catch a specific change in the resource's manifest. + +### Execute Runbook + +Here, you can choose a job that should trigger if your watcher intercepts any changes. If you wish to simply intercept the changes and not perform any action, you may skip this section. + +![Figure 7: Choosing a Job to Trigger](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/resource-watcher/execute-runbook.gif) + +* Choose a job pipeline from the **Run Devtron Job pipeline** dropdown. + +* Select the environment in which the job should run. It can either be `devtron-ci` or the source environment (the intercepted namespace where the event has occurred). + +* If the job expects input parameters, you may add its key and value under **Runtime input parameters**. You can also access the supported environment variables from the tooltip on the right. + +Click **Create Watcher**. Your watcher is now ready to intercept the changes to the selected resources. + +--- + +## Viewing Intercepted Changes + +{% hint style="warning" %} +### Who Can Perform This Action? +Users need to have super-admin permission to view intercepted changes. +{% endhint %} + +This page allows you to view the changes to Kubernetes resources that you have selected for tracking changes. + +![Figure 8: Intercepted Changes - Page](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/resource-watcher/intercepted-changes-page.jpg) + +It comes with the following items to help you locate the resource, where the event has been intercepted: + +* Searchbox +* Cluster filter +* Namespace filter +* Action filter (event type, i.e., `Created`, `Updated`, `Deleted`) +* Watcher filter (to check the intercepted changes of a specific watcher) + +You get the following details in the results shown on the page. + +|Field | Description | +|-------|-------------| +|Change In Resource|Describes the type of change to the Kubernetes resource along with a link to its manifest| +|Cluster/Namespace |Shows the cluster and namespace where the tracked Kubernetes resource belongs to| +|Intercepted By |Shows the name of the watcher that intercepted the change| +|Intercepted At |Shows the date and time when the event occurred | +|Job Execution |Shows the status of the execution of job, e.g., `In Progress`, `Succeeded`, `Failed`| +|Logs |Links to the job log, i.e, the `Run history` page of the job| + From d15b4bef34da5eb11f27e25cee805562aaf801c5 Mon Sep 17 00:00:00 2001 From: ashokdevtron Date: Thu, 6 Jun 2024 17:17:26 +0530 Subject: [PATCH 2/5] Incorporated PM Feedback --- docs/user-guide/resource-watcher.md | 73 +++++++++++++++++++++++++---- 1 file changed, 63 insertions(+), 10 deletions(-) diff --git a/docs/user-guide/resource-watcher.md b/docs/user-guide/resource-watcher.md index 8478df2953..9da4f10c85 100644 --- a/docs/user-guide/resource-watcher.md +++ b/docs/user-guide/resource-watcher.md @@ -51,7 +51,7 @@ Here, you can select the [namespaces](../reference/glossary.md#namespace) whose ![Figure 5: Choosing Namespaces of Specific Clusters](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/resource-watcher/specific-cluster.gif) -{% hint style="warning" %} +{% hint style="info" %} In both the above options, if you choose 'Specific Namespaces', you can further decide whether to track the namespaces you enter (by clicking 'Include selections') or to track the namespaces except the ones you enter (by clicking 'Exclude selections'). {% endhint %} @@ -62,15 +62,23 @@ Here, you can select the exact Kubernetes resource(s) you wish to track for chan ![Figure 6: Picking Resources to Track](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/resource-watcher/intercept-changes.gif) -* You can choose the resource from the **Resource kind(s) to watch** dropdown. Enter the link if it's a custom resource, for e.g., `install.istio.io/v1apha1/IstioOperator` +* You can choose the resource from the **Resource kind(s) to watch** dropdown. Enter the Group Version Kind (GVK) if it's a custom resource definition (CRD), for e.g., `install.istio.io/v1apha1/IstioOperator` * Choose the event type your watcher should listen to: `Created`, `Updated`, `Deleted`. -* (Optional) You may also enter a [CEL expression](https://github.com/google/cel-spec/blob/master/doc/langdef.md) to catch a specific change in the resource's manifest. +* Enter a [CEL expression](https://github.com/google/cel-spec/blob/master/doc/langdef.md) to catch a specific change in the resource's manifest. + +**Example**: `final.status.currentReplicas == final.spec.MaxReplicas` + +{% hint style="info" %} +* **If Resource Is Created** - Use 'final' +* **If Resource Is Updated** - Both 'initial' and 'final' manifest exist +* **If Resource Is Deleted** - Use 'initial' +{% endhint %} ### Execute Runbook -Here, you can choose a job that should trigger if your watcher intercepts any changes. If you wish to simply intercept the changes and not perform any action, you may skip this section. +Here, you can choose a job that should trigger if your watcher intercepts any changes. ![Figure 7: Choosing a Job to Trigger](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/resource-watcher/execute-runbook.gif) @@ -78,9 +86,15 @@ Here, you can choose a job that should trigger if your watcher intercepts any ch * Select the environment in which the job should run. It can either be `devtron-ci` or the source environment (the intercepted namespace where the event has occurred). -* If the job expects input parameters, you may add its key and value under **Runtime input parameters**. You can also access the supported environment variables from the tooltip on the right. +* If the job expects input parameters, you may add its key and value under **Runtime input parameters**. -Click **Create Watcher**. Your watcher is now ready to intercept the changes to the selected resources. + During a job's execution, its container can access the initial and final resource manifest through special environment variables. These variables are: + * `DEVTRON_INITIAL_MANIFEST` + * `DEVTRON_FINAL_MANIFEST` + +* Click **Create Watcher**. + +Your watcher is now ready to intercept the changes to the selected resources. --- @@ -91,6 +105,8 @@ Click **Create Watcher**. Your watcher is now ready to intercept the changes to Users need to have super-admin permission to view intercepted changes. {% endhint %} +### Details + This page allows you to view the changes to Kubernetes resources that you have selected for tracking changes. ![Figure 8: Intercepted Changes - Page](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/resource-watcher/intercepted-changes-page.jpg) @@ -107,10 +123,47 @@ You get the following details in the results shown on the page. |Field | Description | |-------|-------------| -|Change In Resource|Describes the type of change to the Kubernetes resource along with a link to its manifest| -|Cluster/Namespace |Shows the cluster and namespace where the tracked Kubernetes resource belongs to| +|[Change In Resource](#change-in-resource)|Describes the type of change to the Kubernetes resource along with a link to its manifest| +|[Cluster/Namespace](#namespaces-to-watch) |Shows the cluster and namespace where the tracked Kubernetes resource belongs to| |Intercepted By |Shows the name of the watcher that intercepted the change| |Intercepted At |Shows the date and time when the event occurred | -|Job Execution |Shows the status of the execution of job, e.g., `In Progress`, `Succeeded`, `Failed`| -|Logs |Links to the job log, i.e, the `Run history` page of the job| +|[Job Execution](#execute-runbook) |Shows the status of the execution of job, e.g., `In Progress`, `Succeeded`, `Failed`| +|[Logs](#job-execution-log) |Links to the job log, i.e, the `Run history` page of the job| + +### Change in Resource + +You can check the changes in manifest by clicking **View Manifest** in `Change In Resource` column. + +![Figure 9a: Created Resource Manifest - Final Manifest](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/resource-watcher/view-manifest-v1.gif) + + +![Figure 9b: Updated Resource - Initial and Final Manifest](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/resource-watcher/view-manifest-v2.gif) + + +![Figure 9c: Deleted Resource - Initial Manifest](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/resource-watcher/view-manifest.gif) + +### Job Execution Log + +You can check the logs of the job executed when the Resource Watcher intercepts any change by clicking **logs**. + +![Figure 10: Job Progress](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/resource-watcher/job-exec-log.gif) + +--- + +## Use Cases + +### Live Stream Traffic Surge + +A live streaming sports application experiences a surge in viewers during a major game. The Horizontal Pod Autoscaler (HPA) might not be able to handle the unexpected traffic if it's capped at a low max replica count. + +1. Create a watcher named 'Live Stream Scaling Alert'. +2. Monitor updates to HPA resource in the application's namespace. +3. When `currentReplicas` count reaches `MaxReplicas`, trigger a job that contains the script to increase the replica count. + +### Pod Health Monitoring + +A stock trading application constantly updates stock prices for its traders. If the pods become unhealthy, traders might see incorrect stock prices leading to bad investments. + +1. Create a watcher named 'Pod Health Monitor'. +2. Track the pod workload of your application, if `final.status.phase != 'Running'`, trigger a job that sends an Email/Slack alert with pod details. From da553ed4b5d0c5bebefc940337dc4aff5dd7cbb7 Mon Sep 17 00:00:00 2001 From: ashokdevtron Date: Thu, 6 Jun 2024 17:50:54 +0530 Subject: [PATCH 3/5] Variable Change + Fixes --- docs/user-guide/resource-watcher.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/user-guide/resource-watcher.md b/docs/user-guide/resource-watcher.md index 9da4f10c85..93febb8a89 100644 --- a/docs/user-guide/resource-watcher.md +++ b/docs/user-guide/resource-watcher.md @@ -62,18 +62,18 @@ Here, you can select the exact Kubernetes resource(s) you wish to track for chan ![Figure 6: Picking Resources to Track](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/resource-watcher/intercept-changes.gif) -* You can choose the resource from the **Resource kind(s) to watch** dropdown. Enter the Group Version Kind (GVK) if it's a custom resource definition (CRD), for e.g., `install.istio.io/v1apha1/IstioOperator` +* You can choose the resource from the **Resource kind(s) to watch** dropdown. Enter the Group/Version/Kind (GVK) if it's a custom resource definition (CRD), for e.g., `install.istio.io/v1apha1/IstioOperator` * Choose the event type your watcher should listen to: `Created`, `Updated`, `Deleted`. * Enter a [CEL expression](https://github.com/google/cel-spec/blob/master/doc/langdef.md) to catch a specific change in the resource's manifest. -**Example**: `final.status.currentReplicas == final.spec.MaxReplicas` +**Example**: `DEVTRON_FINAL_MANIFEST.status.currentReplicas == DEVTRON_FINAL_MANIFEST.spec.MaxReplicas` {% hint style="info" %} -* **If Resource Is Created** - Use 'final' -* **If Resource Is Updated** - Both 'initial' and 'final' manifest exist -* **If Resource Is Deleted** - Use 'initial' +* **If Resource Is Created** - Use 'DEVTRON_FINAL_MANIFEST' +* **If Resource Is Updated** - Both 'DEVTRON_INITIAL_MANIFEST' and 'DEVTRON_FINAL_MANIFEST' manifest exist +* **If Resource Is Deleted** - Use 'DEVTRON_FINAL_MANIFEST' {% endhint %} ### Execute Runbook @@ -165,5 +165,5 @@ A live streaming sports application experiences a surge in viewers during a majo A stock trading application constantly updates stock prices for its traders. If the pods become unhealthy, traders might see incorrect stock prices leading to bad investments. 1. Create a watcher named 'Pod Health Monitor'. -2. Track the pod workload of your application, if `final.status.phase != 'Running'`, trigger a job that sends an Email/Slack alert with pod details. +2. Track the pod workload of your application, if `DEVTRON_FINAL_MANIFEST.status.phase != 'Running'`, trigger a job that sends an Email/Slack alert with pod details. From 207ff56b15bc2ff340f16fd6c23329e7d9034221 Mon Sep 17 00:00:00 2001 From: ashokdevtron Date: Thu, 6 Jun 2024 17:53:51 +0530 Subject: [PATCH 4/5] Variable Change + Fixes --- docs/user-guide/resource-watcher.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user-guide/resource-watcher.md b/docs/user-guide/resource-watcher.md index 93febb8a89..0106db162b 100644 --- a/docs/user-guide/resource-watcher.md +++ b/docs/user-guide/resource-watcher.md @@ -72,7 +72,7 @@ Here, you can select the exact Kubernetes resource(s) you wish to track for chan {% hint style="info" %} * **If Resource Is Created** - Use 'DEVTRON_FINAL_MANIFEST' -* **If Resource Is Updated** - Both 'DEVTRON_INITIAL_MANIFEST' and 'DEVTRON_FINAL_MANIFEST' manifest exist +* **If Resource Is Updated** - Both 'DEVTRON_INITIAL_MANIFEST' and 'DEVTRON_FINAL_MANIFEST' can exist * **If Resource Is Deleted** - Use 'DEVTRON_FINAL_MANIFEST' {% endhint %} From 43914870cc53721b877ca1463914449524257bfe Mon Sep 17 00:00:00 2001 From: ashokdevtron <141001279+ashokdevtron@users.noreply.github.com> Date: Fri, 7 Jun 2024 06:11:23 +0000 Subject: [PATCH 5/5] Incorporated CO + PM Feedback + Other Fixes --- docs/user-guide/resource-watcher.md | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/docs/user-guide/resource-watcher.md b/docs/user-guide/resource-watcher.md index 0106db162b..93bf83ff17 100644 --- a/docs/user-guide/resource-watcher.md +++ b/docs/user-guide/resource-watcher.md @@ -66,23 +66,29 @@ Here, you can select the exact Kubernetes resource(s) you wish to track for chan * Choose the event type your watcher should listen to: `Created`, `Updated`, `Deleted`. -* Enter a [CEL expression](https://github.com/google/cel-spec/blob/master/doc/langdef.md) to catch a specific change in the resource's manifest. + | Event Type | Description | + | ---------- | ----------------------------------------------------------------------- | + | Created | Triggers the watcher when your Kubernetes resource is created | + | Updated | Triggers the watcher when your existing Kubernetes resource is modified | + | Deleted | Triggers the watcher when your existing Kubernetes resource is deleted | -**Example**: `DEVTRON_FINAL_MANIFEST.status.currentReplicas == DEVTRON_FINAL_MANIFEST.spec.MaxReplicas` +* Enter a [CEL expression](https://github.com/google/cel-spec/blob/master/doc/langdef.md) to catch a specific change in the resource's manifest. {% hint style="info" %} -* **If Resource Is Created** - Use 'DEVTRON_FINAL_MANIFEST' -* **If Resource Is Updated** - Both 'DEVTRON_INITIAL_MANIFEST' and 'DEVTRON_FINAL_MANIFEST' can exist -* **If Resource Is Deleted** - Use 'DEVTRON_FINAL_MANIFEST' +* **If resource is created** - Use 'DEVTRON_FINAL_MANIFEST' +* **If resource is updated** - Both 'DEVTRON_INITIAL_MANIFEST' and 'DEVTRON_FINAL_MANIFEST' can exist +* **If resource is deleted** - Use 'DEVTRON_INITIAL_MANIFEST' {% endhint %} +**Example**: `DEVTRON_FINAL_MANIFEST.status.currentReplicas == DEVTRON_FINAL_MANIFEST.spec.maxReplicas` + ### Execute Runbook Here, you can choose a job that should trigger if your watcher intercepts any changes. ![Figure 7: Choosing a Job to Trigger](https://devtron-public-asset.s3.us-east-2.amazonaws.com/images/resource-watcher/execute-runbook.gif) -* Choose a job pipeline from the **Run Devtron Job pipeline** dropdown. +* Choose a job pipeline from the **Run Devtron Job pipeline** dropdown. If a pipeline is not selected, the watcher won't intercept matching resource changes even if your defined conditions are met. * Select the environment in which the job should run. It can either be `devtron-ci` or the source environment (the intercepted namespace where the event has occurred). @@ -158,7 +164,7 @@ A live streaming sports application experiences a surge in viewers during a majo 1. Create a watcher named 'Live Stream Scaling Alert'. 2. Monitor updates to HPA resource in the application's namespace. -3. When `currentReplicas` count reaches `MaxReplicas`, trigger a job that contains the script to increase the replica count. +3. When `currentReplicas` count reaches `maxReplicas`, trigger a job that contains the script to increase the replica count. ### Pod Health Monitoring