diff --git a/Makefile b/Makefile index 075bba94e4..1d442facff 100644 --- a/Makefile +++ b/Makefile @@ -311,7 +311,7 @@ docs-serve: docs .PHONY: docs-linkcheck docs-linkcheck: /usr/local/bin/lychee - lychee --exclude-path=CHANGELOG.md *.md --include "https://github.com/numaproj/*" $(shell find ./test -type f) $(wildcard ./docs/*.md) + lychee --exclude-path=CHANGELOG.md --exclude-path=./docs/APIs.md --exclude "https://localhost:*" --exclude "http://localhost:*" --exclude "http://127.0.0.1*" *.md $(shell find ./test -type f) $(shell find ./docs -name '*.md') # pre-push checks diff --git a/USERS.md b/USERS.md index 87cd2cfcec..dd4864d851 100644 --- a/USERS.md +++ b/USERS.md @@ -3,7 +3,7 @@ Please add your company name and initial use case (optional) below. 1. [Intuit](https://www.intuit.com/) - Streaming ML inference and to prepare data for ML model training in real-time. -2. [B|Cubed](https://bcubed-corp.com/) - Digital Signal Processing Communication Systems. We receive RF energy and use numaflows to transform the RF to individual bits into intelligible data. +2. [B|Cubed](https://bcubed.com/) - Digital Signal Processing Communication Systems. We receive RF energy and use numaflows to transform the RF to individual bits into intelligible data. 3. [Atlan](https://atlan.com/) - Numaflow powers real time notifications, stream processing ecosystem at Atlan. 4. [Valegachain Analytics](https://www.valegachain.com/) Numaflow is used to extract, transform, and load cryptocurrency blocks and mempool transactions in data lakes, as well as for activity alerts. 5. [Lockheed Martin](https://lockheedmartin.com/) Perform ELT processing on high and low volume data streams of sensor data as recieved from IOT type systems. diff --git a/docs/assets/errors/errors-architecture.png b/docs/assets/errors/errors-architecture.png new file mode 100644 index 0000000000..6c57999435 Binary files /dev/null and b/docs/assets/errors/errors-architecture.png differ diff --git a/docs/assets/errors/ui-errors-tab.png b/docs/assets/errors/ui-errors-tab.png new file mode 100644 index 0000000000..c6dbc75471 Binary files /dev/null and b/docs/assets/errors/ui-errors-tab.png differ diff --git a/docs/assets/logs/logs-add-timestamps.png b/docs/assets/logs/logs-add-timestamps.png new file mode 100644 index 0000000000..54432077ab Binary files /dev/null and b/docs/assets/logs/logs-add-timestamps.png differ diff --git a/docs/assets/logs/logs-ascending.png b/docs/assets/logs/logs-ascending.png new file mode 100644 index 0000000000..dadd820ecc Binary files /dev/null and b/docs/assets/logs/logs-ascending.png differ diff --git a/docs/assets/logs/logs-dark-mode.png b/docs/assets/logs/logs-dark-mode.png new file mode 100644 index 0000000000..f482f80e7c Binary files /dev/null and b/docs/assets/logs/logs-dark-mode.png differ diff --git a/docs/assets/logs/logs-filter-error.png b/docs/assets/logs/logs-filter-error.png new file mode 100644 index 0000000000..a3ef1fc8e6 Binary files /dev/null and b/docs/assets/logs/logs-filter-error.png differ diff --git a/docs/assets/logs/logs-negate-search.png b/docs/assets/logs/logs-negate-search.png new file mode 100644 index 0000000000..603788b92a Binary files /dev/null and b/docs/assets/logs/logs-negate-search.png differ diff --git a/docs/assets/logs/logs-pause.png b/docs/assets/logs/logs-pause.png new file mode 100644 index 0000000000..744c38b529 Binary files /dev/null and b/docs/assets/logs/logs-pause.png differ diff --git a/docs/assets/logs/logs-search.png b/docs/assets/logs/logs-search.png new file mode 100644 index 0000000000..dc43c450d5 Binary files /dev/null and b/docs/assets/logs/logs-search.png differ diff --git a/docs/assets/logs/logs-tab.png b/docs/assets/logs/logs-tab.png new file mode 100644 index 0000000000..0aa71fab7f Binary files /dev/null and b/docs/assets/logs/logs-tab.png differ diff --git a/docs/assets/logs/logs-wrap.png b/docs/assets/logs/logs-wrap.png new file mode 100644 index 0000000000..b5dfde6ae1 Binary files /dev/null and b/docs/assets/logs/logs-wrap.png differ diff --git a/docs/assets/logs/select-container.png b/docs/assets/logs/select-container.png new file mode 100644 index 0000000000..759d6b68c2 Binary files /dev/null and b/docs/assets/logs/select-container.png differ diff --git a/docs/assets/logs/select-pod.png b/docs/assets/logs/select-pod.png new file mode 100644 index 0000000000..228b7fc0ca Binary files /dev/null and b/docs/assets/logs/select-pod.png differ diff --git a/docs/assets/metrics/dimensions.png b/docs/assets/metrics/dimensions.png new file mode 100644 index 0000000000..3da089fbe8 Binary files /dev/null and b/docs/assets/metrics/dimensions.png differ diff --git a/docs/assets/metrics/metrics-tab.png b/docs/assets/metrics/metrics-tab.png new file mode 100644 index 0000000000..edf56fea56 Binary files /dev/null and b/docs/assets/metrics/metrics-tab.png differ diff --git a/docs/assets/metrics/mvtx-histogram.png b/docs/assets/metrics/mvtx-histogram.png new file mode 100644 index 0000000000..c8990fe341 Binary files /dev/null and b/docs/assets/metrics/mvtx-histogram.png differ diff --git a/docs/assets/metrics/quantiles.png b/docs/assets/metrics/quantiles.png new file mode 100644 index 0000000000..f62f798148 Binary files /dev/null and b/docs/assets/metrics/quantiles.png differ diff --git a/docs/assets/metrics/query-window.png b/docs/assets/metrics/query-window.png new file mode 100644 index 0000000000..c9f8435f06 Binary files /dev/null and b/docs/assets/metrics/query-window.png differ diff --git a/docs/assets/metrics/time-range.png b/docs/assets/metrics/time-range.png new file mode 100644 index 0000000000..d45a25683f Binary files /dev/null and b/docs/assets/metrics/time-range.png differ diff --git a/docs/assets/numaflow-ui-advanced-pipeline.png b/docs/assets/numaflow-ui-advanced-pipeline.png deleted file mode 100644 index 3583ed835c..0000000000 Binary files a/docs/assets/numaflow-ui-advanced-pipeline.png and /dev/null differ diff --git a/docs/assets/numaflow-ui-simple-pipeline.png b/docs/assets/numaflow-ui-simple-pipeline.png deleted file mode 100644 index 6940466970..0000000000 Binary files a/docs/assets/numaflow-ui-simple-pipeline.png and /dev/null differ diff --git a/docs/assets/quick-start/cluster-view.png b/docs/assets/quick-start/cluster-view.png new file mode 100644 index 0000000000..646ea2ea7a Binary files /dev/null and b/docs/assets/quick-start/cluster-view.png differ diff --git a/docs/assets/quick-start/default-namespace-view.png b/docs/assets/quick-start/default-namespace-view.png new file mode 100644 index 0000000000..c6ba319faf Binary files /dev/null and b/docs/assets/quick-start/default-namespace-view.png differ diff --git a/docs/assets/quick-start/ui-advanced-pipeline.png b/docs/assets/quick-start/ui-advanced-pipeline.png new file mode 100644 index 0000000000..45c2fa247a Binary files /dev/null and b/docs/assets/quick-start/ui-advanced-pipeline.png differ diff --git a/docs/assets/quick-start/ui-simple-monovertex.png b/docs/assets/quick-start/ui-simple-monovertex.png new file mode 100644 index 0000000000..2a24588627 Binary files /dev/null and b/docs/assets/quick-start/ui-simple-monovertex.png differ diff --git a/docs/assets/quick-start/ui-simple-pipeline.png b/docs/assets/quick-start/ui-simple-pipeline.png new file mode 100644 index 0000000000..311434016b Binary files /dev/null and b/docs/assets/quick-start/ui-simple-pipeline.png differ diff --git a/docs/assets/ui-overview/buffer-info.png b/docs/assets/ui-overview/buffer-info.png new file mode 100644 index 0000000000..117628a83c Binary files /dev/null and b/docs/assets/ui-overview/buffer-info.png differ diff --git a/docs/assets/ui-overview/k8s-events.png b/docs/assets/ui-overview/k8s-events.png new file mode 100644 index 0000000000..385ca15bd2 Binary files /dev/null and b/docs/assets/ui-overview/k8s-events.png differ diff --git a/docs/assets/ui-overview/processing-rates.png b/docs/assets/ui-overview/processing-rates.png new file mode 100644 index 0000000000..e08ec6d9c9 Binary files /dev/null and b/docs/assets/ui-overview/processing-rates.png differ diff --git a/docs/assets/ui-overview/spec-view.png b/docs/assets/ui-overview/spec-view.png new file mode 100644 index 0000000000..8eaabd8c34 Binary files /dev/null and b/docs/assets/ui-overview/spec-view.png differ diff --git a/docs/assets/ui-overview/vertex-view.png b/docs/assets/ui-overview/vertex-view.png new file mode 100644 index 0000000000..385f480cc6 Binary files /dev/null and b/docs/assets/ui-overview/vertex-view.png differ diff --git a/docs/operations/metrics/metrics.md b/docs/operations/metrics/metrics.md index 125cf0da96..fbaf35b7f0 100644 --- a/docs/operations/metrics/metrics.md +++ b/docs/operations/metrics/metrics.md @@ -115,7 +115,7 @@ You can follow the [prometheus operator](https://github.com/prometheus-operator/ You can also set up prometheus operator via [helm](https://bitnami.com/stack/prometheus-operator/helm). -### Configure the below Service/Pod Monitors for scraping your pipeline metrics: +### Configure the below Service/Pod Monitors for scraping your pipeline/monovertex metrics: ```yaml apiVersion: monitoring.coreos.com/v1 diff --git a/docs/quick-start.md b/docs/quick-start.md index 972730b12f..9f5292108e 100644 --- a/docs/quick-start.md +++ b/docs/quick-start.md @@ -1,125 +1,161 @@ # Quick Start -In this page, we will guide you through the steps to: +This guide will walk you through the following steps: -1. Install Numaflow. -2. Create and run a simple pipeline. -3. Create and run an advanced pipeline. +1. [Installing Numaflow](#installing-numaflow) +2. [Creating and running a simple Numaflow Pipeline](#creating-a-simple-pipeline) +3. [Creating and running an advanced Numaflow Pipeline](#creating-an-advanced-pipeline) +4. [Creating and running a Numaflow MonoVertex](#creating-a-monovertex) -## Before you begin: prerequisites +## Before You Begin: Prerequisites -To try Numaflow, you will first need to setup using one of the following options to run container images: +To get started with Numaflow, ensure you have the following tools and setups ready: + +### Container Runtime + +You need a container runtime to run container images. Choose one of the following options: - [Docker Desktop](https://docs.docker.com/get-docker/) -- [podman](https://podman.io/) +- [Podman](https://podman.io/) + +### Local Kubernetes Cluster -Then use one of the following options to create a local Kubernetes cluster: +Set up a local Kubernetes cluster using one of these tools: - [Docker Desktop Kubernetes](https://docs.docker.com/desktop/kubernetes/) - [k3d](https://k3d.io/) - [kind](https://kind.sigs.k8s.io/) - [minikube](https://minikube.sigs.k8s.io/docs/start/) -You will also need `kubectl` to manage the cluster. [Follow these steps to install kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/). In case you need a refresher, all the `kubectl` commands used in this quick start guide can be found in the [kubectl Cheat Sheet](https://kubernetes.io/docs/reference/kubectl/cheatsheet/). +### Kubernetes CLI (`kubectl`) + +Install `kubectl` to manage your Kubernetes cluster. Follow the [official guide](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. If you're unfamiliar with `kubectl`, refer to the [kubectl Quick Reference Page](https://kubernetes.io/docs/reference/kubectl/quick-reference/) for a list of commonly used commands. + +Once these prerequisites are in place, you're ready to proceed with installing and using Numaflow. ## Installing Numaflow -Once you have completed all the prerequisites, run the following command lines to install Numaflow and start the [Inter-Step Buffer Service](./core-concepts/inter-step-buffer-service.md) that handles communication between vertices. +After completing the prerequisites, follow these steps to install Numaflow and set up the [Inter-Step Buffer Service](./core-concepts/inter-step-buffer-service.md), which facilitates communication between pipeline vertices: + +### Create a namespace for Numaflow ```shell kubectl create ns numaflow-system +``` + +### Install Numaflow components + +```shell kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/main/config/install.yaml +``` + +### Deploy the JetStream-based Inter-Step Buffer Service + +```shell kubectl apply -f https://raw.githubusercontent.com/numaproj/numaflow/main/examples/0-isbsvc-jetstream.yaml ``` -## Creating a simple pipeline +Once these steps are complete, Numaflow will be ready for use. + +## Creating a Simple Pipeline -As an example, we will create a `simple pipeline` that contains a source vertex to generate messages, a processing vertex that echos the messages, and a sink vertex that logs the messages. +In this section, we will create a `simple pipeline` that includes a source vertex to generate messages, a processing vertex that echoes the messages, and a sink vertex to log the messages. -Run the command below to create a simple pipeline. +### Deploy the Simple Pipeline + +Run the following command to deploy the simple pipeline: ```shell kubectl apply -f https://raw.githubusercontent.com/numaproj/numaflow/main/examples/1-simple-pipeline.yaml ``` -To view a list of pipelines you've created, run: +### Verify the Pipeline Deployment + +To view the list of pipelines you have created, use the following command: ```shell kubectl get pipeline # or "pl" as a short name ``` -This should create a response like the following, with `AGE` indicating the time elapsed since the creation of your simple pipeline. +You should see an output similar to this, where `AGE` indicates the time elapsed since the pipeline was created: ```shell -NAME PHASE MESSAGE VERTICES AGE -simple-pipeline Running 3 9s +NAME PHASE VERTICES AGE MESSAGE +simple-pipeline Running 3 47s ``` -To inspect the status of the pipeline, use `kubectl get pods`. Note that the pod names will be different from the sample response: +Next, inspect the status of the pipeline by checking the pods. Note that the pod names in your environment may differ from the example below: ```shell # Wait for pods to be ready kubectl get pods -NAME READY STATUS RESTARTS AGE -isbsvc-default-js-0 3/3 Running 0 19s -isbsvc-default-js-1 3/3 Running 0 19s -isbsvc-default-js-2 3/3 Running 0 19s -simple-pipeline-daemon-78b798fb98-qf4t4 1/1 Running 0 10s -simple-pipeline-out-0-xc0pf 1/1 Running 0 10s -simple-pipeline-cat-0-kqrhy 2/2 Running 0 10s -simple-pipeline-in-0-rhpjm 1/1 Running 0 11s +NAME READY STATUS RESTARTS AGE +isbsvc-default-js-0 3/3 Running 0 4m9s +isbsvc-default-js-1 3/3 Running 0 4m9s +isbsvc-default-js-2 3/3 Running 0 4m9s +simple-pipeline-cat-0-xjqbe 3/3 Running 0 99s +simple-pipeline-daemon-784d5cfd97-vpsmk 1/1 Running 0 99s +simple-pipeline-in-0-vvhu1 2/2 Running 0 100s +simple-pipeline-out-0-y1z8e 2/2 Running 0 99s ``` -Now you can watch the log for the `output` vertex. Run the command below and remember to replace `xxxxx` with the appropriate pod name above. +### View Logs for the Output Vertex + +To monitor the logs for the `output` vertex, run the following command. Replace `xxxxx` with the appropriate pod name from the output above: ```shell kubectl logs -f simple-pipeline-out-0-xxxxx ``` -This should generate an output like the sample below: +You should see logs similar to the following: ```shell -2022/08/25 23:59:38 (out) {"Data":"VT+G+/W7Dhc=","Createdts":1661471977707552597} -2022/08/25 23:59:38 (out) {"Data":"0TaH+/W7Dhc=","Createdts":1661471977707615953} -2022/08/25 23:59:38 (out) {"Data":"EEGH+/W7Dhc=","Createdts":1661471977707618576} -2022/08/25 23:59:38 (out) {"Data":"WESH+/W7Dhc=","Createdts":1661471977707619416} -2022/08/25 23:59:38 (out) {"Data":"YEaH+/W7Dhc=","Createdts":1661471977707619936} -2022/08/25 23:59:39 (out) {"Data":"qfomN/a7Dhc=","Createdts":1661471978707942057} -2022/08/25 23:59:39 (out) {"Data":"aUcnN/a7Dhc=","Createdts":1661471978707961705} -2022/08/25 23:59:39 (out) {"Data":"iUonN/a7Dhc=","Createdts":1661471978707962505} -2022/08/25 23:59:39 (out) {"Data":"mkwnN/a7Dhc=","Createdts":1661471978707963034} -2022/08/25 23:59:39 (out) {"Data":"jk4nN/a7Dhc=","Createdts":1661471978707963534} +2025/05/09 11:23:38 (out) Payload - {"Data":{"value":1746789818182898304},"Createdts":1746789818182898304} Keys - [key-0-0] EventTime - 1746789818182 Headers - ID - cat-1526-0-0 +2025/05/09 11:23:38 (out) Payload - {"Data":{"value":1746789818182898304},"Createdts":1746789818182898304} Keys - [key-0-0] EventTime - 1746789818182 Headers - ID - cat-1529-0-0 +2025/05/09 11:23:38 (out) Payload - {"Data":{"value":1746789818182898304},"Createdts":1746789818182898304} Keys - [key-0-0] EventTime - 1746789818182 Headers - ID - cat-1528-0-0 ``` -Numaflow also comes with a built-in user interface. +### Access the Numaflow UI -**NOTE**: Please install the metrics server if your local Kubernetes cluster does not bring it by default (e.g., Kind). -You can install it by running the below command. +Numaflow includes a built-in user interface for monitoring pipelines. If your local Kubernetes cluster does not include a metrics server by default (e.g., Kind), install it using the following commands: ```shell kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml kubectl patch -n kube-system deployment metrics-server --type=json -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]' ``` -To port forward the UI, run the following command. +To access the UI, port-forward the Numaflow server: ```shell -# Port forward the UI to https://localhost:8443/ kubectl -n numaflow-system port-forward deployment/numaflow-server 8443:8443 ``` -This renders the following UI on https://localhost:8443/. +Visit https://localhost:8443/ to view the UI. Below is the UI for the `simple` pipeline: + +#### Cluster View + +![Cluster View](assets/quick-start/cluster-view.png) + +#### Default Namespace View + +![Default Namespace View](assets/quick-start/default-namespace-view.png) + +#### Simple Pipeline View + +![Simple Pipeline View](assets/quick-start/ui-simple-pipeline.png) -![Numaflow UI](assets/numaflow-ui-simple-pipeline.png) +> **Note**: For more details about the UI features and built-in debugging tools, check out the [UI section](./user-guide/UI/). -The pipeline can be deleted by issuing the following command: +### Deleting Pipeline + +To delete the simple pipeline, run the following command: ```shell kubectl delete -f https://raw.githubusercontent.com/numaproj/numaflow/main/examples/1-simple-pipeline.yaml ``` -## Creating an advanced pipeline +## Creating an Advanced Pipeline Now we will walk you through creating an advanced pipeline. In our example, this is called the `even-odd` pipeline, illustrated by the following diagram: @@ -127,68 +163,193 @@ Now we will walk you through creating an advanced pipeline. In our example, this There are five vertices in this example of an advanced pipeline. An [HTTP](./user-guide/sources/http.md) source vertex which serves an HTTP endpoint to receive numbers as source data, a [UDF](user-guide/user-defined-functions/map/map.md) vertex to tag the ingested numbers with the key `even` or `odd`, three [Log](./user-guide/sinks/log.md) sinks, one to print the `even` numbers, one to print the `odd` numbers, and the other one to print both the even and odd numbers. -Run the following command to create the `even-odd` pipeline. +### Deploying the `even-odd` Pipeline + +To deploy the `even-odd` pipeline, run the following command: ```shell kubectl apply -f https://raw.githubusercontent.com/numaproj/numaflow/main/examples/2-even-odd-pipeline.yaml ``` -You may opt to view the list of pipelines you've created so far by running `kubectl get pipeline`. Otherwise, proceed to inspect the status of the pipeline, using `kubectl get pods`. +You can check the list of pipelines you have created so far using: + +```shell +kubectl get pipeline # or "pl" as a short name +``` + +```shell +NAME PHASE VERTICES AGE MESSAGE +even-odd Running 5 51s +``` + +Otherwise, proceed to inspect the status of the pipeline by checking the pods: ```shell # Wait for pods to be ready kubectl get pods NAME READY STATUS RESTARTS AGE -even-odd-daemon-64d65c945d-vjs9f 1/1 Running 0 5m3s -even-odd-even-or-odd-0-pr4ze 2/2 Running 0 30s -even-odd-even-sink-0-unffo 1/1 Running 0 22s -even-odd-in-0-a7iyd 1/1 Running 0 5m3s -even-odd-number-sink-0-zmg2p 1/1 Running 0 7s -even-odd-odd-sink-0-2736r 1/1 Running 0 15s -isbsvc-default-js-0 3/3 Running 0 10m -isbsvc-default-js-1 3/3 Running 0 10m -isbsvc-default-js-2 3/3 Running 0 10m +even-odd-daemon-75cdcd5f4c-nmrrp 1/1 Running 0 95s +even-odd-even-or-odd-0-i72hw 3/3 Running 0 95s +even-odd-even-sink-0-gnhou 2/2 Running 0 95s +even-odd-in-0-tvfef 2/2 Running 0 95s +even-odd-number-sink-0-3s3nc 2/2 Running 0 95s +even-odd-odd-sink-0-zktib 2/2 Running 0 95s +isbsvc-default-js-0 3/3 Running 0 15m +isbsvc-default-js-1 3/3 Running 0 15m +isbsvc-default-js-2 3/3 Running 0 15m ``` -Next, port-forward the HTTP endpoint, and make a `POST` request using `curl`. Remember to replace `xxxxx` with the appropriate pod names both here and in the next step. +### Sending Data to the Pipeline + +Port-forward the HTTP endpoint of the source vertex and send data using `curl`. Replace `xxxxx` with the appropriate pod name: ```shell -kubectl port-forward even-odd-in-0-xxxx 8444:8443 +kubectl port-forward even-odd-in-0-xxxxx 8444:8443 -# Post data to the HTTP endpoint +# Send data to the HTTP endpoint curl -kq -X POST -d "101" https://localhost:8444/vertices/in curl -kq -X POST -d "102" https://localhost:8444/vertices/in curl -kq -X POST -d "103" https://localhost:8444/vertices/in curl -kq -X POST -d "104" https://localhost:8444/vertices/in ``` -Now you can watch the log for the `even` and `odd` vertices by running the commands below. +### Viewing Logs for the `even` and `odd` Vertices + +Monitor the logs for the `even` and `odd` vertices to verify the pipeline's functionality. Replace `xxxxx` with the appropriate pod names: ```shell -# Watch the log for the even vertex +# Logs for the even vertex kubectl logs -f even-odd-even-sink-0-xxxxx -2022/09/07 22:29:40 (even-sink) 102 -2022/09/07 22:29:40 (even-sink) 104 -# Watch the log for the odd vertex +2025/05/09 13:34:26 (even-sink) Payload - 104 Keys - [even] EventTime - 1746797665477 Headers - Content-Length: 3, Content-Type: application/x-www-form-urlencoded, User-Agent: curl/8.7.1, Accept: */*, ID - even-or-odd-1-0-0 +2025/05/09 13:34:26 (even-sink) Payload - 102 Keys - [even] EventTime - 1746797665430 Headers - Content-Type: application/x-www-form-urlencoded, User-Agent: curl/8.7.1, Accept: */*, Content-Length: 3, ID - even-or-odd-2-0-0 + +# Logs for the odd vertex kubectl logs -f even-odd-odd-sink-0-xxxxx -2022/09/07 22:30:19 (odd-sink) 101 -2022/09/07 22:30:19 (odd-sink) 103 + + +2025/05/09 13:34:26 (odd-sink) Payload - 101 Keys - [odd] EventTime - 1746797665407 Headers - Content-Length: 3, Content-Type: application/x-www-form-urlencoded, User-Agent: curl/8.7.1, Accept: */*, ID - even-or-odd-4-0-0 +2025/05/09 13:34:26 (odd-sink) Payload - 103 Keys - [odd] EventTime - 1746797665452 Headers - Content-Length: 3, Content-Type: application/x-www-form-urlencoded, User-Agent: curl/8.7.1, Accept: */*, ID - even-or-odd-3-0-0 ``` -View the UI for the advanced pipeline at https://localhost:8443/. +### Accessing the Numaflow UI -![Numaflow UI](assets/numaflow-ui-advanced-pipeline.png) +To visualize the pipeline, access the Numaflow UI at https://localhost:8443/ after port forwarding. Below is the UI for the `even-odd` pipeline: -The source code of the `even-odd` [user-defined function](user-guide/user-defined-functions/user-defined-functions.md) can be found [here](https://github.com/numaproj/numaflow-go/tree/main/pkg/mapper/examples/even_odd). You also can replace the [Log](./user-guide/sinks/log.md) Sink with some other sinks like [Kafka](./user-guide/sinks/kafka.md) to forward the data to Kafka topics. +![Numaflow UI](assets/quick-start/ui-advanced-pipeline.png) -The pipeline can be deleted by +### Deleting the Pipeline + +To delete the `even-odd` pipeline, run: ```shell kubectl delete -f https://raw.githubusercontent.com/numaproj/numaflow/main/examples/2-even-odd-pipeline.yaml ``` +### Additional Notes + +The source code for the `even-odd` [user-defined function](user-guide/user-defined-functions/user-defined-functions.md) is available [here](https://github.com/numaproj/numaflow-go/tree/main/pkg/mapper/examples/even_odd). You can also replace the [Log](./user-guide/sinks/log.md) Sink with other sinks, such as [Kafka](./user-guide/sinks/kafka.md), to forward data to Kafka topics. + +## Creating a MonoVertex + +This section demonstrates how to create and deploy a simple [MonoVertex](core-concepts/monovertex.md) that includes a [User-Defined Source (UDSource)](user-guide/sources/user-defined-sources.md), a [Transformer](user-guide/sources/transformer/overview.md), and a [User-Defined Sink (UDSink)](user-guide/sinks/user-defined-sinks.md). The sink in this example is a log sink. + +### Deploying the MonoVertex + +To deploy the MonoVertex, run the following command: + +```shell +kubectl apply -f https://raw.githubusercontent.com/numaproj/numaflow/main/examples/21-simple-mono-vertex.yaml +``` + +### Verifying the Deployment + +Check the list of deployed MonoVertices using: + +```shell +kubectl get monovertex # or "mvtx" as a short name +``` + +You should see an output similar to this: + +```shell +NAME PHASE DESIRED CURRENT READY AGE REASON MESSAGE +simple-mono-vertex Running 1 1 1 38s +``` + +Inspect the status of the MonoVertex by listing the pods. Replace `xxxxx` with the appropriate pod name from your environment: + +```shell +# Wait for pods to be ready +kubectl get pods + +NAME READY STATUS RESTARTS AGE +simple-mono-vertex-mv-0-w7fmq 5/5 Running 0 2m30s +simple-mono-vertex-mv-daemon-55bff65db5-mk4g2 1/1 Running 0 2m30s +``` + +### Viewing Pod Details + +To ensure all containers (monitor, udsource, transformer, udsink, and numa) are running, describe the MonoVertex pod: + +```shell +kubectl describe pod simple-mono-vertex-mv-0-xxxxx +``` + +### Monitoring Logs for the Sink Container + +To view logs from the `udsink` container, use the following command. Replace `xxxxx` with the appropriate pod name: + +```shell +kubectl logs -f simple-mono-vertex-mv-0-xxxxx -c udsink +``` + +You should see output similar to: + +```shell +107705990 +107705991 +107705992 +107705993 +107705994 +107705995 +107705996 +107705997 +107705998 +107705999 +``` + +### Accessing the Numaflow UI + +To visualize the MonoVertex, access the Numaflow UI by port-forwarding the Numaflow server: + +```shell +kubectl -n numaflow-system port-forward deployment/numaflow-server 8443:8443 +``` + +Visit https://localhost:8443/ to view the UI. Below is an example of the UI for the MonoVertex: + +![Simple MonoVertex View](assets/quick-start/ui-simple-monovertex.png) + +For more details about the UI features and debugging tools, refer to the UI section.(TODO: add UI section link) + +### Deleting the MonoVertex + +To delete the MonoVertex, run: + +```shell +kubectl delete -f https://raw.githubusercontent.com/numaproj/numaflow/main/examples/21-simple-mono-vertex.yaml +``` + +### Additional Notes + +The source code for the user-defined functions used in this MonoVertex is available here: + +- [UDSource](https://github.com/numaproj/numaflow-rs/blob/main/examples/simple-source/src/main.rs) +- [Transformer](https://github.com/numaproj/numaflow-rs/blob/main/examples/source-transformer-now/src/main.rs) +- [UDSink](https://github.com/numaproj/numaflow-rs/blob/main/examples/sink-log/src/main.rs) + ## A pipeline with reduce (aggregation) To set up an example pipeline with the [Reduce UDF](user-guide/user-defined-functions/reduce/reduce.md), see [Reduce Examples](user-guide/user-defined-functions/reduce/examples.md). diff --git a/docs/user-guide/FAQ.md b/docs/user-guide/FAQ.md index 8a14f72ada..4816e0c711 100644 --- a/docs/user-guide/FAQ.md +++ b/docs/user-guide/FAQ.md @@ -4,32 +4,35 @@ Welcome to the Numaflow FAQ! Here you'll find answers to common questions about --- -## 1. How do I get started with Numaflow? +### 1. How do I get started with Numaflow? To get started, follow the [Quickstart Guide](../quick-start.md). It provides step-by-step instructions for setting up a basic Numaflow pipeline. --- -## 2. How can I check SDK compatibility with Numaflow versions? +### 2. How can I check SDK compatibility with Numaflow versions? Refer to the compatibility matrix [here](../user-guide/sdks/compatibility.md) to ensure your SDK version works with your Numaflow deployment. --- -## 3. Where can I learn about the latest releases of Numaflow? +### 3. Where can I learn about the latest releases of Numaflow? -- Visit the [Numaflow Releases page](https://github.com/numaproj/numaflow/releases) for the latest updates. -- Join our [Slack channel](https://join.slack.com/t/numaproj/shared_invite/zt-19svuv47m-YKHhsQ~~KK9mBv1E7pNzfg) to stay up to date with announcements and community discussions. +- Visit the [Numaflow Releases page](https://github.com/numaproj/numaflow/releases) for the latest updates. +- Join our [Slack channel](https://join.slack.com/t/numaproj/shared_invite/zt-19svuv47m-YKHhsQ~~KK9mBv1E7pNzfg) to stay up to date with announcements and community discussions. --- -## 4. I see `Server Info File not ready` log in the numa container. What does this mean? +### 4. I see `Server Info File not ready` log in the numa container. What does this mean? This message indicates that the server has not started yet. Common reasons include: -- The server was not started correctly in the `udcontainer`. -- The server is still initializing (for example, the application code is downloading cache files or performing setup tasks). +- The server was not started correctly in the `udcontainer`. +- The server is still initializing (for example, the application code is downloading cache files or performing setup tasks). ---- +### 5. What is `partitions` in Numaflow? + +The field `partitions` is the number of streams between the vertices. Larger the parition number, higher the TPS that vertex can take. Note that partitions are owned by the vertex reading the data, to create a +multi-partitioed edge we need to configure the vertex reading the data to have multiple partitions. -If you have additional questions, please refer to the [documentation](../README.md) or reach out on Slack! +If you have additional questions, please refer to the [documentation](../README.md) or reach out on [Slack](https://join.slack.com/t/numaproj/shared_invite/zt-19svuv47m-YKHhsQ~~KK9mBv1E7pNzfg)! diff --git a/docs/user-guide/UI/errors.md b/docs/user-guide/UI/errors.md new file mode 100644 index 0000000000..907d93d562 --- /dev/null +++ b/docs/user-guide/UI/errors.md @@ -0,0 +1,86 @@ +# Errors + +The **Numaflow UI** provides an **Errors Tab** that lists the recent errors (up to 10 per container) that may have occurred in the user-defined function (UDF) code. This feature helps users quickly identify and debug application errors. + +In this guide, we will simulate a panic in a transformer and demonstrate how the errors are displayed in the Errors Tab for a monovertex. + +> **Note**: The Errors Tab and its related functionality are currently available only for MonoVertex. Support for Pipelines is planned for future releases. + +--- + +## High-Level Architecture + +![High-Level Architecture](../../assets/errors/errors-architecture.png) + +--- + +## Simulating an Error in the Transformer + +To simulate an error, we will introduce a panic in the Go SDK for the transformer by modifying the `Transform` function. You can find the relevant code [here](https://github.com/numaproj/numaflow-go/blob/main/pkg/sourcetransformer/examples/assign_event_time/main.go). + +Below is the modified code snippet: + +```go +// AssignEventTime is a source transformer that assigns event time to the message. +type AssignEventTime struct { + counter uint64 +} + +func (a *AssignEventTime) Transform(ctx context.Context, keys []string, d sourcetransformer.Datum) sourcetransformer.Messages { + newCount := atomic.AddUint64(&a.counter, 1) + // Trigger a panic if the counter is a multiple of 5 + if newCount%5 == 0 { + panic("Counter reached a multiple of 5") + } + // Update the message's event time to the current time + eventTime := time.Now() + // Log the action for testing purposes + log.Printf("AssignEventTime: Assigning event time %v to message %s", eventTime, string(d.Value())) + return sourcetransformer.MessagesBuilder().Append(sourcetransformer.NewMessage(d.Value(), eventTime).WithKeys(keys)) +} +``` + +In this example, the code triggers a panic whenever the counter reaches a multiple of 5. This simulates an error scenario that will be captured and displayed in the Errors Tab. + +--- + +## The UI Errors Tab + +![Errors Tab](../../assets/errors/ui-errors-tab.png) + +The Errors Tab provides the following features: + +- **Error Count**: Displays the total number of errors across all pods and containers at the top level. +- **Pod and Container Filters**: Allows users to filter errors by selecting a specific pod or container for more focused debugging. +- **Tabulated Errors**: Errors are displayed in a tabular format with the following columns: + + - `Pod Name` + - `Container` + - `Message` + - `Last Occurred` + +- **Details**: Expanding an error entry reveals a `Details` section that includes the **stack trace** of the error, providing deeper insights for debugging. + +--- + +## Types of Errors Displayed in the Errors Tab + +The Errors Tab in the **Numaflow UI** captures and displays various types of errors that may occur during the execution of user-defined functions (UDFs). Below are the common categories of errors you can expect: + +### 1. Exceptions in User Code + +Any exception that occurs within the user-defined function (UDF) code will be captured and displayed in the Errors Tab. These errors typically indicate issues in the logic or implementation of the UDF. + +### 2. Partial Responses + +Errors may occur when the user code sends partial or incorrect responses. For example: + +- Returning `null` instead of `Message.toDrop` in a UDSink when a message needs to be dropped. + +### 3. Critical Errors Persisted by the User + +Users can invoke the `PersistCriticalError` utility function to log critical errors in the `emptyDir` volume. This is useful for capturing severe issues that require immediate attention. Below are links to the implementations in different SDKs: + +- [Go Implementation](https://github.com/numaproj/numaflow-go/blob/47460a1854b8f58a0e918056ef4d169949193ebe/pkg/errors/errors.go) +- [Java Implementation](https://github.com/numaproj/numaflow-java/blob/2aeaafff1c6dec7fd66e018b142ef6fe4ffcf0a9/src/main/java/io/numaproj/numaflow/errors/PersistCriticalError.java) +- [Python Implementation](https://github.com/numaproj/numaflow-python/blob/6ccd49ef09ad11eac9b0bb9a6b0f5517d858bea4/pynumaflow/errors/errors.py) diff --git a/docs/user-guide/UI/logs.md b/docs/user-guide/UI/logs.md new file mode 100644 index 0000000000..39e43a9730 --- /dev/null +++ b/docs/user-guide/UI/logs.md @@ -0,0 +1,110 @@ +# Logs View + +The **Logs View** allows users to inspect the logs of a specific container within a pod of a vertex. This guide will walk you through the Logs tab and its various features. + +--- + +## Navigating to the Logs Tab + +**Select a Pod** + +Navigate to the `Pods View` tab after selecting the vertex and select a pod by name from the `Select a Pod by Name` dropdown. + +![Select Pod](../../assets/logs/select-pod.png) + +**Select a Container** + +Choose a container from the `Select a Container` section. + +![Select Container](../../assets/logs/select-container.png) + +**View Logs** + +Open the **Logs Tab** on the right to view the container logs. + +![Logs View](../../assets/logs/logs-tab.png) + +--- + +## Features + +### 1. Previous Terminated Container Logs + +- Enable the checkbox to view logs from previously terminated containers. +- This is particularly useful for debugging issues. + +--- + +### 2. Search Logs + +- Filter logs by typing keywords in the **Search Logs** box. + +![Search Logs](../../assets/logs/logs-search.png) + +--- + +### 3. Negate Search + +- Enable the **Negate Search** option to exclude logs matching the search keywords from the view. + +![Negate Search](../../assets/logs/logs-negate-search.png) + +--- + +### 4. Wrap Lines + +- Use the **Wrap Lines** feature to avoid horizontal scrolling for long log lines, improving readability. + +![Wrap Lines](../../assets/logs/logs-wrap.png) + +--- + +### 5. Pause Logs + +- Pause the log stream to inspect logs when there is a high volume of data. + +![Pause Logs](../../assets/logs/logs-pause.png) + +--- + +### 6. Dark Mode + +- Toggle between **Dark Mode** and **Light Mode** for better visibility based on your preference. + +![Dark Mode](../../assets/logs/logs-dark-mode.png) + +--- + +### 7. Ascending/Descending Order + +- Switch between ascending and descending order of log timestamps for easier navigation. + +![Ascending Order](../../assets/logs/logs-ascending.png) + +--- + +### 8. Download Logs + +- Download the last 1000 logs for offline analysis. + +--- + +### 9. Add/Remove Timestamps + +- Toggle timestamps in the logs based on your requirements. + +![Add Timestamps](../../assets/logs/logs-add-timestamps.png) + +--- + +### 10. Level-Based Filtering + +- Filter logs by log levels such as: + - **Info** + - **Error** + - **Warn** + - **Debug** + +![Error Filter](../../assets/logs/logs-filter-error.png) + +--- diff --git a/docs/user-guide/UI/metrics-tab.md b/docs/user-guide/UI/metrics-tab.md new file mode 100644 index 0000000000..ded6d81a81 --- /dev/null +++ b/docs/user-guide/UI/metrics-tab.md @@ -0,0 +1,251 @@ +# Metrics Tab + +Numaflow provides a comprehensive [set of Prometheus metrics](../../operations/metrics/metrics.md), enabling users to monitor their pipelines effectively and set up alerts as needed. This feature enhances the debugging experience by offering contextual insights directly within the UI. By leveraging these metrics, users can identify and resolve issues more efficiently, ensuring smoother pipeline operations. + +--- + +## Prerequisites + +Before visualizing metrics in the UI, ensure the following requirements are met: + +- **Prometheus Server Setup**: A Prometheus server must be configured and running. Refer to the [Prometheus Operator Installation Guide](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/getting-started/installation.md) for detailed instructions. +- **Service Monitors**: Configure Service/Pod Monitors for scraping pipeline/monovertex metrics. Refer to [this section](../../operations/metrics/metrics.md#configure-the-below-servicepod-monitors-for-scraping-your-pipelinemonovertex-metrics) for configuration details. +- **Basic Knowledge of PromQL**: Familiarity with PromQL (Prometheus Query Language) is recommended to effectively query and interpret the metrics. + +Meeting these prerequisites will help you make the most of the metrics visualization and monitoring features in Numaflow. + +--- + +## Metrics Configuration + +To visualize metrics in the UI, you need to configure them using a ConfigMap. Numaflow provides a default [ConfigMap](https://github.com/numaproj/numaflow/blob/main/config/base/numaflow-server/numaflow-server-metrics-proxy-config.yaml) that includes a variety of pre-configured metrics. These metrics cover essential pipeline operations and also include custom metrics, such as CPU and memory usage. + +You can customize this ConfigMap to add or modify metrics based on your specific requirements. For example, you can include additional Prometheus metrics relevant to your use case. + +Refer to the [Numaflow Metrics Documentation](../../operations/metrics/metrics.md) for more details on available metrics and their usage. + +--- + +### Current Configuration Snippet + +Below is a snippet of the default ConfigMap configuration: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: numaflow-server-metrics-proxy-config +data: + config.yaml: | + url: http://prometheus-operated.monitoring.svc.cluster.local:9090 + patterns: + - name: vertex_gauge + objects: + - vertex + title: Vertex Gauge Metrics + description: This pattern represents the gauge metrics for a vertex across different dimensions. + expr: | + sum($metric_name{$filters}) by ($dimension, period) + params: + - name: start_time + required: false + - name: end_time + required: false + metrics: + - metric_name: vertex_pending_messages + display_name: Vertex Pending Messages + metric_description: Tracks the total number of messages waiting to be processed over varying time frames (e.g., 1min, 5min, 15min). + required_filters: + - namespace + - pipeline + - vertex + dimensions: + - name: pod + filters: + - name: pod + required: false + - name: period + required: false + - name: vertex + filters: + - name: period + required: false +``` + +### Key Definitions + +Prometheus metrics are categorized into three main types: Histogram, Gauge, and Counter. The configuration provided above groups metrics of a similar type under a single pattern, making it more generic and easier to manage. + +1. **url:** + The `url` is a required field that specifies the Prometheus service endpoint to which the metrics proxy will connect. If the `url` is not set or is incorrectly configured, the Metrics Tab will not be displayed in the UI. + + - **Example**: If you set up a local Prometheus operator using [Helm](https://bitnami.com/stack/prometheus-operator/helm), the `url` would look like this: + `http://my-release-kube-prometheus-prometheus.default.svc.cluster.local:9090` + +2. **patterns:** + A list of patterns that group metrics of a similar type. + + - **name**: The name of the pattern. + - **objects**: Specifies the object type, which can either be `vertex` or `mono-vertex`. For pipelines, the object is `vertex`, and for MonoVertex, the object is `mono-vertex`. + - **title**: The title of the pattern. + - **description**: A description of the pattern. + - **expr**: The PromQL expression used to construct queries, with placeholders for dynamic values. + - **params**: Common parameters for all metrics within the pattern. These may include: + + - **start_time**: The start time for the PromQL query (optional). + - **end_time**: The end time for the PromQL query (optional). + - **duration**: The query window (required for histograms). + - **quantile**: The quantile value (required for histograms). The quantiles can be 0.99, 0.95, 0.90, or 0.50. + + - **metrics**: + A list of metrics defined within the pattern. + - **metric_name**: The actual name of the metric. + - **display_name**: A user-friendly name for the metric. Avoid editing this for existing metrics. + - **metric_description**: A detailed description of the metric, displayed as an info icon next to the metric name in the UI. + - **required_filters**: Filters that must be included in the PromQL request body. + - **dimensions**: Dimensions allow users to drill down into specific components, such as pods or containers, for more granular data. + - **name**: The name of the dimension (e.g., pod, vertex). + - **filters**: Filters applied to the data for a specific key. + - **name**: The name of the filter. + - **required**: If set to `true`, the filter is automatically added to `$filters`. If `false`, users can select a value for the filter key, which is then added to `$filters`. + - **expr**: (Optional) Overrides the top-level `expr` for a specific metric and dimension. + +This structured configuration ensures flexibility and ease of use when visualizing and analyzing metrics in the Numaflow UI. + +--- + +## Explanation with an Example + +The above configuration might seem complex at first glance. Let’s break it down with an example using the out-of-the-box metric `monovtx_processing_time_bucket`. + +### Example Configuration + +```yaml +- name: mono_vertex_histogram + objects: + - mono-vertex + title: MonoVertex Histogram Metrics + description: This pattern is for P99, P95, P90, and P50 quantiles for a mono-vertex across different dimensions. + expr: | + histogram_quantile($quantile, sum by($dimension,le) (rate($metric_name{$filters}[$duration]))) + params: + - name: quantile + required: true + - name: duration + required: true + - name: start_time + required: false + - name: end_time + required: false + metrics: + - metric_name: monovtx_processing_time_bucket + display_name: MonoVertex Processing Time Latency + metric_description: This metric represents a histogram to keep track of the total time taken to forward a chunk of messages. + required_filters: + - namespace + - mvtx_name + dimensions: + - name: mono-vertex + - name: pod + filters: + - name: pod + required: false +``` + +### Key Points + +1. **Pattern Name**: `mono_vertex_histogram` + This indicates that the pattern includes histogram metrics for a mono-vertex. + +2. **Expression (`expr`)** + The expression calculates quantiles (e.g., P99, P95). Learn more about quantiles in Prometheus [here](https://prometheus.io/docs/practices/histograms/#quantiles). + Placeholders such as `quantile`, `dimension`, `metric_name`, `filters`, and `duration` are dynamically populated based on the configuration. + +3. **Parameters (`params`)** + + - `quantile` and `duration` are required placeholders in the expression. + - `start_time` and `end_time` are optional parameters for defining the query duration. + +4. **Metric Name (`metric_name`)** + This forms the `$metric_name` placeholder in the expression. + +5. **Required Filters (`required_filters`)** + These filters populate the `$filters` placeholder in the expression. + +6. **Dimensions** + + - For the `mono-vertex` dimension, no additional filters are applied. + - For the `pod` dimension, an additional filter (`pod`) is appended to the `$filters` placeholder. + +--- + +### Example Expressions + +Below are examples of how the placeholders are replaced to form the final PromQL expressions: + +- **Dimension**: `mono-vertex` + **Quantile**: `0.99` + **Namespace**: `default` + **MonoVertex Name**: `simple-mono-vertex` + **Duration**: `5m` + **Resultant Expression**: + + ```promql + histogram_quantile(0.99, sum by(mvtx_name,le) (rate(monovtx_processing_time_bucket{namespace="default",mvtx_name="simple-mono-vertex"}[5m]))) + ``` + +- **Dimension**: `pod` + **Quantile**: `0.99` + **Namespace**: `default` + **MonoVertex Name**: `simple-mono-vertex` + **Pod**: `simple-mono-vertex-mv-0-edj2s` + **Duration**: `1m` + **Resultant Expression**: + + ```promql + histogram_quantile(0.99, sum by(pod,le) (rate(monovtx_processing_time_bucket{namespace="default",mvtx_name="simple-mono-vertex",pod="simple-mono-vertex-mv-0-edj2s"}[1m]))) + ``` + +This example demonstrates how the configuration translates into actionable PromQL queries, making it easier to understand and customize. + +## How the Metrics Tab Appears in the UI + +The **Metrics Tab** is located next to the [Logs Tab](./logs.md) in the [Pods View](./pods-view.md). + +![Metrics Tab](../../assets/metrics/metrics-tab.png) + +The following screenshots illustrate how the UI translates the discussed configuration into a visual representation: + +### MonoVertex Histogram Example + +The UI displays the configured metrics, such as the `monovtx_processing_time_bucket` histogram, with options to select dimensions and filters. + +![MonoVertex Histogram](../../assets/metrics/mvtx-histogram.png) + +### Dimension Selection + +Users can choose dimensions for the metric, such as `MonoVertex` or `Pod`, to view data at different levels of granularity. After selecting `Pod` dimension, we see that `Filters` box appears which allow us to add filters for a dimension. Here, `pod` dimension has pod as a filter. + +![Dimensions](../../assets/metrics/dimensions.png) + +### Quantile Selection + +For histogram metrics, users can select quantile values (e.g., P99, P95) from the available options, as explained in the [Key Definitions](#key-definitions) section. + +![Quantiles](../../assets/metrics/quantiles.png) + +### Query Window/Duration + +The query window specifies the time range over which the `rate()` function calculates the per-second average rate of increase for each histogram bucket. + +![Query Window](../../assets/metrics/query-window.png) + +### Time Range Selection + +Users can select a predefined time range or set a custom time range for their PromQL queries. + +![Time Range](../../assets/metrics/time-range.png) + +## Adding Custom Metrics + +To add metrics not provided by Numaflow, follow the configuration patterns described above. For example, you can add a custom metric similar to the `pod_cpu_memory_utilization` pattern [here](#current-configuration). diff --git a/docs/user-guide/UI/overview.md b/docs/user-guide/UI/overview.md new file mode 100644 index 0000000000..d31c8aeabe --- /dev/null +++ b/docs/user-guide/UI/overview.md @@ -0,0 +1,70 @@ +# Numaflow UI Overview + +Numaflow provides a built-in user interface (UI) for monitoring and managing your data pipelines. + +### Accessing the Numaflow UI + +To access the Numaflow UI, use the following command to port-forward the Numaflow server: + +```shell +kubectl -n numaflow-system port-forward deployment/numaflow-server 8443:8443 +``` + +Once port-forwarding is active, open your browser and navigate to [https://localhost:8443](https://localhost:8443). + +### UI Views + +We have already walked through some of the views of UI to monitor pipelines in the [Quick Start](../../quick-start.md) guide. + +- **Cluster View** +- **Namespace View** +- **Simple Pipeline View** + +### Vertex View + +Vertex View provides detailed insights into each pipeline vertex. The following features are available: + +#### **[Pods View](./pods-view.md)** + +Inspect the status and details of pods running in your pipelines. + +#### **Spec** + +View the specification of the vertex. + +![spec view](../../assets/ui-overview/spec-view.png) + +#### **Processing Rates** + +See the last 1m, 5m, and 15m processing rates for each partition of the vertex. + +If a Prometheus server is configured, you can click on the number to see more details in the Metrics tab. + +![Processing Rates](../../assets/ui-overview/processing-rates.png) + +#### **Kubernetes Events** + +View Kubernetes events related to the vertex. + +![k8s events](../../assets/ui-overview/k8s-events.png) + +#### **[Errors](./errors.md)** + +Review errors detected in your pipelines for quick debugging. + +#### **Buffers** + +See buffer details for every partition, including buffer length, usage, and number of pending messages. +Click on the pending number to view complete pending metrics in the [Metrics tab](./metrics-tab.md). + +![Buffer](../../assets/ui-overview/buffer-info.png) + +#### **[Logs](./logs.md)** + +View logs for different containers of pods to help diagnose issues. + +#### **[Metrics](./metrics-tab.md)** + +Monitor pipeline metrics for performance. + +--- diff --git a/docs/user-guide/UI/pods-view.md b/docs/user-guide/UI/pods-view.md new file mode 100644 index 0000000000..6d9c675185 --- /dev/null +++ b/docs/user-guide/UI/pods-view.md @@ -0,0 +1,84 @@ +# Pods View + +The **Pods View** in the Numaflow UI provides a comprehensive overview of the pods associated with a vertex. This view is designed to help users monitor the status, resource usage, and other key details of the pods in their application. It also allows users to perform actions such as filtering and inspecting individual pods. + +--- + +## Features of the Pods View + +### 1. Select a Pod by Name + +- Displays a list of all pods associated with the selected vertex. +- Users can select a pod from the dropdown by its name. + +--- + +### 2. Select a Pod by Resource + +- **CPU** and **Memory** usage of the containers within pods are displayed in a hexagon heat map. +- Users can select a pod by clicking on any of these hexagons. + - This is particularly useful when a pod's heat map is red, indicating high resource usage, and you want to inspect that specific pod. + +--- + +### 3. Select a Container + +- Allows users to select a container within a pod to view its [info](#container-info) and [logs](#container-logs). + +--- + +### 4. Container Info + +Provides detailed information about the selected container, including: + +- **Name**: The name of the container. +- **Status**: The current state of the container (e.g., `Waiting`, `Running`, or `Terminated`). +- **Last Started At**: The relative age of the container since it was last (re-)started. +- **CPU**: The CPU usage compared to the requested amount. +- **Memory**: The memory usage compared to the requested amount. +- **Restart Count**: The number of times the container has been restarted. +- **Last Termination Reason**: The reason for the container's last termination (if applicable). This helps debug container crashes and restarts. +- **Last Termination Message**: A message providing details about the last termination of the container (if applicable). +- **Waiting Reason**: The reason why the container is in a waiting state (if applicable). +- **Waiting Message**: A message providing details about why the container is in a waiting state (if applicable). + +--- + +### 5. Container Logs + +- Quickly access the logs of a specific container within a pod by selecting it. +- Refer to the [Logs View](./logs.md) section for a detailed explanation of log features. + +--- + +### 6. Pod Info + +Provides detailed information about the selected pod, including: + +- **Name**: The name of the pod. +- **Status**: The current phase of the pod. Possible values include: + - `Pending` + - `Running` + - `Succeeded` + - `Failed` + - `Unknown` + + > **Note**: Learn more about pod phases [here](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase). + +- **Restart Count**: The total number of restarts for all containers within the pod. +> Since pods are recreated upon termination, this value is an aggregate of container restarts. +- **CPU**: The aggregated CPU usage of all containers within the pod compared to the requested amount. +- **Memory**: The aggregated memory usage of all containers within the pod compared to the requested amount. +- **Reason**: A brief, CamelCase message indicating why the pod is in its current state. + - This field is populated in cases such as pod scheduling issues, eviction, termination, or node-related problems. +- **Message**: A human-readable message providing additional details about the pod's condition. + - This field is populated in scenarios like pod scheduling issues, eviction, termination, or node-related problems. + +--- + +### 7. Metrics Tab + +- The **Pods View** includes a **Metrics Tab** located next to the Logs Tab. +- For a detailed discussion about the Metrics Tab, refer to the [Metrics](./metrics-tab.md) section. + +--- diff --git a/docs/user-guide/sources/kafka.md b/docs/user-guide/sources/kafka.md index 3f3db417f3..dadc8a43fd 100644 --- a/docs/user-guide/sources/kafka.md +++ b/docs/user-guide/sources/kafka.md @@ -113,7 +113,7 @@ spec: ### How to start the Kafka Source from a specific offset based on datetime? In order to start the Kafka Source from a specific offset based on datetime, we need to reset the offset before we start the pipeline. -For example, we have a topic `quickstart-events` with 3 partitions and a consumer group `console-consumer-94457`. This example uses [Kafka 3.6.1](https://downloads.apache.org/kafka/3.6.1/RELEASE_NOTES.html) and localhost. +For example, we have a topic `quickstart-events` with 3 partitions and a consumer group `console-consumer-94457`. This example uses [Kafka 3.6.1](https://archive.apache.org/dist/kafka/3.6.1/RELEASE_NOTES.html) and localhost. ```shell ➜ kafka_2.13-3.6.1 bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic quickstart-events Topic: quickstart-events TopicId: WqIN6j7hTQqGZUQWdF7AdA PartitionCount: 3 ReplicationFactor: 1 Configs: @@ -172,4 +172,4 @@ bin/kafka-consumer-groups.sh --bootstrap-server --comman Reference: - [How to Use Kafka Tools With Confluent Cloud](https://docs.confluent.io/kafka/operations-tools/use-kafka-tools-ccloud.html#create-a-configuration-file) -- [Apache Kafka Security](https://kafka.apache.org/documentation/#security) +- [Apache Kafka Security](https://kafka.apache.org/documentation/#security) \ No newline at end of file diff --git a/docs/user-guide/user-defined-functions/map/examples.md b/docs/user-guide/user-defined-functions/map/examples.md index 605f660342..02fc6c02b5 100644 --- a/docs/user-guide/user-defined-functions/map/examples.md +++ b/docs/user-guide/user-defined-functions/map/examples.md @@ -7,10 +7,10 @@ Please read [map](./map.md) to get the best out of these examples. ### Inter-Step Buffer Service (ISB Service) #### What is ISB Service? + An Inter-Step Buffer Service is described by a [Custom Resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/), which is used to pass data between vertices of a numaflow pipeline. Please refer to the doc [Inter-Step Buffer Service](../../../core-concepts/inter-step-buffer.md) for more information on ISB. - #### How to install the ISB Service ```shell @@ -20,7 +20,7 @@ kubectl apply -f https://raw.githubusercontent.com/numaproj/numaflow/stable/exam The expected output of the above command is shown below: ```shell -$ kubectl get isbsvc +$ kubectl get isbsvc NAME TYPE PHASE MESSAGE AGE default jetstream Running 3d19h @@ -110,7 +110,7 @@ kubectl logs -f even-odd-odd-sink-0-xxxxx View the UI for a pipeline at https://localhost:8443/. -![Numaflow UI](../../../assets/numaflow-ui-advanced-pipeline.png) +![Numaflow UI](../../../assets/quick-start/ui-advanced-pipeline.png) The source code of the `even-odd` [user-defined function](../user-defined-functions.md) can be found [here](https://github.com/numaproj/numaflow-go/tree/main/pkg/mapper/examples/even_odd). You also can replace the [Log](../../sinks/log.md) Sink with some other sinks like [Kafka](../../sinks/kafka.md) to forward the data to Kafka topics. @@ -118,4 +118,4 @@ The pipeline can be deleted by ```shell kubectl delete -f https://raw.githubusercontent.com/numaproj/numaflow/main/examples/2-even-odd-pipeline.yaml -``` \ No newline at end of file +``` diff --git a/mkdocs.yml b/mkdocs.yml index e4944860c3..f8b6adffdc 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -55,6 +55,7 @@ nav: - user-guide/sources/kafka.md - user-guide/sources/pulsar.md - user-guide/sources/nats.md + - SQS Source: user-guide/sources/sqs.md - user-guide/sources/user-defined-sources.md - Data Transformer: - Overview: "user-guide/sources/transformer/overview.md" @@ -122,6 +123,13 @@ nav: - Use Cases: - user-guide/use-cases/overview.md - user-guide/use-cases/monitoring-and-observability.md + - UI: + - Overview: "user-guide/UI/overview.md" + - Errors: "user-guide/UI/errors.md" + - Logs: "user-guide/UI/logs.md" + - Metrics: "user-guide/UI/metrics-tab.md" + - Pods View: "user-guide/UI/pods-view.md" + - FAQs: user-guide/FAQ.md - Operator Manual: - Releases ⧉: "operations/releases.md" - operations/installation.md @@ -150,4 +158,4 @@ nav: - development/debugging.md - development/static-code-analysis.md - development/releasing.md - - Numaproj: https://numaproj.io + - Numaproj: https://numaproj.io \ No newline at end of file