Skip to content

feat(doc): update quick start and UI docs #2622

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
May 25, 2025
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -311,7 +311,7 @@ docs-serve: docs

.PHONY: docs-linkcheck
docs-linkcheck: /usr/local/bin/lychee
lychee --exclude-path=CHANGELOG.md *.md --include "https://github.com/numaproj/*" $(shell find ./test -type f) $(wildcard ./docs/*.md)
lychee --exclude-path=CHANGELOG.md --exclude-path=./docs/APIs.md --exclude "https://localhost:*" --exclude "http://localhost:*" --exclude "http://127.0.0.1*" *.md $(shell find ./test -type f) $(shell find ./docs -name '*.md')

# pre-push checks

Expand Down
2 changes: 1 addition & 1 deletion USERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
Please add your company name and initial use case (optional) below.

1. [Intuit](https://www.intuit.com/) - Streaming ML inference and to prepare data for ML model training in real-time.
2. [B|Cubed](https://bcubed-corp.com/) - Digital Signal Processing Communication Systems. We receive RF energy and use numaflows to transform the RF to individual bits into intelligible data.
2. [B|Cubed](https://bcubed.com/) - Digital Signal Processing Communication Systems. We receive RF energy and use numaflows to transform the RF to individual bits into intelligible data.
3. [Atlan](https://atlan.com/) - Numaflow powers real time notifications, stream processing ecosystem at Atlan.
4. [Valegachain Analytics](https://www.valegachain.com/) Numaflow is used to extract, transform, and load cryptocurrency blocks and mempool transactions in data lakes, as well as for activity alerts.
5. [Lockheed Martin](https://lockheedmartin.com/) Perform ELT processing on high and low volume data streams of sensor data as recieved from IOT type systems.
Expand Down
Binary file added docs/assets/errors/errors-architecture.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/errors/ui-errors-tab.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/logs/logs-add-timestamps.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/logs/logs-ascending.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/logs/logs-dark-mode.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/logs/logs-filter-error.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/logs/logs-negate-search.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/logs/logs-pause.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/logs/logs-search.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/logs/logs-tab.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/logs/logs-wrap.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/logs/select-container.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/logs/select-pod.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/metrics/dimensions.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/metrics/metrics-tab.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/metrics/mvtx-histogram.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/metrics/quantiles.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/metrics/query-window.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/metrics/time-range.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed docs/assets/numaflow-ui-advanced-pipeline.png
Binary file not shown.
Binary file removed docs/assets/numaflow-ui-simple-pipeline.png
Binary file not shown.
Binary file added docs/assets/quick-start/cluster-view.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/ui-overview/buffer-info.png
Binary file added docs/assets/ui-overview/k8s-events.png
Binary file added docs/assets/ui-overview/processing-rates.png
Binary file added docs/assets/ui-overview/spec-view.png
Binary file added docs/assets/ui-overview/vertex-view.png
2 changes: 1 addition & 1 deletion docs/operations/metrics/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ You can follow the [prometheus operator](https://github.com/prometheus-operator/

You can also set up prometheus operator via [helm](https://bitnami.com/stack/prometheus-operator/helm).

### Configure the below Service/Pod Monitors for scraping your pipeline metrics:
### Configure the below Service/Pod Monitors for scraping your pipeline/monovertex metrics:

```yaml
apiVersion: monitoring.coreos.com/v1
Expand Down
305 changes: 233 additions & 72 deletions docs/quick-start.md

Large diffs are not rendered by default.

23 changes: 13 additions & 10 deletions docs/user-guide/FAQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,32 +4,35 @@ Welcome to the Numaflow FAQ! Here you'll find answers to common questions about

---

## 1. How do I get started with Numaflow?
### 1. How do I get started with Numaflow?

To get started, follow the [Quickstart Guide](../quick-start.md). It provides step-by-step instructions for setting up a basic Numaflow pipeline.

---

## 2. How can I check SDK compatibility with Numaflow versions?
### 2. How can I check SDK compatibility with Numaflow versions?

Refer to the compatibility matrix [here](../user-guide/sdks/compatibility.md) to ensure your SDK version works with your Numaflow deployment.

---

## 3. Where can I learn about the latest releases of Numaflow?
### 3. Where can I learn about the latest releases of Numaflow?

- Visit the [Numaflow Releases page](https://github.com/numaproj/numaflow/releases) for the latest updates.
- Join our [Slack channel](https://join.slack.com/t/numaproj/shared_invite/zt-19svuv47m-YKHhsQ~~KK9mBv1E7pNzfg) to stay up to date with announcements and community discussions.
- Visit the [Numaflow Releases page](https://github.com/numaproj/numaflow/releases) for the latest updates.
- Join our [Slack channel](https://join.slack.com/t/numaproj/shared_invite/zt-19svuv47m-YKHhsQ~~KK9mBv1E7pNzfg) to stay up to date with announcements and community discussions.

---

## 4. I see `Server Info File not ready` log in the numa container. What does this mean?
### 4. I see `Server Info File not ready` log in the numa container. What does this mean?

This message indicates that the server has not started yet. Common reasons include:

- The server was not started correctly in the `udcontainer`.
- The server is still initializing (for example, the application code is downloading cache files or performing setup tasks).
- The server was not started correctly in the `udcontainer`.
- The server is still initializing (for example, the application code is downloading cache files or performing setup tasks).

---
### 5. What is `partitions` in Numaflow?

The field `partitions` is the number of streams between the vertices. Larger the parition number, higher the TPS that vertex can take. Note that partitions are owned by the vertex reading the data, to create a
multi-partitioed edge we need to configure the vertex reading the data to have multiple partitions.

If you have additional questions, please refer to the [documentation](../README.md) or reach out on Slack!
If you have additional questions, please refer to the [documentation](../README.md) or reach out on [Slack](https://join.slack.com/t/numaproj/shared_invite/zt-19svuv47m-YKHhsQ~~KK9mBv1E7pNzfg)!
86 changes: 86 additions & 0 deletions docs/user-guide/UI/errors.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Errors

The **Numaflow UI** provides an **Errors Tab** that lists the recent errors (up to 10 per container) that may have occurred in the user-defined function (UDF) code. This feature helps users quickly identify and debug application errors.

In this guide, we will simulate a panic in a transformer and demonstrate how the errors are displayed in the Errors Tab for a monovertex.

> **Note**: The Errors Tab and its related functionality are currently available only for MonoVertex. Support for Pipelines is planned for future releases.

---

## High-Level Architecture

![High-Level Architecture](../../assets/errors/errors-architecture.png)

---

## Simulating an Error in the Transformer

To simulate an error, we will introduce a panic in the Go SDK for the transformer by modifying the `Transform` function. You can find the relevant code [here](https://github.com/numaproj/numaflow-go/blob/main/pkg/sourcetransformer/examples/assign_event_time/main.go).

Below is the modified code snippet:

```go
// AssignEventTime is a source transformer that assigns event time to the message.
type AssignEventTime struct {
counter uint64
}

func (a *AssignEventTime) Transform(ctx context.Context, keys []string, d sourcetransformer.Datum) sourcetransformer.Messages {
newCount := atomic.AddUint64(&a.counter, 1)
// Trigger a panic if the counter is a multiple of 5
if newCount%5 == 0 {
panic("Counter reached a multiple of 5")
}
// Update the message's event time to the current time
eventTime := time.Now()
// Log the action for testing purposes
log.Printf("AssignEventTime: Assigning event time %v to message %s", eventTime, string(d.Value()))
return sourcetransformer.MessagesBuilder().Append(sourcetransformer.NewMessage(d.Value(), eventTime).WithKeys(keys))
}
```

In this example, the code triggers a panic whenever the counter reaches a multiple of 50. This simulates an error scenario that will be captured and displayed in the Errors Tab.

---

## The UI Errors Tab

![Errors Tab](../../assets/errors/ui-errors-tab.png)

The Errors Tab provides the following features:

- **Error Count**: Displays the total number of errors across all pods and containers at the top level.
- **Pod and Container Filters**: Allows users to filter errors by selecting a specific pod or container for more focused debugging.
- **Tabulated Errors**: Errors are displayed in a tabular format with the following columns:

- `Pod Name`
- `Container`
- `Message`
- `Last Occurred`

- **Details**: Expanding an error entry reveals a `Details` section that includes the stack trace of the error, providing deeper insights for debugging.

---

## Types of Errors Displayed in the Errors Tab

The Errors Tab in the **Numaflow UI** captures and displays various types of errors that may occur during the execution of user-defined functions (UDFs). Below are the common categories of errors you can expect:

### 1. Exceptions in User Code

Any exception that occurs within the user-defined function (UDF) code will be captured and displayed in the Errors Tab. These errors typically indicate issues in the logic or implementation of the UDF.

### 2. Partial Responses

Errors may occur when the user code sends partial or incorrect responses. For example:

- Returning `null` instead of `Message.toDrop` in a UDSink when a message needs to be dropped.

### 3. Critical Errors Persisted by the User

Users can invoke the `PersistCriticalError` utility function to log critical errors in the `emptyDir` volume. This is useful for capturing severe issues that require immediate attention. Below are links to the implementations in different SDKs:

- [Go Implementation](https://github.com/numaproj/numaflow-go/blob/47460a1854b8f58a0e918056ef4d169949193ebe/pkg/errors/errors.go)
- [Java Implementation](https://github.com/numaproj/numaflow-java/blob/2aeaafff1c6dec7fd66e018b142ef6fe4ffcf0a9/src/main/java/io/numaproj/numaflow/errors/PersistCriticalError.java)
- [Python Implementation](https://github.com/numaproj/numaflow-python/blob/6ccd49ef09ad11eac9b0bb9a6b0f5517d858bea4/pynumaflow/errors/errors.py)
111 changes: 111 additions & 0 deletions docs/user-guide/UI/logs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# Logs View

The **Logs View** allows users to inspect the logs of a specific container within a pod of a vertex. This guide will walk you through the Logs tab and its various features.

---

## Navigating to the Logs Tab

**Select a Pod**

Navigate to the `Pods View` tab after selecting the vertex and select a pod by name from the `Select a Pod by Name` dropdown.

![Select Pod](../../assets/logs/select-pod.png)

**Select a Container**

Choose a container from the `Select a Container` section.

![Select Container](../../assets/logs/select-container.png)

**View Logs**

Open the **Logs Tab** on the right to view the container logs.

![Logs View](../../assets/logs/logs-tab.png)

---

## Features

### 1. Previous Terminated Container Logs

- Enable the checkbox to view logs from previously terminated containers.
- This is particularly useful for debugging issues.

---

### 2. Search Logs

- Filter logs by typing keywords in the **Search Logs** box.

![Search Logs](../../assets/logs/logs-search.png)

---

### 3. Negate Search

- Enable the **Negate Search** option to exclude logs matching the search keywords from the view.

![Negate Search](../../assets/logs/logs-negate-search.png)

---

### 4. Wrap Lines

- Use the **Wrap Lines** feature to avoid horizontal scrolling for long log lines, improving readability.

![Wrap Lines](../../assets/logs/logs-wrap.png)

---

### 5. Pause Logs

- Pause the log stream to inspect logs when there is a high volume of data.

![Pause Logs](../../assets/logs/logs-pause.png)

---

### 6. Dark Mode

- Toggle between **Dark Mode** and **Light Mode** for better visibility based on your preference.

![Dark Mode](../../assets/logs/logs-dark-mode.png)

---

### 7. Ascending/Descending Order

- Switch between ascending and descending order of log timestamps for easier navigation.

![Ascending Order](../../assets/logs/logs-ascending.png)

---

### 8. Download Logs

- Download the last 1000 logs for offline analysis.

---

### 9. Add/Remove Timestamps

- Toggle timestamps in the logs based on your requirements.

![Add Timestamps](../../assets/logs/logs-add-timestamps.png)

---

### 10. Level-Based Filtering

- Filter logs by log levels such as:

- **Info**
- **Error**
- **Warn**
- **Debug**

![Error Filter](../../assets/logs/logs-filter-error.png)

---
Loading
Loading