Skip to content

Commit d264f0e

Browse files
authored
chore: update quick start and UI docs (#2622)
Signed-off-by: adarsh0728 <gooneriitk@gmail.com>
1 parent d419358 commit d264f0e

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+865
-92
lines changed

Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -311,7 +311,7 @@ docs-serve: docs
311311

312312
.PHONY: docs-linkcheck
313313
docs-linkcheck: /usr/local/bin/lychee
314-
lychee --exclude-path=CHANGELOG.md *.md --include "https://github.com/numaproj/*" $(shell find ./test -type f) $(wildcard ./docs/*.md)
314+
lychee --exclude-path=CHANGELOG.md --exclude-path=./docs/APIs.md --exclude "https://localhost:*" --exclude "http://localhost:*" --exclude "http://127.0.0.1*" *.md $(shell find ./test -type f) $(shell find ./docs -name '*.md')
315315

316316
# pre-push checks
317317

USERS.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
Please add your company name and initial use case (optional) below.
44

55
1. [Intuit](https://www.intuit.com/) - Streaming ML inference and to prepare data for ML model training in real-time.
6-
2. [B|Cubed](https://bcubed-corp.com/) - Digital Signal Processing Communication Systems. We receive RF energy and use numaflows to transform the RF to individual bits into intelligible data.
6+
2. [B|Cubed](https://bcubed.com/) - Digital Signal Processing Communication Systems. We receive RF energy and use numaflows to transform the RF to individual bits into intelligible data.
77
3. [Atlan](https://atlan.com/) - Numaflow powers real time notifications, stream processing ecosystem at Atlan.
88
4. [Valegachain Analytics](https://www.valegachain.com/) Numaflow is used to extract, transform, and load cryptocurrency blocks and mempool transactions in data lakes, as well as for activity alerts.
99
5. [Lockheed Martin](https://lockheedmartin.com/) Perform ELT processing on high and low volume data streams of sensor data as recieved from IOT type systems.
264 KB
Loading

docs/assets/errors/ui-errors-tab.png

416 KB
Loading
87.3 KB
Loading

docs/assets/logs/logs-ascending.png

265 KB
Loading

docs/assets/logs/logs-dark-mode.png

295 KB
Loading
82.7 KB
Loading
208 KB
Loading

docs/assets/logs/logs-pause.png

257 KB
Loading

docs/assets/logs/logs-search.png

241 KB
Loading

docs/assets/logs/logs-tab.png

296 KB
Loading

docs/assets/logs/logs-wrap.png

226 KB
Loading

docs/assets/logs/select-container.png

28 KB
Loading

docs/assets/logs/select-pod.png

66.5 KB
Loading

docs/assets/metrics/dimensions.png

139 KB
Loading

docs/assets/metrics/metrics-tab.png

271 KB
Loading
132 KB
Loading

docs/assets/metrics/quantiles.png

101 KB
Loading

docs/assets/metrics/query-window.png

88.2 KB
Loading

docs/assets/metrics/time-range.png

181 KB
Loading
-167 KB
Binary file not shown.
-139 KB
Binary file not shown.
108 KB
Loading
Loading
Loading
Loading
769 KB
Loading
38.7 KB
Loading
74.2 KB
Loading
35.2 KB
Loading

docs/assets/ui-overview/spec-view.png

99.1 KB
Loading
111 KB
Loading

docs/operations/metrics/metrics.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -115,7 +115,7 @@ You can follow the [prometheus operator](https://github.com/prometheus-operator/
115115

116116
You can also set up prometheus operator via [helm](https://bitnami.com/stack/prometheus-operator/helm).
117117

118-
### Configure the below Service/Pod Monitors for scraping your pipeline metrics:
118+
### Configure the below Service/Pod Monitors for scraping your pipeline/monovertex metrics:
119119

120120
```yaml
121121
apiVersion: monitoring.coreos.com/v1

docs/quick-start.md

Lines changed: 233 additions & 72 deletions
Large diffs are not rendered by default.

docs/user-guide/FAQ.md

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -4,32 +4,35 @@ Welcome to the Numaflow FAQ! Here you'll find answers to common questions about
44

55
---
66

7-
## 1. How do I get started with Numaflow?
7+
### 1. How do I get started with Numaflow?
88

99
To get started, follow the [Quickstart Guide](../quick-start.md). It provides step-by-step instructions for setting up a basic Numaflow pipeline.
1010

1111
---
1212

13-
## 2. How can I check SDK compatibility with Numaflow versions?
13+
### 2. How can I check SDK compatibility with Numaflow versions?
1414

1515
Refer to the compatibility matrix [here](../user-guide/sdks/compatibility.md) to ensure your SDK version works with your Numaflow deployment.
1616

1717
---
1818

19-
## 3. Where can I learn about the latest releases of Numaflow?
19+
### 3. Where can I learn about the latest releases of Numaflow?
2020

21-
- Visit the [Numaflow Releases page](https://github.com/numaproj/numaflow/releases) for the latest updates.
22-
- Join our [Slack channel](https://join.slack.com/t/numaproj/shared_invite/zt-19svuv47m-YKHhsQ~~KK9mBv1E7pNzfg) to stay up to date with announcements and community discussions.
21+
- Visit the [Numaflow Releases page](https://github.com/numaproj/numaflow/releases) for the latest updates.
22+
- Join our [Slack channel](https://join.slack.com/t/numaproj/shared_invite/zt-19svuv47m-YKHhsQ~~KK9mBv1E7pNzfg) to stay up to date with announcements and community discussions.
2323

2424
---
2525

26-
## 4. I see `Server Info File not ready` log in the numa container. What does this mean?
26+
### 4. I see `Server Info File not ready` log in the numa container. What does this mean?
2727

2828
This message indicates that the server has not started yet. Common reasons include:
2929

30-
- The server was not started correctly in the `udcontainer`.
31-
- The server is still initializing (for example, the application code is downloading cache files or performing setup tasks).
30+
- The server was not started correctly in the `udcontainer`.
31+
- The server is still initializing (for example, the application code is downloading cache files or performing setup tasks).
3232

33-
---
33+
### 5. What is `partitions` in Numaflow?
34+
35+
The field `partitions` is the number of streams between the vertices. Larger the parition number, higher the TPS that vertex can take. Note that partitions are owned by the vertex reading the data, to create a
36+
multi-partitioed edge we need to configure the vertex reading the data to have multiple partitions.
3437

35-
If you have additional questions, please refer to the [documentation](../README.md) or reach out on Slack!
38+
If you have additional questions, please refer to the [documentation](../README.md) or reach out on [Slack](https://join.slack.com/t/numaproj/shared_invite/zt-19svuv47m-YKHhsQ~~KK9mBv1E7pNzfg)!

docs/user-guide/UI/errors.md

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# Errors
2+
3+
The **Numaflow UI** provides an **Errors Tab** that lists the recent errors (up to 10 per container) that may have occurred in the user-defined function (UDF) code. This feature helps users quickly identify and debug application errors.
4+
5+
In this guide, we will simulate a panic in a transformer and demonstrate how the errors are displayed in the Errors Tab for a monovertex.
6+
7+
> **Note**: The Errors Tab and its related functionality are currently available only for MonoVertex. Support for Pipelines is planned for future releases.
8+
9+
---
10+
11+
## High-Level Architecture
12+
13+
![High-Level Architecture](../../assets/errors/errors-architecture.png)
14+
15+
---
16+
17+
## Simulating an Error in the Transformer
18+
19+
To simulate an error, we will introduce a panic in the Go SDK for the transformer by modifying the `Transform` function. You can find the relevant code [here](https://github.com/numaproj/numaflow-go/blob/main/pkg/sourcetransformer/examples/assign_event_time/main.go).
20+
21+
Below is the modified code snippet:
22+
23+
```go
24+
// AssignEventTime is a source transformer that assigns event time to the message.
25+
type AssignEventTime struct {
26+
counter uint64
27+
}
28+
29+
func (a *AssignEventTime) Transform(ctx context.Context, keys []string, d sourcetransformer.Datum) sourcetransformer.Messages {
30+
newCount := atomic.AddUint64(&a.counter, 1)
31+
// Trigger a panic if the counter is a multiple of 5
32+
if newCount%5 == 0 {
33+
panic("Counter reached a multiple of 5")
34+
}
35+
// Update the message's event time to the current time
36+
eventTime := time.Now()
37+
// Log the action for testing purposes
38+
log.Printf("AssignEventTime: Assigning event time %v to message %s", eventTime, string(d.Value()))
39+
return sourcetransformer.MessagesBuilder().Append(sourcetransformer.NewMessage(d.Value(), eventTime).WithKeys(keys))
40+
}
41+
```
42+
43+
In this example, the code triggers a panic whenever the counter reaches a multiple of 5. This simulates an error scenario that will be captured and displayed in the Errors Tab.
44+
45+
---
46+
47+
## The UI Errors Tab
48+
49+
![Errors Tab](../../assets/errors/ui-errors-tab.png)
50+
51+
The Errors Tab provides the following features:
52+
53+
- **Error Count**: Displays the total number of errors across all pods and containers at the top level.
54+
- **Pod and Container Filters**: Allows users to filter errors by selecting a specific pod or container for more focused debugging.
55+
- **Tabulated Errors**: Errors are displayed in a tabular format with the following columns:
56+
57+
- `Pod Name`
58+
- `Container`
59+
- `Message`
60+
- `Last Occurred`
61+
62+
- **Details**: Expanding an error entry reveals a `Details` section that includes the **stack trace** of the error, providing deeper insights for debugging.
63+
64+
---
65+
66+
## Types of Errors Displayed in the Errors Tab
67+
68+
The Errors Tab in the **Numaflow UI** captures and displays various types of errors that may occur during the execution of user-defined functions (UDFs). Below are the common categories of errors you can expect:
69+
70+
### 1. Exceptions in User Code
71+
72+
Any exception that occurs within the user-defined function (UDF) code will be captured and displayed in the Errors Tab. These errors typically indicate issues in the logic or implementation of the UDF.
73+
74+
### 2. Partial Responses
75+
76+
Errors may occur when the user code sends partial or incorrect responses. For example:
77+
78+
- Returning `null` instead of `Message.toDrop` in a UDSink when a message needs to be dropped.
79+
80+
### 3. Critical Errors Persisted by the User
81+
82+
Users can invoke the `PersistCriticalError` utility function to log critical errors in the `emptyDir` volume. This is useful for capturing severe issues that require immediate attention. Below are links to the implementations in different SDKs:
83+
84+
- [Go Implementation](https://github.com/numaproj/numaflow-go/blob/47460a1854b8f58a0e918056ef4d169949193ebe/pkg/errors/errors.go)
85+
- [Java Implementation](https://github.com/numaproj/numaflow-java/blob/2aeaafff1c6dec7fd66e018b142ef6fe4ffcf0a9/src/main/java/io/numaproj/numaflow/errors/PersistCriticalError.java)
86+
- [Python Implementation](https://github.com/numaproj/numaflow-python/blob/6ccd49ef09ad11eac9b0bb9a6b0f5517d858bea4/pynumaflow/errors/errors.py)

docs/user-guide/UI/logs.md

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Logs View
2+
3+
The **Logs View** allows users to inspect the logs of a specific container within a pod of a vertex. This guide will walk you through the Logs tab and its various features.
4+
5+
---
6+
7+
## Navigating to the Logs Tab
8+
9+
**Select a Pod**
10+
11+
Navigate to the `Pods View` tab after selecting the vertex and select a pod by name from the `Select a Pod by Name` dropdown.
12+
13+
![Select Pod](../../assets/logs/select-pod.png)
14+
15+
**Select a Container**
16+
17+
Choose a container from the `Select a Container` section.
18+
19+
![Select Container](../../assets/logs/select-container.png)
20+
21+
**View Logs**
22+
23+
Open the **Logs Tab** on the right to view the container logs.
24+
25+
![Logs View](../../assets/logs/logs-tab.png)
26+
27+
---
28+
29+
## Features
30+
31+
### 1. Previous Terminated Container Logs
32+
33+
- Enable the checkbox to view logs from previously terminated containers.
34+
- This is particularly useful for debugging issues.
35+
36+
---
37+
38+
### 2. Search Logs
39+
40+
- Filter logs by typing keywords in the **Search Logs** box.
41+
42+
![Search Logs](../../assets/logs/logs-search.png)
43+
44+
---
45+
46+
### 3. Negate Search
47+
48+
- Enable the **Negate Search** option to exclude logs matching the search keywords from the view.
49+
50+
![Negate Search](../../assets/logs/logs-negate-search.png)
51+
52+
---
53+
54+
### 4. Wrap Lines
55+
56+
- Use the **Wrap Lines** feature to avoid horizontal scrolling for long log lines, improving readability.
57+
58+
![Wrap Lines](../../assets/logs/logs-wrap.png)
59+
60+
---
61+
62+
### 5. Pause Logs
63+
64+
- Pause the log stream to inspect logs when there is a high volume of data.
65+
66+
![Pause Logs](../../assets/logs/logs-pause.png)
67+
68+
---
69+
70+
### 6. Dark Mode
71+
72+
- Toggle between **Dark Mode** and **Light Mode** for better visibility based on your preference.
73+
74+
![Dark Mode](../../assets/logs/logs-dark-mode.png)
75+
76+
---
77+
78+
### 7. Ascending/Descending Order
79+
80+
- Switch between ascending and descending order of log timestamps for easier navigation.
81+
82+
![Ascending Order](../../assets/logs/logs-ascending.png)
83+
84+
---
85+
86+
### 8. Download Logs
87+
88+
- Download the last 1000 logs for offline analysis.
89+
90+
---
91+
92+
### 9. Add/Remove Timestamps
93+
94+
- Toggle timestamps in the logs based on your requirements.
95+
96+
![Add Timestamps](../../assets/logs/logs-add-timestamps.png)
97+
98+
---
99+
100+
### 10. Level-Based Filtering
101+
102+
- Filter logs by log levels such as:
103+
- **Info**
104+
- **Error**
105+
- **Warn**
106+
- **Debug**
107+
108+
![Error Filter](../../assets/logs/logs-filter-error.png)
109+
110+
---

0 commit comments

Comments
 (0)