Skip to content

Commit 3be9e01

Browse files
davemarcokirkrodriguesjunhaoliaoMarco
authored
docs: Explain how to work with auto-generated keys; Restructure to clarify how to specify keys for authoritative fields. (#205)
Co-authored-by: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com> Co-authored-by: Junhao Liao <junhao.liao@yscope.com> Co-authored-by: Marco <david.marcovitch@yscope.com>
1 parent ad8a0b0 commit 3be9e01

File tree

8 files changed

+156
-38
lines changed

8 files changed

+156
-38
lines changed

‎docs/src/user-guide/index.md‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,10 @@ A guide to viewing log files in the log viewer.
1414
:::
1515

1616
:::{grid-item-card}
17-
:link: format-struct-logs-overview
18-
Formatting structured logs
17+
:link: struct-logs/index
18+
Working with structured logs
1919
^^^
20-
A guide to formatting structured (e.g. JSON) logs as plain text.
20+
Guides and reference docs for working with structured logs.
2121
:::
2222

2323
:::{grid-item-card}
@@ -37,12 +37,12 @@ quick-start
3737

3838
:::{toctree}
3939
:hidden:
40-
:caption: Formatting structured logs
40+
:caption: Working with structured logs
4141
:glob:
4242

43-
format-struct-logs-overview
44-
format-struct-logs-syntax
45-
format-struct-logs-formatters
43+
struct-logs/index
44+
struct-logs/format/index
45+
struct-logs/specifying-keys
4646
:::
4747

4848
:::{toctree}

‎docs/src/user-guide/format-struct-logs-formatters.md‎ renamed to ‎docs/src/user-guide/struct-logs/format/formatters.md‎

File renamed without changes.

‎docs/src/user-guide/format-struct-logs-overview.md‎ renamed to ‎docs/src/user-guide/struct-logs/format/index.md‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Overview
1+
# Formatting as plain text
22

33
The log viewer can format structured (e.g. JSON) logs as plain text using a format string. The
44
format string allows you to select which fields to include and how they should be formatted. You can
@@ -33,5 +33,12 @@ The formatted log would appear as:
3333
```
3434

3535
For reference docs, see:
36-
* [Format string syntax](format-struct-logs-syntax)
37-
* [Formatters](format-struct-logs-formatters)
36+
* [Format string syntax](syntax)
37+
* [Formatters](formatters)
38+
39+
:::{toctree}
40+
:hidden:
41+
42+
syntax
43+
formatters
44+
:::

‎docs/src/user-guide/format-struct-logs-syntax.md‎ renamed to ‎docs/src/user-guide/struct-logs/format/syntax.md‎

Lines changed: 55 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -16,22 +16,14 @@ backslash:
1616
Each field placeholder is enclosed in braces (`{` and `}`) and has the following form, consisting of
1717
three components:
1818
```
19-
{<field-name>[:<formatter-name>[:<formatter-options>]]}
19+
{<field-key>[:<formatter-name>[:<formatter-options>]]}
2020
```
2121

22-
### field-name (required)
22+
### field-key (required)
2323
Defines the key of the field whose value should replace the placeholder.
2424

25-
* Nested fields can be specified using periods (`.`) to denote hierarchy.
26-
* E.g., the field `{"a:" {"b": 0}}` may be denoted by `a.b`.
27-
* Field names can contain any character, except the following characters must be escaped with a
28-
backslash:
29-
* `.`
30-
* `$`
31-
* `{`
32-
* `}`
33-
* `:`
34-
* `\`
25+
:::{include} ../key-syntax.md
26+
:::
3527

3628
### formatter-name (optional)
3729
The name of the formatter to apply to the value before inserting it into the string.
@@ -43,6 +35,8 @@ The name of the formatter to apply to the value before inserting it into the str
4335
* `:`
4436
* `\`
4537

38+
For a list of currently supported formatters, see [Formatters](formatters).
39+
4640
### formatter-options (optional)
4741
Defines any options for the formatter denoted by `formatter-name`.
4842

@@ -63,28 +57,31 @@ Every format string contains an implicit trailing newline so that each formatted
6357
a newline.
6458

6559
## Examples
66-
Consider the following log event:
67-
```
60+
61+
### Formatting JSON logs events
62+
63+
Consider the following JSON log event:
64+
```json
6865
{
69-
"@timestamp": 1427153388942,
66+
"ts": 1427153388942,
7067
"level": "INFO",
7168
"thread": 0,
7269
"latency": {
7370
"msecs": 56400,
74-
"secs": 56.4,
71+
"secs": 56.4
7572
},
76-
"an.odd.key{name}": "org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties"
73+
"@an.odd.key{name}": "org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties"
7774
}
7875
```
7976

80-
We can format this using the following YScope format string:
77+
We can format this using the following format string:
8178

8279
```
83-
{@timestamp:timestamp:YYYY-MM-DD HH\:mm\:ss.SSS} {level} \{{thread}\} latency={latency.secs:round} {an\.odd\.key\{name\}}
80+
{ts:timestamp:YYYY-MM-DD HH\:mm\:ss.SSS} {level} \{{thread}\} latency={latency.secs:round} {\@an\.odd\.key\{name\}}
8481
```
8582

86-
* In the first placeholder, we have the field name `@timestamp`, a formatter called `timestamp`, and
87-
the formatter's options which are a date format string.
83+
* In the first placeholder, we have the field key `ts`, a formatter called `timestamp`, and the
84+
formatter's options which are a date format string.
8885
* The second and third placeholders simply stringify the values of the given fields.
8986
* The fourth placeholder uses the `round` formatter to round a nested field's value; this
9087
placeholder doesn't specify any formatter options, so the defaults will be used.
@@ -95,4 +92,40 @@ The formatted string will be:
9592
2015-03-23 19:29:48.942 INFO {0} latency=56 org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
9693
```
9794

98-
For a list of currently supported formatters, see [Formatters](format-struct-logs-formatters).
95+
### Formatting kv-pair IR log events
96+
97+
Consider the following kv-pair IR log event:
98+
99+
:::{note}
100+
In the example below, for simplicity, we render the log event as JSON with the auto-generated
101+
kv-pairs under the `auto-generated` key, and the user-generated kv-pairs under the `user-generated`
102+
key, but these keys don't exist in the log event.
103+
:::
104+
105+
```json
106+
{
107+
"auto-generated": {
108+
"ts": 1741371422000
109+
},
110+
"user-generated": {
111+
"message": "Callback registered to fire in 5 seconds:",
112+
"ts": 1741371427000
113+
}
114+
}
115+
```
116+
117+
We can format this using the following format string:
118+
119+
```
120+
{@ts:timestamp} {message} {ts:timestamp}
121+
```
122+
123+
* In the first placeholder, we have the auto-generated field key `@ts`. The `@` prefix specifies
124+
that the field is from the auto-generated namespace.
125+
* The second and third placeholders refer to the `message` and `ts` fields in the user-generated
126+
namespace.
127+
128+
The formatted string will be:
129+
```
130+
2025-03-07T18:17:02Z Callback registered to fire in 5 seconds: 2025-03-07T18:17:07Z
131+
```

‎docs/src/user-guide/struct-logs/index.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Overview
2+
3+
The guides below describe how to configure the log viewer to work with structured logs.
4+
5+
::::{grid} 1 1 2 2
6+
:gutter: 2
7+
8+
:::{grid-item-card}
9+
:link: format/index
10+
Formatting as plain text
11+
^^^
12+
How to format structured logs as plain text, using a format string.
13+
:::
14+
15+
:::{grid-item-card}
16+
:link: specifying-keys
17+
Specifying keys
18+
^^^
19+
How to specify keys when configuring the log viewer.
20+
:::
21+
::::

‎docs/src/user-guide/struct-logs/key-syntax.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
* Nested keys can be specified using periods (`.`) to denote hierarchy.
2+
* E.g., the field `{"a:" {"b": 0}}` may be denoted by `a.b`.
3+
* Auto-generated keys in a [Key-Value Pair IR Stream][kv-pair-ir] can be specified by using `@` as
4+
a prefix.
5+
* E.g., the auto-generated key `ts` would be specified as `@ts`.
6+
* Keys can contain any character, except the following characters must be escaped with a backslash:
7+
* `.`
8+
* `@`
9+
* `{`
10+
* `}`
11+
* `:`
12+
* `\`
13+
14+
[kv-pair-ir]: https://docs.yscope.com/clp/main/dev-guide/design-kv-ir-streams/auto-gen-kv-pairs.html

‎docs/src/user-guide/struct-logs/specifying-keys.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Specifying keys
2+
3+
Viewing structured logs requires specifying keys for:
4+
* The format string
5+
* Authoritative fields such as the log level and timestamp
6+
7+
Both options are found the settings dialog ({fas}`gear`).
8+
9+
## Syntax
10+
11+
:::{include} key-syntax.md
12+
:::

‎src/components/CentralContainer/Sidebar/SidebarTabs/SettingsTabPanel/index.tsx‎

Lines changed: 37 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -48,18 +48,19 @@ const getConfigFormFields = () => [
4848
{
4949
helperText: (
5050
<span>
51-
[JSON] Format string for formatting a JSON log event as plain text. See the
51+
[Structured] Format string for formatting a structured log event as plain text.
52+
Leave blank to display the entire log event. See
5253
{" "}
5354
<Link
54-
href={"https://docs.yscope.com/yscope-log-viewer/main/user-guide/format-struct-logs-overview.html"}
55+
href={"https://docs.yscope.com/yscope-log-viewer/main/user-guide/struct-logs/format/index.html"}
5556
level={"body-sm"}
5657
rel={"noopener"}
5758
target={"_blank"}
5859
>
59-
format string syntax docs
60+
here
6061
</Link>
6162
{" "}
62-
or leave this blank to display the entire log event.
63+
for syntax.
6364
</span>
6465
),
6566
initialValue: getConfig(CONFIG_KEY.DECODER_OPTIONS).formatString,
@@ -68,14 +69,44 @@ const getConfigFormFields = () => [
6869
type: "text",
6970
},
7071
{
71-
helperText: "[JSON] Key to extract the log level from.",
72+
helperText: (
73+
<span>
74+
[Structured] Key that maps to each log event&apos;s log level. See
75+
{" "}
76+
<Link
77+
href={"https://docs.yscope.com/yscope-log-viewer/main/user-guide/struct-logs/specifying-keys.html#syntax"}
78+
level={"body-sm"}
79+
rel={"noopener"}
80+
target={"_blank"}
81+
>
82+
here
83+
</Link>
84+
{" "}
85+
for syntax.
86+
</span>
87+
),
7288
initialValue: getConfig(CONFIG_KEY.DECODER_OPTIONS).logLevelKey,
7389
key: LOCAL_STORAGE_KEY.DECODER_OPTIONS_LOG_LEVEL_KEY,
7490
label: "Decoder: Log level key",
7591
type: "text",
7692
},
7793
{
78-
helperText: "[JSON] Key to extract the log timestamp from.",
94+
helperText: (
95+
<span>
96+
[Structured] Key that maps to each log event&apos;s timestamp. See
97+
{" "}
98+
<Link
99+
href={"https://docs.yscope.com/yscope-log-viewer/main/user-guide/struct-logs/specifying-keys.html#syntax"}
100+
level={"body-sm"}
101+
rel={"noopener"}
102+
target={"_blank"}
103+
>
104+
here
105+
</Link>
106+
{" "}
107+
for syntax.
108+
</span>
109+
),
79110
initialValue: getConfig(CONFIG_KEY.DECODER_OPTIONS).timestampKey,
80111
key: LOCAL_STORAGE_KEY.DECODER_OPTIONS_TIMESTAMP_KEY,
81112
label: "Decoder: Timestamp key",

0 commit comments

Comments
 (0)