You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The `add_entries` processor adds entries to an event.
12
12
13
-
###Configuration
13
+
## Configuration
14
14
15
15
You can configure the `add_entries` processor with the following options.
16
16
17
17
| Option | Required | Description |
18
18
| :--- | :--- | :--- |
19
19
|`entries`| Yes | A list of entries to add to an event. |
20
-
|`key`| Yes | The key of the new entry to be added. Some examples of keys include `my_key`, `myKey`, and `object/sub_Key`. |
21
-
|`metadata_key`| Yes | The key for the new metadata attribute. The argument must be a literal string key and not a JSON Pointer. Either one string key or `metadata_key` is required. |
20
+
|`key`| No | The key of the new entry to be added. Some examples of keys include `my_key`, `myKey`, and `object/sub_Key`. The key can also be a format expression, for example, `${/key1}` to use the value of field `key1` as the key. |
21
+
|`metadata_key`| No | The key for the new metadata attribute. The argument must be a literal string key and not a JSON Pointer. Either one string key or `metadata_key` is required. |
22
+
|`value`| No | The value of the new entry to be added, which can be used with any of the following data types: strings, Booleans, numbers, null, nested objects, and arrays. |
22
23
|`format`| No | A format string to use as the value of the new entry, for example, `${key1}-${key2}`, where `key1` and `key2` are existing keys in the event. Required if neither `value` nor `value_expression` is specified. |
23
24
|`value_expression`| No | An expression string to use as the value of the new entry. For example, `/key` is an existing key in the event with a type of either a number, a string, or a Boolean. Expressions can also contain functions returning number/string/integer. For example, `length(/key)` will return the length of the key in the event when the key is a string. For more information about keys, see [Expression syntax](https://opensearch.org/docs/latest/data-prepper/pipelines/expression-syntax/). |
24
25
|`add_when`| No | A [conditional expression](https://opensearch.org/docs/latest/data-prepper/pipelines/expression-syntax/), such as `/some-key == "test"'`, that will be evaluated to determine whether the processor will be run on the event. |
25
-
|`value`| Yes | The value of the new entry to be added. You can use the following data types: strings, Booleans, numbers, null, nested objects, and arrays. |
26
26
|`overwrite_if_key_exists`| No | When set to `true`, the existing value is overwritten if `key` already exists in the event. The default value is `false`. |
27
+
|`append_if_key_exists`| No | When set to `true`, the existing value will be appended if a `key` already exists in the event. An array will be created if the existing value is not an array. Default is `false`. |
27
28
28
-
### Usage
29
29
30
-
To get started, create the following `pipeline.yaml` file:
30
+
## Usage
31
+
32
+
The following examples show how the `add_entries` processor can be used in different cases.
33
+
34
+
### Example: Add entries with simple values
35
+
36
+
The following example shows you how to configure the processor to add entries with simple values:
31
37
32
38
```yaml
33
-
pipeline:
34
-
source:
35
-
...
36
-
....
39
+
...
37
40
processor:
38
41
- add_entries:
39
42
entries:
40
-
- key: "newMessage"
41
-
value: 3
42
-
overwrite_if_key_exists: true
43
-
- metadata_key: myMetadataKey
44
-
value_expression: 'length("newMessage")'
45
-
add_when: '/some_key == "test"'
46
-
sink:
43
+
- key: "name"
44
+
value: "John"
45
+
- key: "age"
46
+
value: 20
47
+
...
47
48
```
48
49
{% include copy.html %}
49
50
51
+
When the input event contains the following data:
52
+
53
+
```json
54
+
{"message": "hello"}
55
+
```
50
56
51
-
For example, when your source contains the following event record:
57
+
The processed event will contain the following data:
58
+
59
+
```json
60
+
{"message": "hello", "name": "John", "age": 20}
61
+
```
62
+
63
+
### Example: Add entries using format strings
64
+
65
+
The following example shows you how to configure the processor to add entries with values from other fields:
66
+
67
+
```yaml
68
+
...
69
+
processor:
70
+
- add_entries:
71
+
entries:
72
+
- key: "date"
73
+
format: "${month}-${day}"
74
+
...
75
+
```
76
+
{% include copy.html %}
77
+
78
+
When the input event contains the following data:
79
+
80
+
```json
81
+
{"month": "Dec", "day": 1}
82
+
```
83
+
84
+
The processed event will contain the following data:
85
+
86
+
```json
87
+
{"month": "Dec", "day": 1, "date": "Dec-1"}
88
+
```
89
+
90
+
### Example: Add entries using value expressions
91
+
92
+
The following example shows you how to configure the processor to use the `value_expression` option:
93
+
94
+
```yaml
95
+
...
96
+
processor:
97
+
- add_entries:
98
+
entries:
99
+
- key: "length"
100
+
value_expression: "length(/message)"
101
+
...
102
+
```
103
+
{% include copy.html %}
104
+
105
+
When the input event contains the following data:
52
106
53
107
```json
54
108
{"message": "hello"}
55
109
```
56
110
57
-
And then you run the `add_entries` processor using the example pipeline, it adds a new entry, `{"newMessage": 3}`, to the existing event, `{"message": "hello"}`, so that the new event contains two entries in the final output:
111
+
The processed event will contain the following data:
112
+
113
+
```json
114
+
{"message": "hello", "length": 5}
115
+
```
116
+
117
+
### Example: Add metadata
118
+
119
+
The following example shows you how to configure the processor to add metadata to events:
120
+
121
+
```yaml
122
+
...
123
+
processor:
124
+
- add_entries:
125
+
entries:
126
+
- metadata_key: "length"
127
+
value_expression: "length(/message)"
128
+
...
129
+
```
130
+
{% include copy.html %}
131
+
132
+
When the input event contains the following data:
58
133
59
134
```json
60
-
{"message": "hello", "newMessage": 3}
135
+
{"message": "hello"}
61
136
```
62
137
63
-
If `newMessage` already exists, its existing value is overwritten with a value of `3`.
138
+
The processed event will have the same data, with the metadata, `{"length": 5}`, attached. You can subsequently use expressions like `getMetadata("length")` in the pipeline. For more information, see the [`getMetadata` function](https://opensearch.org/docs/latest/data-prepper/pipelines/expression-syntax/#getmetadata) documentation.
139
+
140
+
141
+
### Example: Add a dynamic key
64
142
143
+
The following example shows you how to configure the processor to add metadata to events using a dynamic key:
144
+
145
+
```yaml
146
+
...
147
+
processor:
148
+
- add_entries:
149
+
entries:
150
+
- key: "${/param_name}"
151
+
value_expression: "/param_value"
152
+
...
153
+
```
154
+
{% include copy.html %}
155
+
156
+
When the input event contains the following data:
157
+
158
+
```json
159
+
{"param_name": "cpu", "param_value": 50}
160
+
```
161
+
162
+
The processed event will contain the following data:
0 commit comments