Skip to content

SubList function documentation #9718

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions _data-prepper/pipelines/functions.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,6 @@ OpenSearch Data Prepper offers a range of built-in functions that can be used wi
- [`hasTags()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/has-tags/)
- [`join()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/join/)
- [`length()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/length/)
- [`subList()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/sublist/)
- [`startsWith()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/startswith/)

79 changes: 79 additions & 0 deletions _data-prepper/pipelines/sublist.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
---
layout: default
title: subList()
parent: Functions
grand_parent: Pipelines
nav_order: 35
---

# subList(<key>, <start_index, inclusive>, <end_index, exclusive>)

Check failure on line 9 in _data-prepper/pipelines/sublist.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.HeadingCapitalization] 'subList(, <, inclusive>, <, exclusive>)' is a heading and should be in sentence case. Raw Output: {"message": "[OpenSearch.HeadingCapitalization] 'subList(, \u003c, inclusive\u003e, \u003c, exclusive\u003e)' is a heading and should be in sentence case.", "location": {"path": "_data-prepper/pipelines/sublist.md", "range": {"start": {"line": 9, "column": 1}}}, "severity": "ERROR"}

The `subList()` function extracts a sublist from a list field in an event. It takes the following arguments:

Check failure on line 11 in _data-prepper/pipelines/sublist.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.Spelling] Error: sublist. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks. Raw Output: {"message": "[OpenSearch.Spelling] Error: sublist. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/sublist.md", "range": {"start": {"line": 11, "column": 37}}}, "severity": "ERROR"}

- A JSON pointer to a list field in the event
- A start index (inclusive)
- An end index (exclusive)

The function returns the portion of the list between the specified start and end indexes. If the end index is `-1`, the function extracts elements from the start index to the end of the list.


## Examples

The following examples show how the `sublist()` function works.

### add_entries processor

Check failure on line 24 in _data-prepper/pipelines/sublist.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.HeadingCapitalization] 'processor' is a heading and should be in sentence case. Raw Output: {"message": "[OpenSearch.HeadingCapitalization] 'processor' is a heading and should be in sentence case.", "location": {"path": "_data-prepper/pipelines/sublist.md", "range": {"start": {"line": 24, "column": 17}}}, "severity": "ERROR"}

You can use `subList()` in the `add_entries` processor, as shown in the following example.

For example, the function will use the following input to extact a sublist:

Check failure on line 28 in _data-prepper/pipelines/sublist.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.Spelling] Error: sublist. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks. Raw Output: {"message": "[OpenSearch.Spelling] Error: sublist. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/sublist.md", "range": {"start": {"line": 28, "column": 68}}}, "severity": "ERROR"}

Check failure on line 28 in _data-prepper/pipelines/sublist.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.Spelling] Error: extact. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks. Raw Output: {"message": "[OpenSearch.Spelling] Error: extact. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/sublist.md", "range": {"start": {"line": 28, "column": 59}}}, "severity": "ERROR"}

```
input: {"my_list": [ 0, 1, 2, 3, 4, 5, 6]}
```

Then, the following configuration uses the `add_entries` processor to extract a sublist from `my_list`, starting at index 1 and ending before index 4:

Check failure on line 34 in _data-prepper/pipelines/sublist.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.Spelling] Error: sublist. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks. Raw Output: {"message": "[OpenSearch.Spelling] Error: sublist. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/sublist.md", "range": {"start": {"line": 34, "column": 81}}}, "severity": "ERROR"}

```yaml
add_entries:
entries:
- key: "my_list"
value_expression: '/subList(/my_list, 1, 4)'
overwrite_if_key_exists: true
```
{% include copy-curl.html %}

The following output shows the resulting list after extracting elements from index 1 to 3 and overwriting the original list:

```
output: my_list: [1, 2, 3]
```

### Specific ranges

Each of the following examples demonstrates how the `subList()` function extracts a specific range of elements from a list.

The following example extracts elements from index `0` to `2` (excluding index `3`), resulting in the first three elements of the list:

```json
{
"event": {
"/my_list": [0, 1, 2, 3, 4, 5, 6]
},
"expression": "subList(/my_list, 4, -1)",
"expected_output": [4, 5, 6]
}
```
{% include copy-curl.html %}

The following example uses `-1` as the end index, which tells the function to include all elements from index `4` to the end of the list:

```json
{
"event": {
"/my_list": [0, 1, 2, 3, 4, 5, 6]
},
"expression": "subList(/my_list, 4, -1)",
"expected_output": [4, 5, 6]
}
```
{% include copy-curl.html %}
Loading