From 41b1384f06f79504098b673436b63acc8e074b36 Mon Sep 17 00:00:00 2001 From: Sai charan raj Gudala Date: Wed, 23 Apr 2025 23:35:34 +0530 Subject: [PATCH 1/2] SubList function documentation * Sublist function documentation Signed-off-by: Sai charan raj Gudala --- _data-prepper/pipelines/functions.md | 3 +- _data-prepper/pipelines/sublist.md | 51 ++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+), 1 deletion(-) create mode 100644 _data-prepper/pipelines/sublist.md diff --git a/_data-prepper/pipelines/functions.md b/_data-prepper/pipelines/functions.md index caed78ac550..45d80f90bfa 100644 --- a/_data-prepper/pipelines/functions.md +++ b/_data-prepper/pipelines/functions.md @@ -15,4 +15,5 @@ OpenSearch Data Prepper offers a range of built-in functions that can be used wi - [`getMetadata()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/get-metadata/) - [`hasTags()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/has-tags/) - [`join()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/join/) -- [`length()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/length/) \ No newline at end of file +- [`length()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/length/) +- [`subList()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/sublist/) \ No newline at end of file diff --git a/_data-prepper/pipelines/sublist.md b/_data-prepper/pipelines/sublist.md new file mode 100644 index 00000000000..d76f1daa802 --- /dev/null +++ b/_data-prepper/pipelines/sublist.md @@ -0,0 +1,51 @@ +--- +layout: default +title: subList() +parent: Functions +grand_parent: Pipelines +nav_order: +--- + +# subList(\, , ) + +The `subList()` function takes first argument as JSON pointer type to a list field in events and takes second and third arguments as start index (inclusive) and end index (exclusive) for the subList, and returns the subList of the passed list. + +An end index value of -1, could signal to extract from start_index to the end of the list. + +This function could then be used in the add_entries processor like this + +``` +input: {"my_list": [ 0, 1, 2, 3, 4, 5, 6]} +``` +```yaml +add_entries: + entries: + - key: "my_list" + value_expression: '/subList(/my_list, 1, 4)' + overwrite_if_key_exists: true +``` +``` +output: my_list: [1, 2, 3] +``` + +#### Example + +```json +{ + "event": { + "/my_list": [0, 1, 2, 3, 4, 5, 6] + + }, + "expression": "subList(/message, "0", "3")", + "expected_output": [0, 1, 2] +} +{ + "event": { + "/my_list": [0, 1, 2, 3, 4, 5, 6] + + }, + "expression": "subList(/message, "4", "-1")", + "expected_output": [4, 5, 6] +} +``` +{% include copy-curl.html %} \ No newline at end of file From e575d090b1bf3c9089ea9fe025210df43bf38dce Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Tue, 3 Jun 2025 10:37:59 -0500 Subject: [PATCH 2/2] Add writer edits. Break up examples. Use function in pipeline. Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _data-prepper/pipelines/sublist.md | 56 ++++++++++++++++++++++-------- 1 file changed, 42 insertions(+), 14 deletions(-) diff --git a/_data-prepper/pipelines/sublist.md b/_data-prepper/pipelines/sublist.md index d76f1daa802..9bcfcc8c92d 100644 --- a/_data-prepper/pipelines/sublist.md +++ b/_data-prepper/pipelines/sublist.md @@ -3,20 +3,36 @@ layout: default title: subList() parent: Functions grand_parent: Pipelines -nav_order: +nav_order: 35 --- -# subList(\, , ) +# subList(, , ) -The `subList()` function takes first argument as JSON pointer type to a list field in events and takes second and third arguments as start index (inclusive) and end index (exclusive) for the subList, and returns the subList of the passed list. +The `subList()` function extracts a sublist from a list field in an event. It takes the following arguments: -An end index value of -1, could signal to extract from start_index to the end of the list. +- A JSON pointer to a list field in the event +- A start index (inclusive) +- An end index (exclusive) -This function could then be used in the add_entries processor like this +The function returns the portion of the list between the specified start and end indexes. If the end index is `-1`, the function extracts elements from the start index to the end of the list. + + +## Examples + +The following examples show how the `sublist()` function works. + +### add_entries processor + +You can use `subList()` in the `add_entries` processor, as shown in the following example. + +For example, the function will use the following input to extact a sublist: ``` input: {"my_list": [ 0, 1, 2, 3, 4, 5, 6]} ``` + +Then, the following configuration uses the `add_entries` processor to extract a sublist from `my_list`, starting at index 1 and ending before index 4: + ```yaml add_entries: entries: @@ -24,28 +40,40 @@ add_entries: value_expression: '/subList(/my_list, 1, 4)' overwrite_if_key_exists: true ``` +{% include copy-curl.html %} + +The following output shows the resulting list after extracting elements from index 1 to 3 and overwriting the original list: + ``` output: my_list: [1, 2, 3] ``` -#### Example +### Specific ranges + +Each of the following examples demonstrates how the `subList()` function extracts a specific range of elements from a list. + +The following example extracts elements from index `0` to `2` (excluding index `3`), resulting in the first three elements of the list: ```json { "event": { - "/my_list": [0, 1, 2, 3, 4, 5, 6] - + "/my_list": [0, 1, 2, 3, 4, 5, 6] }, - "expression": "subList(/message, "0", "3")", - "expected_output": [0, 1, 2] + "expression": "subList(/my_list, 4, -1)", + "expected_output": [4, 5, 6] } +``` +{% include copy-curl.html %} + +The following example uses `-1` as the end index, which tells the function to include all elements from index `4` to the end of the list: + +```json { "event": { - "/my_list": [0, 1, 2, 3, 4, 5, 6] - + "/my_list": [0, 1, 2, 3, 4, 5, 6] }, - "expression": "subList(/message, "4", "-1")", + "expression": "subList(/my_list, 4, -1)", "expected_output": [4, 5, 6] } ``` -{% include copy-curl.html %} \ No newline at end of file +{% include copy-curl.html %}