From b9cc326e580b6a36fd185ae1417ddd3e2bec90bd Mon Sep 17 00:00:00 2001 From: Oleg Mukhin Date: Sat, 19 Jul 2025 15:50:40 +0100 Subject: [PATCH 01/21] pipeline: filters: lookup added new filter New documentation page outlines description, example configuration and CSV handling for the new LookUp filter. Signed-off-by: Oleg Mukhin --- pipeline/filters/lookup.md | 134 +++++++++++++++++++++++++++++++++++++ 1 file changed, 134 insertions(+) create mode 100644 pipeline/filters/lookup.md diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md new file mode 100644 index 000000000..298c57187 --- /dev/null +++ b/pipeline/filters/lookup.md @@ -0,0 +1,134 @@ +# Lookup + +The Lookup plugin looks up a key value from a record in a specified CSV file and, if a match is found, adds the corresponding value from the CSV as a new key-value pair to the record. + +## Configuration parameters + +The plugin supports the following configuration parameters + +| Key | Description | Default | +| :-- | :---------- | :------ | +| `file` | The CSV file that Fluent Bit will use as a lookup table. The file should contain two columns (key and value), with the first row as an optional header that is skipped. Supports quoted fields and escaped quotes. | _none_ | +| `lookup_key` | The specific key in the input record to look up in the CSV file's first column. Supports [record accessor](../../administration/configuring-fluent-bit/record-accessor). | _none_ | +| `result_key` | The name of the key to add to the output record with the matched value from the CSV file's second column if a match is found. | _none_ | +| `ignore_case` | Ignore case when matching the lookup key against the CSV keys. | `false` | + +## Example configuration + +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +parsers: + - name: json + format: json + +pipeline: + inputs: + - name: tail + tag: test + path: devices.log + read_from_head: true + parser: json + + filters: + - name: lookup + match: test + file: device-bu.csv + lookup_key: $hostname + result_key: business_line + ignore_case: true + + outputs: + - name: stdout + match: test +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[PARSER] + Name json + Format json + +[INPUT] + Name tail + Tag test + Path devices.log + Read_from_head On + Parser json + +[FILTER] + Name lookup + Match test + File device-bu.csv + Lookup_key $hostname + Result_key business_line + Ignore_case On + +[OUTPUT] + Name stdout + Match test +``` + +{% endtab %} +{% endtabs %} + +The following configuration reads log records from `devices.log` that includes the following values for device hostnames: + +```text +{"hostname": "server-prod-001"} +{"hostname": "Server-Prod-001"} +{"hostname": "db-test-abc"} +{"hostname": 123} +{"hostname": true} +{"hostname": " host with space "} +{"hostname": "quoted \"host\""} +{"hostname": "unknown-host"} +{} +{"hostname": [1,2,3]} +{"hostname": {"sub": "val"}} +{"hostname": " "} +``` + +It uses the value of the `hostname` field (which has been set as the `lookup_key`) to find matching values in column 1 of the (`device-bu.csv`) CSV file. + +```text +hostname,business_line +server-prod-001,Finance +db-test-abc,Engineering +db-test-abc,Marketing +web-frontend-xyz,Marketing +app-backend-123,Operations +"legacy-system true","Legacy IT" +" host with space ","Infrastructure" +"quoted ""host""", "R&D" +no-match-host,Should Not Appear +``` + +Where a match is found the filter adds new key (name of which is set by the `result_key` input) with the value from the second column of the CSV file of the matched row. + +For above configuration the following output can be expected (when matching case is ignored as `ignore_case` is set to true): + +```text +{"hostname"=>"server-prod-001", "business_line"=>"Finance"} +{"hostname"=>"Server-Prod-001", "business_line"=>"Finance"} +{"hostname"=>"db-test-abc", "business_line"=>"Marketing"} +{"hostname"=>123} +{"hostname"=>true} +{"hostname"=>" host with space ", "business_line"=>"Infrastructure"} +{"hostname"=>"quoted "host"", "business_line"=>"R&D"} +{"hostname"=>"unknown-host"} +{} +{"hostname"=>[1, 2, 3]} +{"hostname"=>{"sub"=>"val"}} +``` + +## CSV import + +The CSV is used to create an in-memory key value lookup table. Column 1 of the CSV is always used as key, while column 2 is assumed to be the value. All other columns in the CSV are ignored. + +This filter is intended for static datasets. CSV is loaded once when Fluent Bit starts and is not reloaded. + +Multiline values in CSV file are not currently supported. From 6fb3490496011a655a48543f41b964cc1ed00652 Mon Sep 17 00:00:00 2001 From: Alexa Kreizinger Date: Mon, 21 Jul 2025 13:49:49 -0700 Subject: [PATCH 02/21] SUMMARY: add link to new page Signed-off-by: Alexa Kreizinger --- SUMMARY.md | 1 + 1 file changed, 1 insertion(+) diff --git a/SUMMARY.md b/SUMMARY.md index 2d6f3b6fd..ecc870909 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -160,6 +160,7 @@ * [Grep](pipeline/filters/grep.md) * [Kubernetes](pipeline/filters/kubernetes.md) * [Log to Metrics](pipeline/filters/log_to_metrics.md) + * [Lookup](pipeline/filters/lookup.md) * [Lua](pipeline/filters/lua.md) * [Parser](pipeline/filters/parser.md) * [Record Modifier](pipeline/filters/record-modifier.md) From 805c61914d4746bb6bd47a6a3896e68ba8e2a734 Mon Sep 17 00:00:00 2001 From: Oleg Mukhin Date: Mon, 21 Jul 2025 22:19:15 +0100 Subject: [PATCH 03/21] Update pipeline/filters/lookup.md Co-authored-by: Alexa Kreizinger Signed-off-by: Oleg Mukhin --- pipeline/filters/lookup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md index 298c57187..215d5f737 100644 --- a/pipeline/filters/lookup.md +++ b/pipeline/filters/lookup.md @@ -4,7 +4,7 @@ The Lookup plugin looks up a key value from a record in a specified CSV file and ## Configuration parameters -The plugin supports the following configuration parameters +The plugin supports the following configuration parameters: | Key | Description | Default | | :-- | :---------- | :------ | From ed65192808649d137fc7c52608c0192262ebd268 Mon Sep 17 00:00:00 2001 From: Oleg Mukhin Date: Mon, 21 Jul 2025 22:21:04 +0100 Subject: [PATCH 04/21] Update pipeline/filters/lookup.md Co-authored-by: Alexa Kreizinger Signed-off-by: Oleg Mukhin --- pipeline/filters/lookup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md index 215d5f737..c10589791 100644 --- a/pipeline/filters/lookup.md +++ b/pipeline/filters/lookup.md @@ -9,7 +9,7 @@ The plugin supports the following configuration parameters: | Key | Description | Default | | :-- | :---------- | :------ | | `file` | The CSV file that Fluent Bit will use as a lookup table. The file should contain two columns (key and value), with the first row as an optional header that is skipped. Supports quoted fields and escaped quotes. | _none_ | -| `lookup_key` | The specific key in the input record to look up in the CSV file's first column. Supports [record accessor](../../administration/configuring-fluent-bit/record-accessor). | _none_ | +| `lookup_key` | Specifies the key to search for in the CSV file's first column. Supports [record accessor](../administration/configuring-fluent-bit/classic-mode/record-accessor) syntax. | _none_ | | `result_key` | The name of the key to add to the output record with the matched value from the CSV file's second column if a match is found. | _none_ | | `ignore_case` | Ignore case when matching the lookup key against the CSV keys. | `false` | From 1e14ae8d1d8daaa1a07904ffe54a1db0fe6161b2 Mon Sep 17 00:00:00 2001 From: Oleg Mukhin Date: Mon, 21 Jul 2025 22:21:51 +0100 Subject: [PATCH 05/21] Update pipeline/filters/lookup.md Co-authored-by: Alexa Kreizinger Signed-off-by: Oleg Mukhin --- pipeline/filters/lookup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md index c10589791..7548ad233 100644 --- a/pipeline/filters/lookup.md +++ b/pipeline/filters/lookup.md @@ -75,7 +75,7 @@ pipeline: {% endtab %} {% endtabs %} -The following configuration reads log records from `devices.log` that includes the following values for device hostnames: +The previous configuration reads log records from `devices.log` that includes the following values for device hostnames: ```text {"hostname": "server-prod-001"} From 6cb38bb8aebf4e83fcfb470e4f752de135495fed Mon Sep 17 00:00:00 2001 From: Oleg Mukhin Date: Mon, 21 Jul 2025 22:22:11 +0100 Subject: [PATCH 06/21] Update pipeline/filters/lookup.md Co-authored-by: Alexa Kreizinger Signed-off-by: Oleg Mukhin --- pipeline/filters/lookup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md index 7548ad233..e2111768b 100644 --- a/pipeline/filters/lookup.md +++ b/pipeline/filters/lookup.md @@ -107,7 +107,7 @@ app-backend-123,Operations no-match-host,Should Not Appear ``` -Where a match is found the filter adds new key (name of which is set by the `result_key` input) with the value from the second column of the CSV file of the matched row. +When the filter finds a match, it adds a new key with the name specified by `result_key` and a value from the second column of the CSV file of the row where `lookup_key` was found. For above configuration the following output can be expected (when matching case is ignored as `ignore_case` is set to true): From 0c3163b4b1174ed83098a407c605815eb01e8de1 Mon Sep 17 00:00:00 2001 From: Oleg Mukhin Date: Mon, 21 Jul 2025 22:22:26 +0100 Subject: [PATCH 07/21] Update pipeline/filters/lookup.md Co-authored-by: Alexa Kreizinger Signed-off-by: Oleg Mukhin --- pipeline/filters/lookup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md index e2111768b..274109d1f 100644 --- a/pipeline/filters/lookup.md +++ b/pipeline/filters/lookup.md @@ -109,7 +109,7 @@ no-match-host,Should Not Appear When the filter finds a match, it adds a new key with the name specified by `result_key` and a value from the second column of the CSV file of the row where `lookup_key` was found. -For above configuration the following output can be expected (when matching case is ignored as `ignore_case` is set to true): +This results in output that resembles the following: ```text {"hostname"=>"server-prod-001", "business_line"=>"Finance"} From fcfe91618586967fb2069b50fbf3b1ddf090b99c Mon Sep 17 00:00:00 2001 From: Oleg Mukhin Date: Mon, 21 Jul 2025 22:23:52 +0100 Subject: [PATCH 08/21] Update pipeline/filters/lookup.md Co-authored-by: Alexa Kreizinger Signed-off-by: Oleg Mukhin --- pipeline/filters/lookup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md index 274109d1f..d7a8dd959 100644 --- a/pipeline/filters/lookup.md +++ b/pipeline/filters/lookup.md @@ -10,7 +10,7 @@ The plugin supports the following configuration parameters: | :-- | :---------- | :------ | | `file` | The CSV file that Fluent Bit will use as a lookup table. The file should contain two columns (key and value), with the first row as an optional header that is skipped. Supports quoted fields and escaped quotes. | _none_ | | `lookup_key` | Specifies the key to search for in the CSV file's first column. Supports [record accessor](../administration/configuring-fluent-bit/classic-mode/record-accessor) syntax. | _none_ | -| `result_key` | The name of the key to add to the output record with the matched value from the CSV file's second column if a match is found. | _none_ | +| `result_key` | If `lookup_key` is found, sets the name of the new key to add to the output record. This new key uses the corresponding value from the second column of the CSV file in the same row where `lookup_key` was found. | _none_ | | `ignore_case` | Ignore case when matching the lookup key against the CSV keys. | `false` | ## Example configuration From 90e78ef04c74fb1cf592fb32bbf9a96d8b7c05e1 Mon Sep 17 00:00:00 2001 From: Oleg Mukhin Date: Mon, 21 Jul 2025 22:24:46 +0100 Subject: [PATCH 09/21] Update pipeline/filters/lookup.md Co-authored-by: Alexa Kreizinger Signed-off-by: Oleg Mukhin --- pipeline/filters/lookup.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md index d7a8dd959..fee3b9a54 100644 --- a/pipeline/filters/lookup.md +++ b/pipeline/filters/lookup.md @@ -127,8 +127,9 @@ This results in output that resembles the following: ## CSV import -The CSV is used to create an in-memory key value lookup table. Column 1 of the CSV is always used as key, while column 2 is assumed to be the value. All other columns in the CSV are ignored. -This filter is intended for static datasets. CSV is loaded once when Fluent Bit starts and is not reloaded. +Fluent Bit creates an in-memory key/value lookup table from the CSV file that you provide. The first column of this CSV is always treated as a key, and its second column as a value. Any other columns are ignored. -Multiline values in CSV file are not currently supported. +This filter is intended for static datasets. After Fluent Bit loads the CSV file, it won't reload that file, which means the filter's lookup table won't update to reflect any changes. + +This filter doesn't support multiline values in CSV files. From 47163aefcd698ee2c5f3778217fad8ef339135e4 Mon Sep 17 00:00:00 2001 From: Oleg Mukhin Date: Mon, 21 Jul 2025 22:25:37 +0100 Subject: [PATCH 10/21] Update pipeline/filters/lookup.md Co-authored-by: Alexa Kreizinger Signed-off-by: Oleg Mukhin --- pipeline/filters/lookup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md index fee3b9a54..35911fb90 100644 --- a/pipeline/filters/lookup.md +++ b/pipeline/filters/lookup.md @@ -92,7 +92,7 @@ The previous configuration reads log records from `devices.log` that includes th {"hostname": " "} ``` -It uses the value of the `hostname` field (which has been set as the `lookup_key`) to find matching values in column 1 of the (`device-bu.csv`) CSV file. +It uses the value of the `hostname` field (which was set as the `lookup_key`) to find matching values in the first column of `device-bu.csv`. ```text hostname,business_line From 0f5b54bc31b6e48ffab960bb7078abe903540191 Mon Sep 17 00:00:00 2001 From: Oleg Mukhin Date: Mon, 21 Jul 2025 22:29:01 +0100 Subject: [PATCH 11/21] Update pipeline/filters/lookup.md Co-authored-by: Alexa Kreizinger Signed-off-by: Oleg Mukhin --- pipeline/filters/lookup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md index 35911fb90..ed257fb91 100644 --- a/pipeline/filters/lookup.md +++ b/pipeline/filters/lookup.md @@ -8,7 +8,7 @@ The plugin supports the following configuration parameters: | Key | Description | Default | | :-- | :---------- | :------ | -| `file` | The CSV file that Fluent Bit will use as a lookup table. The file should contain two columns (key and value), with the first row as an optional header that is skipped. Supports quoted fields and escaped quotes. | _none_ | +| `file` | The CSV file that the Lookup filter will use as a lookup table. This file must contain one column of keys and one column of values. The Lookup filter treats the first row as a header and ignores it. Supported quoted fields and escaped quotes. | _none_ | | `lookup_key` | Specifies the key to search for in the CSV file's first column. Supports [record accessor](../administration/configuring-fluent-bit/classic-mode/record-accessor) syntax. | _none_ | | `result_key` | If `lookup_key` is found, sets the name of the new key to add to the output record. This new key uses the corresponding value from the second column of the CSV file in the same row where `lookup_key` was found. | _none_ | | `ignore_case` | Ignore case when matching the lookup key against the CSV keys. | `false` | From 82adfa155f1589a059140463478482dcaa1f149f Mon Sep 17 00:00:00 2001 From: Alexa Kreizinger Date: Mon, 21 Jul 2025 14:33:56 -0700 Subject: [PATCH 12/21] Update pipeline/filters/lookup.md Signed-off-by: Alexa Kreizinger --- pipeline/filters/lookup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md index ed257fb91..c4f543f9b 100644 --- a/pipeline/filters/lookup.md +++ b/pipeline/filters/lookup.md @@ -10,7 +10,7 @@ The plugin supports the following configuration parameters: | :-- | :---------- | :------ | | `file` | The CSV file that the Lookup filter will use as a lookup table. This file must contain one column of keys and one column of values. The Lookup filter treats the first row as a header and ignores it. Supported quoted fields and escaped quotes. | _none_ | | `lookup_key` | Specifies the key to search for in the CSV file's first column. Supports [record accessor](../administration/configuring-fluent-bit/classic-mode/record-accessor) syntax. | _none_ | -| `result_key` | If `lookup_key` is found, sets the name of the new key to add to the output record. This new key uses the corresponding value from the second column of the CSV file in the same row where `lookup_key` was found. | _none_ | +| `result_key` | If `lookup_key` is found, specifies the name of the new key to add to the output record. This new key uses the corresponding value from the second column of the CSV file in the same row where `lookup_key` was found. | _none_ | | `ignore_case` | Ignore case when matching the lookup key against the CSV keys. | `false` | ## Example configuration From 131f1be2f099cc400d81bf9e5784f9883ef77d0a Mon Sep 17 00:00:00 2001 From: Alexa Kreizinger Date: Mon, 21 Jul 2025 15:43:36 -0700 Subject: [PATCH 13/21] Apply suggestions from code review Signed-off-by: Alexa Kreizinger --- pipeline/filters/lookup.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md index c4f543f9b..7acfef591 100644 --- a/pipeline/filters/lookup.md +++ b/pipeline/filters/lookup.md @@ -77,7 +77,7 @@ pipeline: The previous configuration reads log records from `devices.log` that includes the following values for device hostnames: -```text +```shell {"hostname": "server-prod-001"} {"hostname": "Server-Prod-001"} {"hostname": "db-test-abc"} @@ -111,7 +111,7 @@ When the filter finds a match, it adds a new key with the name specified by `res This results in output that resembles the following: -```text +```shell {"hostname"=>"server-prod-001", "business_line"=>"Finance"} {"hostname"=>"Server-Prod-001", "business_line"=>"Finance"} {"hostname"=>"db-test-abc", "business_line"=>"Marketing"} From 58deea080be9e4c93ecebb61f3807133c5875be3 Mon Sep 17 00:00:00 2001 From: Oleg Mukhin Date: Wed, 23 Jul 2025 20:53:31 +0100 Subject: [PATCH 14/21] Update pipeline/filters/lookup.md Co-authored-by: Alexa Kreizinger Signed-off-by: Oleg Mukhin --- pipeline/filters/lookup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md index 7acfef591..3442ebf51 100644 --- a/pipeline/filters/lookup.md +++ b/pipeline/filters/lookup.md @@ -11,7 +11,7 @@ The plugin supports the following configuration parameters: | `file` | The CSV file that the Lookup filter will use as a lookup table. This file must contain one column of keys and one column of values. The Lookup filter treats the first row as a header and ignores it. Supported quoted fields and escaped quotes. | _none_ | | `lookup_key` | Specifies the key to search for in the CSV file's first column. Supports [record accessor](../administration/configuring-fluent-bit/classic-mode/record-accessor) syntax. | _none_ | | `result_key` | If `lookup_key` is found, specifies the name of the new key to add to the output record. This new key uses the corresponding value from the second column of the CSV file in the same row where `lookup_key` was found. | _none_ | -| `ignore_case` | Ignore case when matching the lookup key against the CSV keys. | `false` | +| `ignore_case` | Specifies whether to ignore case when searching for `lookup_key`. If `true`, searches are case-insensitive. If `false`, searches are case-sensitive. Possible values: `true`, `false`. | `false` | ## Example configuration From 15241a3f5cc8a86ffed6390a6d5ded3c8ac2b737 Mon Sep 17 00:00:00 2001 From: Oleg Mukhin Date: Wed, 23 Jul 2025 20:56:12 +0100 Subject: [PATCH 15/21] Update pipeline/filters/lookup.md Co-authored-by: Alexa Kreizinger Signed-off-by: Oleg Mukhin --- pipeline/filters/lookup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md index 3442ebf51..c83471845 100644 --- a/pipeline/filters/lookup.md +++ b/pipeline/filters/lookup.md @@ -1,6 +1,6 @@ # Lookup -The Lookup plugin looks up a key value from a record in a specified CSV file and, if a match is found, adds the corresponding value from the CSV as a new key-value pair to the record. +The _Lookup_ plugin searches for a specified key in a CSV file and, if it finds a match, adds a new key to the output record whose value matches the corresponding CSV value. ## Configuration parameters From d5b3822c15541cbce78f600ae41746e43d93b1cf Mon Sep 17 00:00:00 2001 From: Oleg Mukhin Date: Wed, 23 Jul 2025 20:58:37 +0100 Subject: [PATCH 16/21] Update pipeline/filters/lookup.md Co-authored-by: Alexa Kreizinger Signed-off-by: Oleg Mukhin --- pipeline/filters/lookup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md index c83471845..33a115301 100644 --- a/pipeline/filters/lookup.md +++ b/pipeline/filters/lookup.md @@ -75,7 +75,7 @@ pipeline: {% endtab %} {% endtabs %} -The previous configuration reads log records from `devices.log` that includes the following values for device hostnames: +The previous configuration reads log records from `devices.log` that includes the following values in the `hostname` field: ```shell {"hostname": "server-prod-001"} From 40389b392c94925d78a1951d4b6a84dfcb8a0e20 Mon Sep 17 00:00:00 2001 From: Oleg Mukhin Date: Wed, 23 Jul 2025 20:59:30 +0100 Subject: [PATCH 17/21] Update pipeline/filters/lookup.md Co-authored-by: Alexa Kreizinger Signed-off-by: Oleg Mukhin --- pipeline/filters/lookup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md index 33a115301..346b49bd1 100644 --- a/pipeline/filters/lookup.md +++ b/pipeline/filters/lookup.md @@ -9,7 +9,7 @@ The plugin supports the following configuration parameters: | Key | Description | Default | | :-- | :---------- | :------ | | `file` | The CSV file that the Lookup filter will use as a lookup table. This file must contain one column of keys and one column of values. The Lookup filter treats the first row as a header and ignores it. Supported quoted fields and escaped quotes. | _none_ | -| `lookup_key` | Specifies the key to search for in the CSV file's first column. Supports [record accessor](../administration/configuring-fluent-bit/classic-mode/record-accessor) syntax. | _none_ | +| `lookup_key` | Specifies the record key whose value search for in the CSV file's first column. Supports [record accessor](../administration/configuring-fluent-bit/classic-mode/record-accessor) syntax. | _none_ | | `result_key` | If `lookup_key` is found, specifies the name of the new key to add to the output record. This new key uses the corresponding value from the second column of the CSV file in the same row where `lookup_key` was found. | _none_ | | `ignore_case` | Specifies whether to ignore case when searching for `lookup_key`. If `true`, searches are case-insensitive. If `false`, searches are case-sensitive. Possible values: `true`, `false`. | `false` | From 06f0c2df9bc702950499a5295d1a3246c066ca61 Mon Sep 17 00:00:00 2001 From: Oleg Mukhin Date: Wed, 23 Jul 2025 21:05:42 +0100 Subject: [PATCH 18/21] Update pipeline/filters/lookup.md Co-authored-by: Alexa Kreizinger Signed-off-by: Oleg Mukhin --- pipeline/filters/lookup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md index 346b49bd1..2fc36c554 100644 --- a/pipeline/filters/lookup.md +++ b/pipeline/filters/lookup.md @@ -8,7 +8,7 @@ The plugin supports the following configuration parameters: | Key | Description | Default | | :-- | :---------- | :------ | -| `file` | The CSV file that the Lookup filter will use as a lookup table. This file must contain one column of keys and one column of values. The Lookup filter treats the first row as a header and ignores it. Supported quoted fields and escaped quotes. | _none_ | +| `file` | The CSV file that the Lookup filter will use as a lookup table. This file must contain one column of keys and one column of values. The Lookup filter treats the first row as a header and ignores these values. Supports quoted fields and escaped quotes. | _none_ | | `lookup_key` | Specifies the record key whose value search for in the CSV file's first column. Supports [record accessor](../administration/configuring-fluent-bit/classic-mode/record-accessor) syntax. | _none_ | | `result_key` | If `lookup_key` is found, specifies the name of the new key to add to the output record. This new key uses the corresponding value from the second column of the CSV file in the same row where `lookup_key` was found. | _none_ | | `ignore_case` | Specifies whether to ignore case when searching for `lookup_key`. If `true`, searches are case-insensitive. If `false`, searches are case-sensitive. Possible values: `true`, `false`. | `false` | From 72c4a790b0015e538d75a0c84b173c5af04af2e9 Mon Sep 17 00:00:00 2001 From: Oleg Mukhin Date: Wed, 23 Jul 2025 21:06:21 +0100 Subject: [PATCH 19/21] Update pipeline/filters/lookup.md Co-authored-by: Alexa Kreizinger Signed-off-by: Oleg Mukhin --- pipeline/filters/lookup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md index 2fc36c554..ab83ad6f3 100644 --- a/pipeline/filters/lookup.md +++ b/pipeline/filters/lookup.md @@ -95,7 +95,7 @@ The previous configuration reads log records from `devices.log` that includes th It uses the value of the `hostname` field (which was set as the `lookup_key`) to find matching values in the first column of `device-bu.csv`. ```text -hostname,business_line +host,department server-prod-001,Finance db-test-abc,Engineering db-test-abc,Marketing From 8a2f5637352c6fe3e857a678eb0c54b7add13d66 Mon Sep 17 00:00:00 2001 From: Oleg Mukhin Date: Wed, 23 Jul 2025 21:07:04 +0100 Subject: [PATCH 20/21] Update pipeline/filters/lookup.md Co-authored-by: Alexa Kreizinger Signed-off-by: Oleg Mukhin --- pipeline/filters/lookup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md index ab83ad6f3..536810f71 100644 --- a/pipeline/filters/lookup.md +++ b/pipeline/filters/lookup.md @@ -92,7 +92,7 @@ The previous configuration reads log records from `devices.log` that includes th {"hostname": " "} ``` -It uses the value of the `hostname` field (which was set as the `lookup_key`) to find matching values in the first column of `device-bu.csv`. +Because `hostname` was set as the `lookup_key`, the Lookup filter uses the value of each `hostname` key within the record to search for matching values in the first column of the CSV file. ```text host,department From 72329960a3a80d4a50b9bdd8811d4b01aa7cf3c1 Mon Sep 17 00:00:00 2001 From: Oleg Mukhin Date: Wed, 23 Jul 2025 21:07:21 +0100 Subject: [PATCH 21/21] Update pipeline/filters/lookup.md Co-authored-by: Alexa Kreizinger Signed-off-by: Oleg Mukhin --- pipeline/filters/lookup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/lookup.md b/pipeline/filters/lookup.md index 536810f71..b9e2730fc 100644 --- a/pipeline/filters/lookup.md +++ b/pipeline/filters/lookup.md @@ -10,7 +10,7 @@ The plugin supports the following configuration parameters: | :-- | :---------- | :------ | | `file` | The CSV file that the Lookup filter will use as a lookup table. This file must contain one column of keys and one column of values. The Lookup filter treats the first row as a header and ignores these values. Supports quoted fields and escaped quotes. | _none_ | | `lookup_key` | Specifies the record key whose value search for in the CSV file's first column. Supports [record accessor](../administration/configuring-fluent-bit/classic-mode/record-accessor) syntax. | _none_ | -| `result_key` | If `lookup_key` is found, specifies the name of the new key to add to the output record. This new key uses the corresponding value from the second column of the CSV file in the same row where `lookup_key` was found. | _none_ | +| `result_key` | If a CSV entry whose value matches the value of `lookup_key` is found, specifies the name of the new key to add to the output record. This new key uses the corresponding value from the second column of the CSV file in the same row where `lookup_key` was found. | _none_ | | `ignore_case` | Specifies whether to ignore case when searching for `lookup_key`. If `true`, searches are case-insensitive. If `false`, searches are case-sensitive. Possible values: `true`, `false`. | `false` | ## Example configuration