[DOCS-11526] Add OP buffering options

content/en/observability_pipelines/destinations/http_client.md

@@ -20,6 +20,9 @@ Set up the HTTP Client destination and its environment variables when you [set u
     - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
     - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
     - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.
+1. Optionally, toggle the switch to enable **Buffering Options**. If left disabled, the maximum size for buffering is 500 events. If enabled:
+	1. Select the buffer type you want to set (**Memory** or **Disk**).
+	1. Enter the buffer size and select the unit.
 
 ## Set the environment variables
 

content/en/observability_pipelines/performance.md

@@ -30,12 +30,16 @@ Backpressure determines if the system should slow down the consumption or accept
 
 ## In-memory buffering for components
 
-All components in Observability Pipelines have a small in-memory buffer between them. The buffer is the channel that two components communicate over. It ensures that there is a small amount of space, typically 100 events, that can be used to send events even if the component on the receiving end is busy. This allows maximizing throughput when workloads are not entirely uniform.
+All components in Observability Pipelines have a small in-memory buffer between them. In-memory buffering can also be configured for all Observability Pipelines destinations. The buffer is the channel that two components communicate over. It ensures that there is a small amount of space, typically 100 events, that can be used to send events even if the component on the receiving end is busy. This allows maximizing throughput when workloads are not entirely uniform.
 
 Buffering protects against temporary overloads or outages for a given workload. The buffering model prioritizes performance when handling an excess of events, an amount that is beyond what a destination can process, by using in-memory buffers on destinations. By default, a destination's default buffer size is increased from 100 events to 500 events. The buffer capacity is increased because destinations are typically the primary source of backpressure in any given Observability Pipelines topology. They communicate to services over the network, where latency may be introduced or outages may temporarily occur.
 
 Observability Pipelines destination's buffers are configured to block events, which means it waits indefinitely to write to a buffer that is full. This is to make sure observability data is reliably processed in the order it was given. Additionally, as mentioned earlier, blocking induces backpressure and signals upstream components to slow down event acceptance or consumption. As a result, although the system retains all data, it accumulates at the edge.
 
+## Disk buffers
+
+Observability Pipelines destination can be configured with disk buffers. When disk buffering is enabled for a destination, every event is first sent through the buffer and written to the data files, before the data is sent to the downstream integration. By default, data is not synchronized for every write, but instead synchronized on an interval (500 milliseconds), which allows for high throughput with a reduced risk of data loss.
+
 ## Further reading
 
 {{< partial name="whats-next/whats-next.html" >}}

content/en/observability_pipelines/processors/_index.md

@@ -13,10 +13,18 @@ further_reading:
 
 Use Observability Pipelines' processors to parse, structure, and enrich your logs. When you create a pipeline in the UI, pre-selected processors are added to your processor group based on the selected template. You can add additional processors and delete any existing ones based on your processing needs.
 
-Processor groups are executed from top to bottom. The order of the processors is important because logs are checked by each processor, but only logs that match the processor's filters are processed. To modify the order of the processors, use the drag handle on the top left corner of the processor you want to move.
-
 Select a processor in the left navigation menu to see more information about it.
 
+## Processor groups
+
+<div class="alert alert-warning">Configuring a pipeline with processor groups is only available for Worker versions 2.7 and later.</div>
+
+You can organize your processors into logical groups to help you manage them. Each processor group has a Group Filter so that those processors are only applied to specific logs. For example, if you want the group processors to only process logs coming from `vpc`, then use the group filter `source:vpc`. You can also add filters for each individual processor.
+
+Processor groups and the processors within each group are executed from top to bottom. The order of the processors is important because logs are checked by each processor, but only logs that match the processor's filters are processed. To change the order of the processors, use the drag handle on the top left corner of the processor you want to move.
+
+**Note**: There is a limit of 10 processor groups for a pipeline canvas. For example, if you have a dual ship pipeline, where there are two destinations and each destination has its own set of processor groups, the combined number of processor groups from both sets is limited to 10.
+
 {{% observability_pipelines/processors/filter_syntax %}}
 
 [1]: https://app.datadoghq.com/observability-pipelines

content/en/observability_pipelines/set_up_pipelines/_index.md

@@ -43,6 +43,20 @@ Set up your pipelines and its sources, processors, and destinations in the Obser
 1. Select and set up your [source][1].
 1. Select and set up your [destinations][2].
 1. Set up your [processors][3].
+    - If you want to copy a processor, click the copy icon for that processor and then use `command-v` to paste it.
+1. If you want to add another group of processors for a destination:
+    1. Click the plus sign (**+**) at the bottom of the existing processor group.
+    1. Click the name of the processor group to update it.
+    1. Optionally, enter a group filter. See [Filter Syntax](#filter-query-syntax) for more information.
+    1. Click **Add** to add processors to the group.
+    1. If you want to copy all the processors in a group and paste them into the same processor group or a different group:
+       1. Click the three dots on the processor group.
+       1. Select **Copy all processors**.
+       1. Select the processor group you want to paste the processors into and then use `command-v` to paste them.
+    1. You can toggle the switch to enable and disable the processor group and also each individual processor.
+  <br>**Notes**:
+  <br>- Configuring a pipeline with processor groups is available for Worker versions 2.7 and later.
+  <br>- There is a limit of 10 processor groups for a pipeline canvas.
 1. If you want to add another set of processors and destinations, click the plus sign (**+**) to the left of the processor group to add another set of processors and destinations to the source.
     - To delete a processor group, you need to delete all destinations linked to that processor group. When the last destination is deleted, the processor group is removed with it.
 1. If you want to add an additional destination to a processor group, click the plus sign (**+**) to the right of the processor group.

layouts/shortcodes/observability_pipelines/destination_settings/amazon_opensearch.en.md

@@ -1,4 +1,7 @@
 1. Optionally, enter the name of the Amazon OpenSearch index. See [template syntax][10051] if you want to route logs to different indexes based on specific fields in your logs.
 1. Select an authentication strategy, **Basic** or **AWS**. For **AWS**, enter the AWS region.
+1. Optionally, toggle the switch to enable **Buffering Options**. If left disabled, the maximum size for buffering is 500 events. If enabled:
+	1. Select the buffer type you want to set (**Memory** or **Disk**).
+	1. Enter the buffer size and select the unit.
 
 [10051]: /observability_pipelines/destinations/#template-syntax

layouts/shortcodes/observability_pipelines/destination_settings/amazon_security_lake.md

@@ -8,6 +8,9 @@
     - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
     - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
     - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.
+1. Optionally, toggle the switch to enable **Buffering Options**. If left disabled, the maximum size for buffering is 500 events. If enabled:
+	1. Select the buffer type you want to set (**Memory** or **Disk**).
+	1. Enter the buffer size and select the unit.
 
 **Notes**:
 - When you add the Amazon Security Lake destination, the OCSF processor is automatically added so that you can convert your logs to Parquet before they are sent to Amazon Security Lake. See [Remap to OCSF documentation][10081] for setup instructions.

layouts/shortcodes/observability_pipelines/destination_settings/chronicle.en.md

@@ -6,6 +6,9 @@ To set up the Worker's Google Chronicle destination:
     - The Worker uses standard [Google authentication methods][10005].
 1. Select **JSON** or **Raw** encoding in the dropdown menu.
 1. Enter the log type. See [template syntax][10002] if you want to route logs to different log types based on specific fields in your logs.
+1. Optionally, toggle the switch to enable **Buffering Options**. If left disabled, the maximum size for buffering is 500 events. If enabled:
+	1. Select the buffer type you want to set (**Memory** or **Disk**).
+	1. Enter the buffer size and select the unit.
 
 **Note**: Logs sent to the Google Chronicle destination must have ingestion labels. For example, if the logs are from a A10 load balancer, it must have the ingestion label `A10_LOAD_BALANCER`. See Google Cloud's [Support log types with a default parser][10003] for a list of available log types and their respective ingestion labels.
 

layouts/shortcodes/observability_pipelines/destination_settings/crowdstrike_ng_siem.md

@@ -6,6 +6,9 @@ To use the CrowdStrike NG-SIEM destination, you need to set up a CrowdStrike dat
     - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
     - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
     - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.
+1. Optionally, toggle the switch to enable **Buffering Options**. If left disabled, the maximum size for buffering is 500 events. If enabled:
+	1. Select the buffer type you want to set (**Memory** or **Disk**).
+	1. Enter the buffer size and select the unit.
 
 [10171]: https://falcon.us-2.crowdstrike.com/documentation/page/bdded008/hec-http-event-connector-guide
 [10172]: /observability_pipelines/advanced_configurations/

layouts/shortcodes/observability_pipelines/destination_settings/datadog.en.md

@@ -1,3 +1,5 @@
 <div class="alert alert-warning">Observability Pipelines compresses logs with the zstd (level 1) algorithm.</div>
 
-There are no configuration steps for your Datadog destination.
+Optionally, toggle the switch to enable **Buffering Options**. If left disabled, the maximum size for buffering is 500 events. If enabled:
+    1. Select the buffer type you want to set (**Memory** or **Disk**).
+	1. Enter the buffer size and select the unit.

layouts/shortcodes/observability_pipelines/destination_settings/datadog_archives_amazon_s3.en.md

@@ -16,6 +16,9 @@
     1. Enter the ARN of the IAM role you want to assume.
     1. Optionally, enter the assumed role session name and external ID.
     - **Note:** The [user or role you created earlier][10054] must have permission to assume this role so that the Worker can authenticate with AWS.
+1. Optionally, toggle the switch to enable **Buffering Options**. If left disabled, the maximum size for buffering is 500 events. If enabled:
+	1. Select the buffer type you want to set (**Memory** or **Disk**).
+	1. Enter the buffer size and select the unit.
 
 #### Example destination and log archive setup
 

layouts/shortcodes/observability_pipelines/destination_settings/datadog_archives_azure_storage.en.md

@@ -1,6 +1,9 @@
 1. Enter the name of the Azure container you created earlier.
-2. Optionally, enter a prefix.
+1. Optionally, enter a prefix.
     - Prefixes are useful for partitioning objects. For example, you can use a prefix as an object key to store objects under a particular directory. If using a prefix for this purpose, it must end in `/` to act as a directory path; a trailing `/` is not automatically added.
     - See [template syntax][10051] if you want to route logs to different object keys based on specific fields in your logs.
+1. Optionally, toggle the switch to enable **Buffering Options**. If left disabled, the maximum size for buffering is 500 events. If enabled:
+	1. Select the buffer type you want to set (**Memory** or **Disk**).
+	1. Enter the buffer size and select the unit.
 
 [10051]: /observability_pipelines/destinations/#template-syntax

layouts/shortcodes/observability_pipelines/destination_settings/datadog_archives_google_cloud_storage.en.md

@@ -8,6 +8,9 @@
     - Prefixes are useful for partitioning objects. For example, you can use a prefix as an object key to store objects under a particular directory. If using a prefix for this purpose, it must end in `/` to act as a directory path; a trailing `/` is not automatically added.
     - See [template syntax][10051] if you want to route logs to different object keys based on specific fields in your logs.
 1. Optionally, click **Add Header** to add metadata.
+1. Optionally, toggle the switch to enable **Buffering Options**. If left disabled, the maximum size for buffering is 500 events. If enabled:
+	1. Select the buffer type you want to set (**Memory** or **Disk**).
+	1. Enter the buffer size and select the unit.
 
 [10051]: /observability_pipelines/destinations/#template-syntax
 [10052]: https://cloud.google.com/docs/authentication#auth-flowchart

layouts/shortcodes/observability_pipelines/destination_settings/elasticsearch.en.md

@@ -1,6 +1,9 @@
 The following fields are optional:
 
 1. Enter the name for the Elasticsearch index. See [template syntax][10051] if you want to route logs to different indexes based on specific fields in your logs.
-2. Enter the Elasticsearch version.
+1. Enter the Elasticsearch version.
+1. Optionally, toggle the switch to enable **Buffering Options**. If left disabled, the maximum size for buffering is 500 events. If enabled:
+	1. Select the buffer type you want to set (**Memory** or **Disk**).
+	1. Enter the buffer size and select the unit.
 
 [10051]: /observability_pipelines/destinations/#template-syntax

layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md

@@ -33,6 +33,9 @@ To set up the Microsoft Sentinel destination, you need to create a Workspace in
     1. On the Members page, select **User, group, or service principal**.
     1. Click **Select Members** and search for the application you created in the app registration step.
     1. Click **Review + Assign**. **Note**: It can take up to 10 minutes for the IAM change to take effect.
+1. Optionally, toggle the switch to enable **Buffering Options**. If left disabled, the maximum size for buffering is 500 events. If enabled:
+	1. Select the buffer type you want to set (**Memory** or **Disk**).
+	1. Enter the buffer size and select the unit.
 
 The table below summarizes the Azure and Microsoft Sentinel information you need when you [set up the Observability Pipelines Microsoft Sentinel destination](#set-up-the-destination-in-observability-pipelines):
 

layouts/shortcodes/observability_pipelines/destination_settings/new_relic.en.md

@@ -1 +1,4 @@
-Select the data center region (**US** or **EU**) of your New Relic account.
+1. Select the data center region (**US** or **EU**) of your New Relic account.
+1. Optionally, toggle the switch to enable **Buffering Options**. If left disabled, the maximum size for buffering is 500 events. If enabled:
+	1. Select the buffer type you want to set (**Memory** or **Disk**).
+	1. Enter the buffer size and select the unit.

layouts/shortcodes/observability_pipelines/destination_settings/opensearch.en.md

@@ -1,3 +1,6 @@
-Optionally, enter the name of the OpenSearch index. See [template syntax][10051] if you want to route logs to different indexes based on specific fields in your logs.
+1. Optionally, enter the name of the OpenSearch index. See [template syntax][10051] if you want to route logs to different indexes based on specific fields in your logs.
+1. Optionally, toggle the switch to enable **Buffering Options**. If left disabled, the maximum size for buffering is 500 events. If enabled:
+	1. Select the buffer type you want to set (**Memory** or **Disk**).
+	1. Enter the buffer size and select the unit.
 
 [10051]: /observability_pipelines/destinations/#template-syntax

layouts/shortcodes/observability_pipelines/destination_settings/sentinelone.md

@@ -1 +1,4 @@
-Select your SentinelOne logs environment in the dropdown menu.
+1. Select your SentinelOne logs environment in the dropdown menu.
+1. Optionally, toggle the switch to enable **Buffering Options**. If left disabled, the maximum size for buffering is 500 events. If enabled:
+	1. Select the buffer type you want to set (**Memory** or **Disk**).
+	1. Enter the buffer size and select the unit.

layouts/shortcodes/observability_pipelines/destination_settings/socket.md

@@ -4,3 +4,6 @@
     -   `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
     -   `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
     -   `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.
+1. Optionally, toggle the switch to enable **Buffering Options**. If left disabled, the maximum size for buffering is 500 events. If enabled:
+	1. Select the buffer type you want to set (**Memory** or **Disk**).
+	1. Enter the buffer size and select the unit.

layouts/shortcodes/observability_pipelines/destination_settings/splunk_hec.en.md

@@ -4,5 +4,8 @@ The following fields are optional:
 1. Enter the name of the Splunk index you want your data in. This has to be an allowed index for your HEC. See [template syntax][10051] if you want to route logs to different indexes based on specific fields in your logs.
 1.  Select whether the timestamp should be auto-extracted. If set to `true`, Splunk extracts the timestamp from the message with the expected format of `yyyy-mm-dd hh:mm:ss`.
 1. Optionally, set the `sourcetype` to override Splunk's default value, which is `httpevent` for HEC data. See [template syntax][10051] if you want to route logs to different source types based on specific fields in your logs.
+1. Optionally, toggle the switch to enable **Buffering Options**. If left disabled, the maximum size for buffering is 500 events. If enabled:
+	1. Select the buffer type you want to set (**Memory** or **Disk**).
+	1. Enter the buffer size and select the unit.
 
 [10051]: /observability_pipelines/destinations/#template-syntax

layouts/shortcodes/observability_pipelines/destination_settings/sumo_logic.en.md

@@ -5,3 +5,6 @@ The following fields are optional:
 1. Enter a **host name** to override the default `host` value configured for your Sumo Logic collector's source.
 1. Enter a **category name** to override the default `category` value configured for your Sumo Logic collector's source.
 1. Click **Add Header** to add any custom header fields and values.
+1. Optionally, toggle the switch to enable **Buffering Options**. If left disabled, the maximum size for buffering is 500 events. If enabled:
+	1. Select the buffer type you want to set (**Memory** or **Disk**).
+	1. Enter the buffer size and select the unit.

layouts/shortcodes/observability_pipelines/destination_settings/syslog.en.md

@@ -19,4 +19,7 @@ The following destination settings are optional:
     - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
     - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
     - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.
-1. Enter the number of seconds to wait before sending TCP keepalive probes on an idle connection.
+1. Enter the number of seconds to wait before sending TCP keepalive probes on an idle connection.
+1. Optionally, toggle the switch to enable **Buffering Options**. If left disabled, the maximum size for buffering is 500 events. If enabled:
+	1. Select the buffer type you want to set (**Memory** or **Disk**).
+	1. Enter the buffer size and select the unit.