Figure out how to route documents to a different data stream in Elasticsearch

[zmoog]

Elastic introduced rhe reroute processor with elastic/elasticsearch#76511 and it is available since 8.8.0.

I want to explore how to use the new Elasticsearch rerouting capabilities to support new use cases and bring more value to end users.

[zmoog]

Basic example of a pipeline that routes document based on Kubernetes labels:

---
description: Pipeline for Kubernetes container logs
processors:
  - reroute:
      dataset:
        - '{{kubernetes.labels.elastic_co/dataset}}'
        - '{{data_stream.dataset}}'
      namespace:
        - '{{kubernetes.labels.elastic_co/namespace}}'
        - '{{data_stream.namespace}}'

The dataset and namespace can be a single value or and array. It will use the first valid value in the list.