loki.write - flp sends data without "structured_metadata" which leads to error messages on loki for every send batch

[Nachtfalkeaw]

Hello,

I send netflow logs via flp to loki. As we can not configure any struchtured_metadata in flp this seems to be "empty" or "0 Bytes".
However this leads to repeating error messages on the loki server for every batch sent by flp.

I am not sure if this is a bug in flp or loki I opened a ticket at Grafa Loki Github:
grafana/loki#17569

My config is this:

log-level: debug
pipeline:
  - name: ipfix_netflow_ingest
  - name: transform_add_subnet
    follows: ipfix_netflow_ingest
  - name: loki_write
    follows: transform_add_subnet

parameters:
  - name: ipfix_netflow_ingest
    ingest:
      type: collector
      collector:
        hostName: u999fmlab001l
        port: 2055

  - name: transform_add_subnet
    transform:
      type: network
      network:
        rules:
          - type: add_subnet
            add_subnet:
              input: SrcAddr
              output: SrcSubnet24
              subnet_mask: /24
          - type: add_subnet
            add_subnet:
              input: SrcAddr
              output: SrcSubnet16
              subnet_mask: /16
          - type: add_subnet
            add_subnet:
              input: DstAddr
              output: DstSubnet24
              subnet_mask: /24
          - type: add_subnet
            add_subnet:
              input: DstAddr
              output: DstSubnet16
              subnet_mask: /16
          - type: decode_tcp_flags
            decode_tcp_flags:
              input: TcpFlags
              output: TcpFlagsString

  - name: loki_write
    write:
      type: loki
      loki:
        url: https://prometheus.sub.domain.de:3100
        tenantID: tenant_02
        batchWait: 15s
        batchSize: 1000000
        labels:
          - SamplerAddress
        staticLabels:
          service_name: flowlogs-pipeline
          level: info
          instance: u999fmlab001l
        ignoreList:
          - MplsCount
          - CustomList_1
          - CustomList_2
          - CustomList_3
          - CustomList_4
          - CustomList_5
          - MplsLastTtl
          - CustomInteger_1
          - CustomInteger_2
          - CustomInteger_3
          - CustomInteger_4
          - CustomInteger_5
          - Mpls_1Label
          - Mpls_2Label
          - Mpls_3Label
          - Mpls_4Label
          - Mpls_5Label
          - CustomBytes_1
          - CustomBytes_2
          - CustomBytes_3
          - CustomBytes_4
          - CustomBytes_5
          - MplsLastLabel
          - MplsLabelIp
          - HasMpls
          - Mpls_1Ttl
          - Mpls_2Ttl
          - Mpls_3Ttl
          - Mpls_4Ttl
          - Mpls_5Ttl
        timestampLabel: TimeReceived
        timestampScale: 1s

health:
  address: 127.0.0.1
  port: 9103
metricsSettings:
  address: 127.0.0.1
  suppressGoMetrics: false
  prefix: flp_operational_
  port: 9102

From a first perspective this is a loki error as it should not complain if structured_metadata is empty I think.
On the other hand it would be nice to have this feature in flp loki.write because adding "labels" is not what is recommended in loki. labels should as few as possible. If we want to add something interesting which is not in the log line itself we would like to add and which is high cardinality we should add this as structured_metadata.

So having this feature in flp would help and allign with loki features.