loki.write - flp sends data without "structured_metadata" which leads to error messages on loki for every send batch

[Nachtfalkeaw]

Hello,

I send netflow logs via flp to loki. As we can not configure any struchtured_metadata in flp this seems to be "empty" or "0 Bytes".
However this leads to repeating error messages on the loki server for every batch sent by flp.

I am not sure if this is a bug in flp or loki I opened a ticket at Grafa Loki Github:
grafana/loki#17569

My config is this:

log-level: debug
pipeline:
  - name: ipfix_netflow_ingest
  - name: transform_add_subnet
    follows: ipfix_netflow_ingest
  - name: loki_write
    follows: transform_add_subnet

parameters:
  - name: ipfix_netflow_ingest
    ingest:
      type: collector
      collector:
        hostName: u999fmlab001l
        port: 2055

  - name: transform_add_subnet
    transform:
      type: network
      network:
        rules:
          - type: add_subnet
            add_subnet:
              input: SrcAddr
              output: SrcSubnet24
              subnet_mask: /24
          - type: add_subnet
            add_subnet:
              input: SrcAddr
              output: SrcSubnet16
              subnet_mask: /16
          - type: add_subnet
            add_subnet:
              input: DstAddr
              output: DstSubnet24
              subnet_mask: /24
          - type: add_subnet
            add_subnet:
              input: DstAddr
              output: DstSubnet16
              subnet_mask: /16
          - type: decode_tcp_flags
            decode_tcp_flags:
              input: TcpFlags
              output: TcpFlagsString

  - name: loki_write
    write:
      type: loki
      loki:
        url: https://prometheus.sub.domain.de:3100
        tenantID: tenant_02
        batchWait: 15s
        batchSize: 1000000
        labels:
          - SamplerAddress
        staticLabels:
          service_name: flowlogs-pipeline
          level: info
          instance: u999fmlab001l
        ignoreList:
          - MplsCount
          - CustomList_1
          - CustomList_2
          - CustomList_3
          - CustomList_4
          - CustomList_5
          - MplsLastTtl
          - CustomInteger_1
          - CustomInteger_2
          - CustomInteger_3
          - CustomInteger_4
          - CustomInteger_5
          - Mpls_1Label
          - Mpls_2Label
          - Mpls_3Label
          - Mpls_4Label
          - Mpls_5Label
          - CustomBytes_1
          - CustomBytes_2
          - CustomBytes_3
          - CustomBytes_4
          - CustomBytes_5
          - MplsLastLabel
          - MplsLabelIp
          - HasMpls
          - Mpls_1Ttl
          - Mpls_2Ttl
          - Mpls_3Ttl
          - Mpls_4Ttl
          - Mpls_5Ttl
        timestampLabel: TimeReceived
        timestampScale: 1s

health:
  address: 127.0.0.1
  port: 9103
metricsSettings:
  address: 127.0.0.1
  suppressGoMetrics: false
  prefix: flp_operational_
  port: 9102

From a first perspective this is a loki error as it should not complain if structured_metadata is empty I think.
On the other hand it would be nice to have this feature in flp loki.write because adding "labels" is not what is recommended in loki. labels should as few as possible. If we want to add something interesting which is not in the log line itself we would like to add and which is high cardinality we should add this as structured_metadata.

So having this feature in flp would help and allign with loki features.

[jotak]

thanks @Nachtfalkeaw for mentioning this. I'll check in our team if someone already investigated structured metadata, and if not I agree that we should take a look into it.
Regarding the use of labels, that's right that it's important to care about the cardinality, but that doesn't mean they should be entirely avoided. In the netobserv operator, which runs in kubernetes, we chose to have labels on items that have a "reasonable" cardinality: such as namespaces or workloads. But not on pods or IPs, which are indeed higher cardinality.

From your config, as I see that you create several layers of subnets, I would imagine that those subnets are good candidate for being labels. Did you try that, and did you find performance issues with that?

For information, I opened another ticket to track the structured metadata enhancement => https://issues.redhat.com/browse/NETOBSERV-2264

[Nachtfalkeaw]

Hi,

main reason was the Error Log i noticed in my Loki Environment and i thought it could be related to a Bug in flp. But later I understood that the reason is Loki itself.

The subnets are Not planned for Labels or structured_metadata. Just want it in the Log Line to later Parse These as json within Grafana.

However, structured_metadata is a Feature in Loki and it would of course be nice to have in the config, too.

[jotak]

The subnets are Not planned for Labels or structured_metadata. Just want it in the Log Line to later Parse These as json within Grafana.

Ok, I just wanted to point that it can be interesting to leverage labels for faster queries. Labels are indexes, so let's say we need to retrieve all netflows for a given IP, searching in log lines can be very slow depending on the volume of data; it can be much faster with /16 or /24 subnets labels, querying the corresponding subnet first, and then iterating through the smaller subset of results to filter for the desired IP.

[Nachtfalkeaw]

Good Point. Maybe /16 subnets May Help to narrow down logs without too high cardinality.