Getting an EOF error when sending the events to the Opensearch

[RavinduWeerakoon]

I've configured the Kubernetes event exporter Helm release to forward events to our OpenSearch cluster, but I'm encountering an EOF error, preventing successful event delivery.

Configuration:

The following YAML configuration is being used for the event exporter:

config:
  route:
    routes:
      - match:
          - kind : "Job|CronJob"
            receiver: dump

  receivers:
    - name: dump
      opensearch:
        hosts:
          - http://opensearch-cluster-master:9200
        index: kube-events
        username: "admin"
        password: "my-password"
        useEventID: false
        deDot: false

Events are not being sent to OpenSearch. The event exporter logs show an EOF error.

OpenSearch Version:

Our OpenSearch cluster is running version 2.18.0:

 {
  "name" : "opensearch-cluster-master-1",
  "cluster_name" : "opensearch-cluster",
  "cluster_uuid" : "uYj-8XbVTF6r-QR1-s97BQ",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.18.0",
    "build_type" : "tar",
    "build_hash" : "99a9a81da366173b0c2b963b26ea92e15ef34547",
    "build_date" : "2024-10-31T19:08:04.231254959Z",
    "build_snapshot" : false,
    "lucene_version" : "9.12.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  }

what is the reason for this behaviour Is it a problem with my configuration or OpenSearch version conflict with the client library and the cluster version?

[LHozzan]

Hi there.

Try something like this:

config:
  kubeBurst: 500 # new setting
  kubeQPS: 100 # new setting
  maxEventAgeSeconds: 60 # new setting

  receivers:
    - name: dump
      opensearch:
        hosts:
          - http://opensearch-cluster-master:9200
        index: kube-events
        indexFormat: kube-events-{2003.01.02} # new setting
        username: "admin"
        password: "my-password"
        useEventID: false
        deDot: true # changed setting

It will be better for investigation, start with deDot (I think, that this is important for OpenSearch, if you want to use default layout), continue with kubeBurst and kubeQPS (there is a chance, that the exporter attempting to store too much data in short time) and last try to add indexFormat.

Try to consider incorporating maxEventAgeSeconds, if you have larger K8s cluster.

Hope, that helps :).