Skip to content

Jikkou - kafka config #517

@rubinaprusty

Description

@rubinaprusty

Hi Team,

I am trying to run Jikkou within docker image - streamthoughts/jikkou-api-server:main in mac and it is running fine. Now I would like to automate kafka topic creation by passing a config yml file -
My kafka cluster is running locally using localhost:9092.
I am using below yml file to run the topic to be created in kafka cluster -

apiVersion: io.streamthoughts.jikkou/v1beta1
kind: ResourceList
resources:
  # Kafka connection configuration
  - apiVersion: jikkou.io/v1beta1
    kind: KafkaConfiguration
    metadata:
      name: kafka-cluster
    spec:
      bootstrap.servers: "localhost:9092"
      security.protocol: PLAINTEXT
      
  # Define a Kafka topic
  - apiVersion: kafka.jikkou.io/v1beta1
    kind: KafkaTopic
    metadata:
      name: test-topic-20012025
    spec:
      partitions: 3
      replicas: 2
      config:
        cleanup.policy: "delete"
        retention.ms: "604800000"  # 7 days
        max.message.bytes: "1048576"

My docker file -

# Use the base image for Jikkou API Server
FROM streamthoughts/jikkou-api-server:main
COPY kafka-topics.yml /config/kafka-topics.yml
# Default command to apply Kafka topic configurations using Jikkou
CMD ["jikkou", "apply", "--files", "/config/kafka-topics.yml"]

Now I am able to run Jikkou on my machine but could not able to connect to the kafka cluster.

below is my admin config -

AdminClientConfig values: 
        auto.include.jmx.reporter = true
        bootstrap.controllers = []
        bootstrap.servers = [localhost:9092]
        client.dns.lookup = use_all_dns_ips
        client.id = 
        connections.max.idle.ms = 300000
        default.api.timeout.ms = 60000
        enable.metrics.push = true
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        receive.buffer.bytes = 65536
        reconnect.backoff.max.ms = 1000
        reconnect.backoff.ms = 50
        request.timeout.ms = 30000
        retries = 2147483647
        retry.backoff.max.ms = 1000
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = null
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.connect.timeout.ms = null
        sasl.login.read.timeout.ms = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.login.retry.backoff.max.ms = 10000
        sasl.login.retry.backoff.ms = 100
        sasl.mechanism = GSSAPI
        sasl.oauthbearer.clock.skew.seconds = 30
        sasl.oauthbearer.expected.audience = null
        sasl.oauthbearer.expected.issuer = null
        sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
        sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
        sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
        sasl.oauthbearer.jwks.endpoint.url = null
        sasl.oauthbearer.scope.claim.name = scope
        sasl.oauthbearer.sub.claim.name = sub
        sasl.oauthbearer.token.endpoint.url = null
        security.protocol = PLAINTEXT
        security.providers = null
        send.buffer.bytes = 131072
        socket.connection.setup.timeout.max.ms = 30000
        socket.connection.setup.timeout.ms = 10000
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
        ssl.endpoint.identification.algorithm = https
        ssl.engine.factory.class = null
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.certificate.chain = null
        ssl.keystore.key = null
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLSv1.3
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.certificates = null
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS

10:27:02.791 [scheduled-executor-thread-1] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka version: 3.7.0
10:27:02.792 [scheduled-executor-thread-1] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka commitId: 2ae524ed625438c5
10:27:02.792 [scheduled-executor-thread-1] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1737455222791
10:27:02.794 [scheduled-executor-thread-1] INFO  i.s.j.k.internals.KafkaBrokersReady - Checking for Kafka to be ready. Expected broker(s): 1
10:27:32.799 [kafka-admin-client-thread | adminclient-52] INFO  o.a.k.c.a.i.AdminMetadataManager - [AdminClient clientId=adminclient-52] Metadata update failed
org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: fetchMetadata
10:28:02.800 [kafka-admin-client-thread | adminclient-52] INFO  o.a.k.c.a.i.AdminMetadataManager - [AdminClient clientId=adminclient-52] Metadata update failed

I am able to create the same topic using cli command -

Please help me here what I am doing wrong - I am guessing my configuration is wrong or provide me a sample config and docker file.

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions