🌱 Promtail to alloy migration #11945

irapandey · 2025-03-10T09:53:23Z

What this PR does / why we need it:
This PR is to migrate from promtail to alloy

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
#11881

k8s-ci-robot · 2025-03-10T09:53:31Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign enxebre for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-03-10T09:53:33Z

Hi @irapandey. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

irapandey · 2025-03-10T09:54:54Z

irapandey · 2025-03-10T11:04:34Z

Hey @chrischdi

I have opened this PR for promtail/alloy migration
Please take a look here.

Thanks 😄

chrischdi · 2025-03-13T14:35:59Z

/ok-to-test

/area devtools

Can you also check grafana if you're able to query logs?

chrischdi · 2025-03-13T14:41:01Z

hack/observability/alloy/values.yaml

@@ -1,5 +1,3 @@
-# Configuration for promtail chart, see https://github.com/grafana/helm-charts/tree/main/charts/promtail


Can we add the right link for alloy?

Maybe it is this? https://github.com/grafana/alloy/blob/main/operations/helm/charts/alloy/values.yaml

❯ helm show chart grafana/alloy apiVersion: v2 appVersion: v1.7.2 dependencies: - condition: crds.create name: crds repository: "" version: 0.0.0 description: Grafana Alloy icon: https://raw.githubusercontent.com/grafana/alloy/main/docs/sources/assets/alloy_icon_orange.svg name: alloy type: application version: 0.12.3

chrischdi · 2025-03-13T14:44:37Z

hack/observability/alloy/values.yaml

@@ -1,5 +1,3 @@
-# Configuration for promtail chart, see https://github.com/grafana/helm-charts/tree/main/charts/promtail
-
 config:


Pretty sure we have to change things here as its a different helm chart.

I couldn't find dedicated charts for alloy like for promtail - https://grafana.com/docs/alloy/latest/set-up/install/kubernetes/

helm show values grafana/alloy does not even have a config field.

So all configuration below would be not used (and we want them to work).

So I'm curious if this even shows us logs.

This page maybe helps: https://grafana.com/docs/alloy/latest/set-up/migrate/from-promtail/

Note: the values.yaml is not the promtail configuration.

You should get the promtail.yaml via: kustomize build --enable-helm --load-restrictor LoadRestrictionsNone hack/observability/promtail | grep stringData -A 38 | yq '.stringData[]' > promtail.yaml

Let me take a look and get back - I kinda had the same feeling but needed some input
Thanks 😄

Apologies for being MIA

I believe since Alloy used .alloy files to run - we simply add the converted promtail config to ./alloy dir and run alloy using this file

I commiting the file so you can review

hack/observability/alloy/kustomization.yaml

chrischdi · 2025-04-15T09:12:05Z

I ran tilt-up and accessed grafana.

I'm not able to see any logs so looks like alloy does not work in that configuration which is currently on the PR.

chrischdi · 2025-04-15T09:25:50Z

hack/observability/alloy/values.yaml

@@ -1,5 +1,4 @@
-# Configuration for promtail chart, see https://github.com/grafana/helm-charts/tree/main/charts/promtail
-
+# Configuration for alloy - https://github.com/grafana/alloy/blob/main/operations/helm/charts/alloy/values.yaml
 config:


I think we need to replace the file with this content:

# Configuration for alloy - https://github.com/grafana/alloy/blob/main/operations/helm/charts/alloy/values.yaml alloy: configMap: create: true name: alloy-config key: config.alloy mounts: varlog: true

hack/observability/alloy/kustomization.yaml

chrischdi · 2025-04-15T09:28:01Z

Let's have these two changes, then please check in grafana if we get logs.

After that works we can start to verify that the new config has the same effect as the previous promtail config.

irapandey · 2025-04-26T19:33:58Z

irapandey · 2025-04-27T08:07:19Z

Hey @chrischdi

I made those changes and didn't see any logs - Added a configmap separately and got logs

I tried creating it from the ./alloy/values.yaml but it didn't see to create what we were expecting

Hope this works

Thanks

chrischdi · 2025-05-09T13:59:40Z

hack/observability/alloy/values.yaml

+# config:
+#   # publish data to loki
+#   clients:
+#     - url: http://loki:3100/loki/api/v1/push
+#       # We have to set the tenant_id/OrgID to push logs to Loki
+#       # otherwise we get an auth error.
+#       tenant_id: 1
+
+#   snippets:
+#     pipelineStages:
+#       # Parse cluster and machine to make them available as labels.
+#       - cri: {}
+#       - json:
+#           expressions:
+#             controller:
+#             Cluster: join('/',[Cluster.namespace,Cluster.name])
+#             Machine: join('/',[Machine.namespace,Machine.name])
+#             KubeadmControlPlane: join('/',[KubeadmControlPlane.namespace,KubeadmControlPlane.name])
+#             MachineDeployment: join('/',[MachineDeployment.namespace,MachineDeployment.name])
+#             MachineSet: join('/',[MachineSet.namespace,MachineSet.name])
+#             MachinePool: join('/',[MachinePool.namespace,MachinePool.name])
+#       - labels:
+#           controller:
+#           Cluster:
+#           Machine:
+#           KubeadmControlPlane:
+#           MachineDeployment:
+#           MachineSet:
+#           MachinePool:
+
+
+  # Add a yaml file to generate
+
+
+


Suggested change

# config:

# # publish data to loki

# clients:

# - url: http://loki:3100/loki/api/v1/push

# # We have to set the tenant_id/OrgID to push logs to Loki

# # otherwise we get an auth error.

# tenant_id: 1

# snippets:

# pipelineStages:

# # Parse cluster and machine to make them available as labels.

# - cri: {}

# - json:

# expressions:

# controller:

# Cluster: join('/',[Cluster.namespace,Cluster.name])

# Machine: join('/',[Machine.namespace,Machine.name])

# KubeadmControlPlane: join('/',[KubeadmControlPlane.namespace,KubeadmControlPlane.name])

# MachineDeployment: join('/',[MachineDeployment.namespace,MachineDeployment.name])

# MachineSet: join('/',[MachineSet.namespace,MachineSet.name])

# MachinePool: join('/',[MachinePool.namespace,MachinePool.name])

# - labels:

# controller:

# Cluster:

# Machine:

# KubeadmControlPlane:

# MachineDeployment:

# MachineSet:

# MachinePool:

# Add a yaml file to generate

chrischdi · 2025-05-09T14:01:40Z

hack/observability/alloy/kustomization.yaml

@@ -1,13 +1,16 @@
 resources:
  - ../namespace.yaml
+  - alloy-config.yaml


Let's still use the other approach so we would get a new configmap on config changes + auto-recreation of the pods.

Suggested change

- alloy-config.yaml

And below add

configMapGenerator: - name: alloy-config files: - config.alloy

chrischdi · 2025-05-09T14:02:50Z

hack/observability/alloy/alloy-config.yaml

+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: alloy-config
+  namespace: observability
+data:
+  config.alloy: |


Let's remove this stuff and rename the file to config.alloy

chrischdi · 2025-05-09T14:03:59Z

Hey @chrischdi

I made those changes and didn't see any logs - Added a configmap separately and got logs

I tried creating it from the ./alloy/values.yaml but it didn't see to create what we were expecting

Hope this works

Thanks

Let's try this again, this should normally work :-) Let me know if you need help on it.

irapandey · 2025-05-14T10:55:28Z

Sure - now that I know where the problem was - this will be a easier fix

Co-authored-by: Christian Schlotter <chrischdi@users.noreply.github.com>

irapandey · 2025-05-20T13:29:02Z

Hey @chrischdi - I might be missing something here for sure

I have been referring to this file for values.yaml that are used by the alloy helm chart - https://github.com/grafana/alloy/blob/main/operations/helm/charts/alloy/values.yaml

## Various Alloy settings. For backwards compatibility with the grafana-agent
## chart, this field may also be called "agent". Naming this field "agent" is
## deprecated and will be removed in a future release.
alloy:
  configMap:
    # -- Create a new ConfigMap for the config file.
    create: true
    # -- Content to assign to the new ConfigMap.  This is passed into `tpl` allowing for templating from values.
    content: ''

    # -- Name of existing ConfigMap to use. Used when create is false.
    name: null
    # -- Key in ConfigMap to get config from.
    key: null

Looking at this above - I have been trying to generate a configmap with the name alloy-config, based on the comments in the code above

qq - do we want to use alloy.config to generate the configmap or adding alloy.config.content would suffice

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 10, 2025

k8s-ci-robot requested review from elmiko and jackfrancis March 10, 2025 09:53

k8s-ci-robot added do-not-merge/needs-area PR is missing an area label needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 10, 2025

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Mar 10, 2025

irapandey changed the title ~~Promtail to alloy migration~~ 🌱 Promtail to alloy migration Mar 10, 2025

chrischdi reviewed Mar 13, 2025

View reviewed changes

chrischdi reviewed Mar 14, 2025

View reviewed changes

chrischdi reviewed Mar 26, 2025

View reviewed changes

hack/observability/alloy/kustomization.yaml Show resolved Hide resolved

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 15, 2025

chrischdi reviewed Apr 15, 2025

View reviewed changes

hack/observability/alloy/kustomization.yaml Show resolved Hide resolved

irapandey force-pushed the promtail-to-alloy-migration branch from 0aefd45 to bfdcc7a Compare April 26, 2025 19:33

chrischdi reviewed May 9, 2025

View reviewed changes

irapandey and others added 5 commits May 20, 2025 17:17

Promtail to alloy migration

b0f9b50

Update hack/observability/alloy/kustomization.yaml

ddbf098

Co-authored-by: Christian Schlotter <chrischdi@users.noreply.github.com>

Added Configuration link

9136fdf

Temp-adding alloy.config for review

4cb1248

Fixing alloy config

ed83b2f

irapandey force-pushed the promtail-to-alloy-migration branch from 269b5d7 to 3d54610 Compare May 20, 2025 11:47

Fixing config alloy

861ed75

irapandey force-pushed the promtail-to-alloy-migration branch from 3d54610 to 861ed75 Compare May 20, 2025 13:27

		@@ -1,5 +1,3 @@
		# Configuration for promtail chart, see https://github.com/grafana/helm-charts/tree/main/charts/promtail

		@@ -1,5 +1,3 @@
		# Configuration for promtail chart, see https://github.com/grafana/helm-charts/tree/main/charts/promtail

		config:

🌱 Promtail to alloy migration #11945

Are you sure you want to change the base?

🌱 Promtail to alloy migration #11945

Uh oh!

Conversation

irapandey commented Mar 10, 2025

Uh oh!

k8s-ci-robot commented Mar 10, 2025

Uh oh!

k8s-ci-robot commented Mar 10, 2025

Uh oh!

irapandey commented Mar 10, 2025

Uh oh!

irapandey commented Mar 10, 2025

Uh oh!

chrischdi commented Mar 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chrischdi Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chrischdi commented Apr 15, 2025

Uh oh!

chrischdi Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chrischdi commented Apr 15, 2025

Uh oh!

irapandey commented Apr 26, 2025

Uh oh!

irapandey commented Apr 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chrischdi commented May 9, 2025

Uh oh!

irapandey commented May 14, 2025

Uh oh!

irapandey commented May 20, 2025

Uh oh!

Uh oh!

chrischdi Mar 26, 2025 •

edited

Loading

chrischdi Apr 15, 2025 •

edited

Loading