[CORE-14831] Cloud Topics: Metrics for L0 GC#29556
Open
oleiman wants to merge 9 commits intoredpanda-data:devfrom
Open
[CORE-14831] Cloud Topics: Metrics for L0 GC#29556oleiman wants to merge 9 commits intoredpanda-data:devfrom
oleiman wants to merge 9 commits intoredpanda-data:devfrom
Conversation
oleiman
force-pushed
the
ct/core-14831/gc-metrics
branch
2 times, most recently
from
2c6d262 to
089091b
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds comprehensive metrics instrumentation for the L0 garbage collection (GC) process in cloud topics. The changes enable monitoring of GC progress, performance, and operational health.
Changes:
- Added metrics tracking for GC operations including objects listed, skipped, deleted, collection rounds, and error counts
- Implemented epoch-based lag tracking to monitor GC progress relative to eligible epochs
- Added integration tests to verify metrics are correctly reported during GC operations
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/rptest/tests/cloud_topics/l0_gc_test.py | Adds new test class with methods to verify GC metrics are properly reported |
| src/v/cloud_topics/level_zero/gc/level_zero_gc_probe.h | Declares new metric tracking methods and internal state for epoch and operation counters |
| src/v/cloud_topics/level_zero/gc/level_zero_gc_probe.cc | Implements metric registration and epoch lag calculation logic |
| src/v/cloud_topics/level_zero/gc/level_zero_gc.cc | Instruments GC operations with probe calls to track metrics |
| src/v/cloud_topics/level_zero/gc/BUILD | Adds dependency on cloud_topics types for epoch handling |
oleiman
force-pushed
the
ct/core-14831/gc-metrics
branch
from
089091b to
895b9f5
Compare
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
oleiman
force-pushed
the
ct/core-14831/gc-metrics
branch
from
895b9f5 to
37d46a7
Compare
Given a globally max GC eligible epoch and a shard doing garbage collection, define that shard's epoch lag as the difference between the max eligible epoch and the oldest epoch we are still working on. "Oldest epoch" is a bit subtle. To a first approximation, we can say that at a given time for a given shard, the oldest epoch not yet collected is the numerically smallest eligible epoch found in list results in the bucket. Since we're tracking a monotonically increasing value, we can't track an all-time minimum, so instead we say that the point-in-time min deletion epoch is the minimum epoch found in a round of garbage collection, and we reset that value on each trip through the GC loop. This way, the tracked value has an opportunity to inch forward once all the objects of a given epoch have been collected. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
oleiman
force-pushed
the
ct/core-14831/gc-metrics
branch
from
37d46a7 to
0efd5e5
Compare
oleiman
added
the
claude-review
Collaborator
CI test resultstest results on build#80319
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Grab bag of metrics for L0 GC. Some might not be that useful.
Backports Required
Release Notes