Open
Description
We're seeing pretty consistant errors in some of our larger clusters where a lot of workloads are using pub/sub backlog scaling:
unable to get external metric pendo-io/pubsub.googleapis.com|subscription|num_undelivered_messages/&LabelSelector{MatchLabels:map[string]string{resource.labels.project_id: redacted,resource.labels.subscription_id: redacted,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server could not find the descriptor for metric pubsub.googleapis.com/subscription/num_undelivered_messages: googleapi: Error 429: Quota exceeded for quota metric 'Descriptor read requests' and limit 'Descriptor read requests per minute' of service 'monitoring.googleapis.com' for consumer 'project_number:redacted'.
Details:
[
{
"@type": "type.googleapis.com/google.rpc.ErrorInfo",
"domain": "googleapis.com",
"metadata": {
"consumer": "projects/redacted",
"quota_limit": "DescriptorReadRequestsPerMinutePerProject",
"quota_limit_value": "6000",
"quota_location": "global",
"quota_metric": "monitoring.googleapis.com/descriptor_read_requests",
"service": "monitoring.googleapis.com"
},
"reason": "RATE_LIMIT_EXCEEDED"
}
]
, rateLimitExceeded
This seems to be coming from calls to (t *Translator) GetMetricKind
which makes this call every time it's called
Metadata
Metadata
Assignees
Labels
No labels