-
Notifications
You must be signed in to change notification settings - Fork 1
feature: #146 add database discovery receiver #145
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 13 commits
dde3657
c39090f
5b49a11
9fc9a8a
54c44a7
10aa6f8
8be7bfd
64ac1e0
6f15497
7c4f78e
770cf22
c849946
2076c2f
96361cc
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
# Normalize and enforce LF line endings for all text files | ||
* text=auto eol=lf |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
include ../../Makefile.Common |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,98 @@ | ||
## swok8sdiscovery Receiver | ||
|
||
| Status | | | ||
| ------------- |-----------| | ||
| Stability | [beta]: logs | | ||
| Distributions | [k8s] | | ||
| Issues | [](https://github.com/solarwinds/solarwinds-otel-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Areceiver%2Fswok8sdiscovery) [](https://github.com/solarwinds/solarwinds-otel-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Areceiver%2Fswok8sdiscovery) | | ||
|
||
[beta]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#beta | ||
[k8s]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-k8s | ||
|
||
|
||
|
||
The `swok8sdiscovery` receiver performs periodic discovery of databases used in a Kubernetes cluster and emits **entity events as OpenTelemetry log records** describing discovered database instances and their relationships to owning Kubernetes workloads (Deployment / StatefulSet / DaemonSet / Job / CronJob). | ||
|
||
Discovery currently supports two complementary strategies (both optional – enable either or both): | ||
|
||
1. Image-based discovery (`database.image_rules`): | ||
- Matches container image names against user-provided regular expressions. | ||
- Optionally constrains to a single default port when specified. | ||
- Resolves a stable endpoint using the best matching Service (selector overlaps chosen ports) or falls back to the Pod name. | ||
2. Domain-based discovery (`database.domain_rules`): | ||
- Matches `ExternalName` Services whose external DNS name matches configured patterns. | ||
- When multiple rules match, the one whose `database_type` or any of its `domain_hints` appears in either the service name or external domain is preferred. | ||
|
||
Each discovered database produces: | ||
* An entity state log (type = `entity_state`) with attributes under `otel.entity.id` identifying the database (`sw.discovery.dbo.address`, `sw.discovery.dbo.type`, `sw.discovery.id`). | ||
* (If workload ownership resolved) A relationship log (type = `entity_relationship_state`) linking the database entity to a Kubernetes workload (relation type `DiscoveredBy`). | ||
|
||
### Emitted Attributes (selection) | ||
| Attribute | Description | | ||
|-----------|-------------| | ||
| `otel.entity.event.type` | `entity_state` or `entity_relationship_state` | | ||
| `otel.entity.type` | Always `DiscoveredDatabaseInstance` for entity events | | ||
| `sw.discovery.dbo.address` | Endpoint + (resolved) port list, e.g. `mongo-svc:27017` | | ||
| `sw.discovery.dbo.type` | Logical database type (e.g. `mongo`, `postgres`, `redis`) | | ||
| `sw.discovery.dbo.name` | Endpoint plus workload name (`<endpoint>#<workload>`) when workload present | | ||
| `sw.discovery.source` | Value of configured `reporter` (for provenance) | | ||
| `k8s.<workload kind>.name` | Name of owning workload (when resolved) | | ||
| `k8s.namespace.name` | Namespace of the workload/pod/service | | ||
| `sw.k8s.cluster.uid` | Cluster UID (from environment `CLUSTER_UID`) | | ||
|
||
### Configuration | ||
|
||
Top-level settings: | ||
|
||
| Field | Type | Default | Description | | ||
|-------|------|---------|-------------| | ||
| `interval` | duration | `5m` | Time between discovery cycles. Shorten in tests (e.g. `15s`). | | ||
| `reporter` | string | empty | Optional source label recorded as `sw.discovery.source`. | | ||
| `k8s` auth fields | (inlined via `APIConfig`) | | Standard Kubernetes client auth (service account, kubeconfig, etc.). | | ||
| `database` | object | nil | Enables database discovery if provided. | | ||
|
||
`database.image_rules` entries: | ||
| Field | Type | Required | Description | | ||
|-------|------|----------|-------------| | ||
| `database_type` | string | yes | Logical database type label. | | ||
| `patterns` | []string (regex) | yes | Regex patterns matched against full container image (e.g. `docker.io/library/mongo:.*`). | | ||
| `default_port` | int | no | If present and exists among container ports, only that port will be emitted (deduping multi-port images). | | ||
|
||
`database.domain_rules` entries: | ||
| Field | Type | Required | Description | | ||
|-------|------|----------|-------------| | ||
| `database_type` | string | yes | Logical database type label. | | ||
| `patterns` | []string (regex) | yes | Patterns matched against `ExternalName` value. | | ||
| `domain_hints` | []string | no | Tie-break hints (substring matches in service name or external domain). | | ||
|
||
### Example Configuration | ||
|
||
```yaml | ||
receivers: | ||
swok8sdiscovery: | ||
interval: 30s | ||
reporter: "agent" | ||
# Kubernetes auth (service account in-cluster example) | ||
auth_type: serviceAccount | ||
database: | ||
image_rules: | ||
- database_type: mongo | ||
patterns: [".*/mongo:.*"] | ||
default_port: 27017 | ||
- database_type: postgres | ||
patterns: [".*/postgres:.*"] | ||
default_port: 5432 | ||
domain_rules: | ||
- database_type: redis | ||
patterns: [".*redis.example.com"] | ||
|
||
exporters: | ||
debug: | ||
verbosity: detailed | ||
|
||
service: | ||
pipelines: | ||
logs: | ||
receivers: [swok8sdiscovery] | ||
exporters: [debug] | ||
``` |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,162 @@ | ||
// Copyright 2025 SolarWinds Worldwide, LLC. All rights reserved. | ||
// | ||
// Licensed under the Apache License, Version 2.0 (the "License"); | ||
// you may not use this file except in compliance with the License. | ||
// You may obtain a copy of the License at | ||
// | ||
// http://www.apache.org/licenses/LICENSE-2.0 | ||
// | ||
// Unless required by applicable law or agreed to in writing, software | ||
// distributed under the License is distributed on an "AS IS" BASIS, | ||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
// See the License for the specific language governing permissions and | ||
// limitations under the License. | ||
|
||
package swok8sdiscovery | ||
|
||
import ( | ||
"context" | ||
"errors" | ||
"regexp" | ||
"strings" | ||
"time" | ||
|
||
"github.com/solarwinds/solarwinds-otel-collector-contrib/internal/k8sconfig" | ||
corev1 "k8s.io/api/core/v1" | ||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" | ||
k8s "k8s.io/client-go/kubernetes" | ||
) | ||
|
||
const ( | ||
defaultInterval time.Duration = time.Minute * 5 | ||
) | ||
|
||
type Config struct { | ||
k8sconfig.APIConfig `mapstructure:",squash"` | ||
|
||
Interval time.Duration `mapstructure:"interval"` | ||
|
||
Reporter string `mapstructure:"reporter"` | ||
|
||
Database *DatabaseDiscoveryConfig `mapstructure:"database"` | ||
|
||
// For mocking purposes only. | ||
makeClient func() (k8s.Interface, error) | ||
} | ||
|
||
type DatabaseDiscoveryConfig struct { | ||
ImageRules []*ImageRule `mapstructure:"image_rules"` | ||
DomainRules []*DomainRule `mapstructure:"domain_rules"` | ||
} | ||
|
||
type ImageRule struct { | ||
DatabaseType string `mapstructure:"database_type"` | ||
// regular expressions patterns to match against container images | ||
Patterns []string `mapstructure:"patterns"` | ||
PatternsCompiled []*regexp.Regexp `mapstructure:"-"` // compiled from Patterns during validation | ||
|
||
// default port for database communitation if not specified elsewhere | ||
DefaultPort int32 `mapstructure:"default_port"` | ||
} | ||
|
||
type DomainRule struct { | ||
DatabaseType string `mapstructure:"database_type"` | ||
// communication endpoint must match at least one of these patterns | ||
Patterns []string `mapstructure:"patterns"` | ||
PatternsCompiled []*regexp.Regexp `mapstructure:"-"` // compiled from Patterns during validation | ||
|
||
// in case more DomainRules match, this one will be preferred to be found in service name or endpoint self | ||
DomainHints []string `mapstructure:"domain_hints"` | ||
} | ||
|
||
func (c *Config) Validate() error { | ||
if err := c.APIConfig.Validate(); err != nil { | ||
return err | ||
} | ||
|
||
if c.Interval == 0 { | ||
c.Interval = defaultInterval | ||
} | ||
|
||
// validate that rules doesn't have databaseType empty | ||
if c.Database != nil { | ||
if err := ValidateDatabaseDiscovery(c.Database); err != nil { | ||
return err | ||
} | ||
} | ||
|
||
return nil | ||
} | ||
|
||
func ValidateDatabaseDiscovery(databaseDiscovery *DatabaseDiscoveryConfig) error { | ||
for i := range databaseDiscovery.ImageRules { | ||
r := databaseDiscovery.ImageRules[i] | ||
if r.DatabaseType == "" { | ||
return errors.New("database_type must be specified for all image_rules") | ||
} | ||
r.DatabaseType = strings.ToLower(r.DatabaseType) | ||
|
||
if len(r.Patterns) == 0 { | ||
return errors.New("at least one match pattern must be specified for all image_rules") | ||
} | ||
|
||
r.PatternsCompiled = make([]*regexp.Regexp, len(r.Patterns)) | ||
for j, pattern := range r.Patterns { | ||
compiled, err := regexp.Compile(pattern) | ||
if err != nil { | ||
return err | ||
} | ||
r.PatternsCompiled[j] = compiled | ||
} | ||
} | ||
|
||
for i := range databaseDiscovery.DomainRules { | ||
r := databaseDiscovery.DomainRules[i] | ||
if r.DatabaseType == "" { | ||
return errors.New("database_type must be specified for all domain_rules") | ||
} | ||
r.DatabaseType = strings.ToLower(r.DatabaseType) | ||
if len(r.Patterns) == 0 { | ||
return errors.New("at least one match pattern must be specified for all domain_rules") | ||
} | ||
|
||
r.PatternsCompiled = make([]*regexp.Regexp, len(r.Patterns)) | ||
for j, pattern := range r.Patterns { | ||
compiled, err := regexp.Compile(pattern) | ||
if err != nil { | ||
return err | ||
} | ||
r.PatternsCompiled[j] = compiled | ||
} | ||
|
||
for j, hint := range r.DomainHints { | ||
r.DomainHints[j] = strings.ToLower(hint) | ||
} | ||
} | ||
return nil | ||
} | ||
|
||
func (c *Config) getClient() (k8s.Interface, error) { | ||
if c.makeClient != nil { | ||
return c.makeClient() | ||
} | ||
return k8sconfig.MakeClient(c.APIConfig) | ||
} | ||
|
||
// listPods lists all pods across all namespaces using the typed client. | ||
func (c *Config) listPods(ctx context.Context, client k8s.Interface) ([]corev1.Pod, error) { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why are There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. just as Client is a part of config, so I placed it here. |
||
pl, err := client.CoreV1().Pods(metav1.NamespaceAll).List(ctx, metav1.ListOptions{}) | ||
if err != nil { | ||
return nil, err | ||
} | ||
return pl.Items, nil | ||
} | ||
|
||
// listServices lists all services across all namespaces using the typed client. | ||
func (c *Config) listServices(ctx context.Context, client k8s.Interface) ([]corev1.Service, error) { | ||
sl, err := client.CoreV1().Services(metav1.NamespaceAll).List(ctx, metav1.ListOptions{}) | ||
if err != nil { | ||
return nil, err | ||
} | ||
return sl.Items, nil | ||
} |
Uh oh!
There was an error while loading. Please reload this page.