Skip to content

Commit 8ccf2f2

Browse files
authored
feature: #146 add database discovery receiver (#145)
1 parent b94db58 commit 8ccf2f2

24 files changed

+2544
-0
lines changed

.gitattributes

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Normalize and enforce LF line endings for all text files
2+
* text=auto eol=lf

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
## vNext
44
- Add `mqttreceiver`
55
- `swohostmetricsreceiver` utilizes OTEL generated code instead of original internal implementation. Relevant code cleanup.
6+
- Added receiver `swok8sdiscovery` to publish known database entities and relationships
67

78
## v0.131.9
89
- Updates golang to 1.25.1

receiver/swok8sdiscovery/Makefile

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
include ../../Makefile.Common

receiver/swok8sdiscovery/README.md

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
## swok8sdiscovery Receiver
2+
3+
| Status | |
4+
| ------------- |-----------|
5+
| Stability | [alpha]: logs |
6+
| Distributions | [k8s] |
7+
| Issues | [![Open issues](https://img.shields.io/github/issues-search/solarwinds/solarwinds-otel-collector-contrib?query=is%3Aissue%20is%3Aopen%20label%3Areceiver%2Fswok8sdiscovery%20&label=open&color=orange&logo=opentelemetry)](https://github.com/solarwinds/solarwinds-otel-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Areceiver%2Fswok8sdiscovery) [![Closed issues](https://img.shields.io/github/issues-search/solarwinds/solarwinds-otel-collector-contrib?query=is%3Aissue%20is%3Aclosed%20label%3Areceiver%2Fswok8sdiscovery%20&label=closed&color=blue&logo=opentelemetry)](https://github.com/solarwinds/solarwinds-otel-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Areceiver%2Fswok8sdiscovery) |
8+
9+
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
10+
[k8s]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-k8s
11+
12+
13+
14+
The `swok8sdiscovery` receiver performs periodic discovery of databases used in a Kubernetes cluster and emits **entity events as OpenTelemetry log records** describing discovered database instances and their relationships to owning Kubernetes workloads (Deployment / StatefulSet / DaemonSet / Job / CronJob).
15+
16+
Discovery currently supports two complementary strategies (both optional – enable either or both):
17+
18+
1. Image-based discovery (`database.image_rules`):
19+
- Matches container image names against user-provided regular expressions.
20+
- Optionally constrains to a single default port when specified.
21+
- Resolves a stable endpoint using the best matching Service (selector overlaps chosen ports) or falls back to the Pod name.
22+
2. Domain-based discovery (`database.domain_rules`):
23+
- Matches `ExternalName` Services whose external DNS name matches configured patterns.
24+
- When multiple rules match, the one whose `database_type` or any of its `domain_hints` appears in either the service name or external domain is preferred.
25+
26+
Each discovered database produces:
27+
* An entity state log (type = `entity_state`) with attributes under `otel.entity.id` identifying the database (`sw.discovery.dbo.address`, `sw.discovery.dbo.type`, `sw.discovery.id`).
28+
* (If workload ownership resolved) A relationship log (type = `entity_relationship_state`) linking the database entity to a Kubernetes workload (relation type `DiscoveredBy`).
29+
30+
### Emitted Attributes (selection)
31+
| Attribute | Description |
32+
|-----------|-------------|
33+
| `otel.entity.event.type` | `entity_state` or `entity_relationship_state` |
34+
| `otel.entity.type` | Always `DiscoveredDatabaseInstance` for entity events |
35+
| `sw.discovery.dbo.address` | Database endpoint |
36+
| `sw.discovery.dbo.port` | Exact port on which database is running |
37+
| `sw.discovery.dbo.possible.ports` | Detected ports on database endpoint |
38+
| `sw.discovery.dbo.type` | Logical database type (e.g. `mongo`, `postgres`, `redis`) |
39+
| `sw.discovery.dbo.name` | Endpoint plus workload name (`<endpoint>#<workload>`) when workload present |
40+
| `sw.discovery.source` | Value of configured `reporter` (for provenance) |
41+
| `k8s.<workload kind>.name` | Name of owning workload (when resolved) |
42+
| `k8s.namespace.name` | Namespace of the workload/pod/service |
43+
| `sw.k8s.cluster.uid` | Cluster UID (from environment `CLUSTER_UID`) |
44+
45+
### Configuration
46+
47+
Top-level settings:
48+
49+
| Field | Type | Default | Description |
50+
|-------|------|---------|-------------|
51+
| `interval` | duration | `5m` | Time between discovery cycles. Shorten in tests (e.g. `15s`). |
52+
| `reporter` | string | empty | Optional source label recorded as `sw.discovery.source`. |
53+
| `k8s` auth fields | (inlined via `APIConfig`) | | Standard Kubernetes client auth (service account, kubeconfig, etc.). |
54+
| `database` | object | nil | Enables database discovery if provided. |
55+
56+
`database.image_rules` entries:
57+
| Field | Type | Required | Description |
58+
|-------|------|----------|-------------|
59+
| `database_type` | string | yes | Logical database type label. |
60+
| `patterns` | []string (regex) | yes | Regex patterns matched against full container image (e.g. `docker.io/library/mongo:.*`). |
61+
| `default_port` | int | no | If present and exists among container ports, only that port will be emitted (deduping multi-port images). |
62+
63+
`database.domain_rules` entries:
64+
| Field | Type | Required | Description |
65+
|-------|------|----------|-------------|
66+
| `database_type` | string | yes | Logical database type label. |
67+
| `patterns` | []string (regex) | yes | Patterns matched against `ExternalName` value. |
68+
| `domain_hints` | []string | no | Tie-break hints (substring matches in service name or external domain). |
69+
70+
### Example Configuration
71+
72+
```yaml
73+
receivers:
74+
swok8sdiscovery:
75+
interval: 30s
76+
reporter: "agent"
77+
# Kubernetes auth (service account in-cluster example)
78+
auth_type: serviceAccount
79+
database:
80+
image_rules:
81+
- database_type: mongo
82+
patterns: [".*/mongo:.*"]
83+
default_port: 27017
84+
- database_type: postgres
85+
patterns: [".*/postgres:.*"]
86+
default_port: 5432
87+
domain_rules:
88+
- database_type: redis
89+
patterns: [".*redis.example.com"]
90+
91+
exporters:
92+
debug:
93+
verbosity: detailed
94+
95+
service:
96+
pipelines:
97+
logs:
98+
receivers: [swok8sdiscovery]
99+
exporters: [debug]
100+
```

receiver/swok8sdiscovery/config.go

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
// Copyright 2025 SolarWinds Worldwide, LLC. All rights reserved.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
package swok8sdiscovery
16+
17+
import (
18+
"context"
19+
"errors"
20+
"regexp"
21+
"strings"
22+
"time"
23+
24+
"github.com/solarwinds/solarwinds-otel-collector-contrib/internal/k8sconfig"
25+
corev1 "k8s.io/api/core/v1"
26+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
27+
k8s "k8s.io/client-go/kubernetes"
28+
)
29+
30+
const (
31+
defaultInterval time.Duration = time.Minute * 5
32+
)
33+
34+
type Config struct {
35+
k8sconfig.APIConfig `mapstructure:",squash"`
36+
37+
Interval time.Duration `mapstructure:"interval"`
38+
39+
Reporter string `mapstructure:"reporter"`
40+
41+
Database *DatabaseDiscoveryConfig `mapstructure:"database"`
42+
43+
// For mocking purposes only.
44+
makeClient func() (k8s.Interface, error)
45+
}
46+
47+
type DatabaseDiscoveryConfig struct {
48+
ImageRules []*ImageRule `mapstructure:"image_rules"`
49+
DomainRules []*DomainRule `mapstructure:"domain_rules"`
50+
}
51+
52+
type ImageRule struct {
53+
DatabaseType string `mapstructure:"database_type"`
54+
// regular expressions patterns to match against container images
55+
Patterns []string `mapstructure:"patterns"`
56+
PatternsCompiled []*regexp.Regexp `mapstructure:"-"` // compiled from Patterns during validation
57+
58+
// default port for database communitation if not specified elsewhere
59+
DefaultPort int32 `mapstructure:"default_port"`
60+
}
61+
62+
type DomainRule struct {
63+
DatabaseType string `mapstructure:"database_type"`
64+
// communication endpoint must match at least one of these patterns
65+
Patterns []string `mapstructure:"patterns"`
66+
PatternsCompiled []*regexp.Regexp `mapstructure:"-"` // compiled from Patterns during validation
67+
68+
// in case more DomainRules match, this one will be preferred to be found in service name or endpoint self
69+
DomainHints []string `mapstructure:"domain_hints"`
70+
}
71+
72+
func (c *Config) Validate() error {
73+
if err := c.APIConfig.Validate(); err != nil {
74+
return err
75+
}
76+
77+
if c.Interval == 0 {
78+
c.Interval = defaultInterval
79+
}
80+
81+
// validate that rules doesn't have databaseType empty
82+
if c.Database != nil {
83+
if err := ValidateDatabaseDiscovery(c.Database); err != nil {
84+
return err
85+
}
86+
}
87+
88+
return nil
89+
}
90+
91+
func ValidateDatabaseDiscovery(databaseDiscovery *DatabaseDiscoveryConfig) error {
92+
for i := range databaseDiscovery.ImageRules {
93+
r := databaseDiscovery.ImageRules[i]
94+
if r.DatabaseType == "" {
95+
return errors.New("database_type must be specified for all image_rules")
96+
}
97+
r.DatabaseType = strings.ToLower(r.DatabaseType)
98+
99+
if len(r.Patterns) == 0 {
100+
return errors.New("at least one match pattern must be specified for all image_rules")
101+
}
102+
103+
r.PatternsCompiled = make([]*regexp.Regexp, len(r.Patterns))
104+
for j, pattern := range r.Patterns {
105+
compiled, err := regexp.Compile(pattern)
106+
if err != nil {
107+
return err
108+
}
109+
r.PatternsCompiled[j] = compiled
110+
}
111+
}
112+
113+
for i := range databaseDiscovery.DomainRules {
114+
r := databaseDiscovery.DomainRules[i]
115+
if r.DatabaseType == "" {
116+
return errors.New("database_type must be specified for all domain_rules")
117+
}
118+
r.DatabaseType = strings.ToLower(r.DatabaseType)
119+
if len(r.Patterns) == 0 {
120+
return errors.New("at least one match pattern must be specified for all domain_rules")
121+
}
122+
123+
r.PatternsCompiled = make([]*regexp.Regexp, len(r.Patterns))
124+
for j, pattern := range r.Patterns {
125+
compiled, err := regexp.Compile(pattern)
126+
if err != nil {
127+
return err
128+
}
129+
r.PatternsCompiled[j] = compiled
130+
}
131+
132+
for j, hint := range r.DomainHints {
133+
r.DomainHints[j] = strings.ToLower(hint)
134+
}
135+
}
136+
return nil
137+
}
138+
139+
func (c *Config) getClient() (k8s.Interface, error) {
140+
if c.makeClient != nil {
141+
return c.makeClient()
142+
}
143+
return k8sconfig.MakeClient(c.APIConfig)
144+
}
145+
146+
// listPods lists all pods across all namespaces using the typed client.
147+
func (c *Config) listPods(ctx context.Context, client k8s.Interface) ([]corev1.Pod, error) {
148+
pl, err := client.CoreV1().Pods(metav1.NamespaceAll).List(ctx, metav1.ListOptions{})
149+
if err != nil {
150+
return nil, err
151+
}
152+
return pl.Items, nil
153+
}
154+
155+
// listServices lists all services across all namespaces using the typed client.
156+
func (c *Config) listServices(ctx context.Context, client k8s.Interface) ([]corev1.Service, error) {
157+
sl, err := client.CoreV1().Services(metav1.NamespaceAll).List(ctx, metav1.ListOptions{})
158+
if err != nil {
159+
return nil, err
160+
}
161+
return sl.Items, nil
162+
}

0 commit comments

Comments
 (0)