-
Notifications
You must be signed in to change notification settings - Fork 1
feature: #146 add database discovery receiver #145
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
etichy
wants to merge
14
commits into
main
Choose a base branch
from
feature/add-database-discovery-receiver
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+2,505
−0
Open
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
dde3657
Add database discovery receiver
etichy c39090f
Merge branch 'main' into feature/add-database-discovery-receiver
etichy 5b49a11
Additional changes
etichy 9fc9a8a
Fix license header
etichy 54c44a7
Normalize Go files to LF line endings
etichy 10aa6f8
fix eoln
etichy 8be7bfd
Fix
etichy 64ac1e0
update configuration and readme
etichy 6f15497
Fix PR comments
etichy 7c4f78e
Fix PR comments
etichy 770cf22
Merge branch 'main' into feature/add-database-discovery-receiver
etichy c849946
Fix
etichy 2076c2f
Update changelog
etichy 96361cc
Update receiver/swok8sdiscovery/README.md
etichy File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
# Normalize and enforce LF line endings for all text files | ||
* text=auto eol=lf |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
include ../../Makefile.Common |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,98 @@ | ||
## swok8sdiscovery Receiver | ||
|
||
| Status | | | ||
| ------------- |-----------| | ||
| Stability | [alpha]: logs | | ||
| Distributions | [k8s] | | ||
| Issues | [](https://github.com/solarwinds/solarwinds-otel-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Areceiver%2Fswok8sdiscovery) [](https://github.com/solarwinds/solarwinds-otel-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Areceiver%2Fswok8sdiscovery) | | ||
|
||
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha | ||
[k8s]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-k8s | ||
|
||
|
||
|
||
The `swok8sdiscovery` receiver performs periodic discovery of databases used in a Kubernetes cluster and emits **entity events as OpenTelemetry log records** describing discovered database instances and their relationships to owning Kubernetes workloads (Deployment / StatefulSet / DaemonSet / Job / CronJob). | ||
|
||
Discovery currently supports two complementary strategies (both optional – enable either or both): | ||
|
||
1. Image-based discovery (`database.image_rules`): | ||
- Matches container image names against user-provided regular expressions. | ||
- Optionally constrains to a single default port when specified. | ||
- Resolves a stable endpoint using the best matching Service (selector overlaps chosen ports) or falls back to the Pod name. | ||
2. Domain-based discovery (`database.domain_rules`): | ||
- Matches `ExternalName` Services whose external DNS name matches configured patterns. | ||
- When multiple rules match, the one whose `database_type` or any of its `domain_hints` appears in either the service name or external domain is preferred. | ||
|
||
Each discovered database produces: | ||
* An entity state log (type = `entity_state`) with attributes under `otel.entity.id` identifying the database (`sw.discovery.dbo.address`, `sw.discovery.dbo.type`, `sw.discovery.id`). | ||
* (If workload ownership resolved) A relationship log (type = `entity_relationship_state`) linking the database entity to a Kubernetes workload (relation type `DiscoveredBy`). | ||
|
||
### Emitted Attributes (selection) | ||
| Attribute | Description | | ||
|-----------|-------------| | ||
| `otel.entity.event.type` | `entity_state` or `entity_relationship_state` | | ||
| `otel.entity.type` | Always `DiscoveredDatabaseInstance` for entity events | | ||
| `sw.discovery.dbo.address` | Endpoint + (resolved) port list, e.g. `mongo-svc:27017` | | ||
| `sw.discovery.dbo.type` | Logical database type (e.g. `mongo`, `postgres`, `redis`) | | ||
| `sw.discovery.dbo.name` | Endpoint plus workload name (`<endpoint>#<workload>`) when workload present | | ||
| `sw.discovery.source` | Value of configured `reporter` (for provenance) | | ||
| `k8s.<workload kind>.name` | Name of owning workload (when resolved) | | ||
| `k8s.namespace.name` | Namespace of the workload/pod/service | | ||
| `sw.k8s.cluster.uid` | Cluster UID (from environment `CLUSTER_UID`) | | ||
|
||
### Configuration | ||
|
||
Top-level settings: | ||
|
||
| Field | Type | Default | Description | | ||
|-------|------|---------|-------------| | ||
| `interval` | duration | `5m` | Time between discovery cycles. Shorten in tests (e.g. `15s`). | | ||
| `reporter` | string | empty | Optional source label recorded as `sw.discovery.source`. | | ||
| `k8s` auth fields | (inlined via `APIConfig`) | | Standard Kubernetes client auth (service account, kubeconfig, etc.). | | ||
| `database` | object | nil | Enables database discovery if provided. | | ||
|
||
`database.image_rules` entries: | ||
| Field | Type | Required | Description | | ||
|-------|------|----------|-------------| | ||
| `database_type` | string | yes | Logical database type label. | | ||
| `patterns` | []string (regex) | yes | Regex patterns matched against full container image (e.g. `docker.io/library/mongo:.*`). | | ||
| `default_port` | int | no | If present and exists among container ports, only that port will be emitted (deduping multi-port images). | | ||
|
||
`database.domain_rules` entries: | ||
| Field | Type | Required | Description | | ||
|-------|------|----------|-------------| | ||
| `database_type` | string | yes | Logical database type label. | | ||
| `patterns` | []string (regex) | yes | Patterns matched against `ExternalName` value. | | ||
| `domain_hints` | []string | no | Tie-break hints (substring matches in service name or external domain). | | ||
|
||
### Example Configuration | ||
|
||
```yaml | ||
receivers: | ||
swok8sdiscovery: | ||
interval: 30s | ||
reporter: "agent" | ||
# Kubernetes auth (service account in-cluster example) | ||
auth_type: serviceAccount | ||
database: | ||
image_rules: | ||
- database_type: mongo | ||
patterns: [".*/mongo:.*"] | ||
default_port: 27017 | ||
- database_type: postgres | ||
patterns: [".*/postgres:.*"] | ||
default_port: 5432 | ||
domain_rules: | ||
- database_type: redis | ||
patterns: [".*redis.example.com"] | ||
|
||
exporters: | ||
debug: | ||
verbosity: detailed | ||
|
||
service: | ||
pipelines: | ||
logs: | ||
receivers: [swok8sdiscovery] | ||
exporters: [debug] | ||
``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,162 @@ | ||
// Copyright 2025 SolarWinds Worldwide, LLC. All rights reserved. | ||
// | ||
// Licensed under the Apache License, Version 2.0 (the "License"); | ||
// you may not use this file except in compliance with the License. | ||
// You may obtain a copy of the License at | ||
// | ||
// http://www.apache.org/licenses/LICENSE-2.0 | ||
// | ||
// Unless required by applicable law or agreed to in writing, software | ||
// distributed under the License is distributed on an "AS IS" BASIS, | ||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
// See the License for the specific language governing permissions and | ||
// limitations under the License. | ||
|
||
package swok8sdiscovery | ||
|
||
import ( | ||
"context" | ||
"errors" | ||
"regexp" | ||
"strings" | ||
"time" | ||
|
||
"github.com/solarwinds/solarwinds-otel-collector-contrib/internal/k8sconfig" | ||
corev1 "k8s.io/api/core/v1" | ||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" | ||
k8s "k8s.io/client-go/kubernetes" | ||
) | ||
|
||
const ( | ||
defaultInterval time.Duration = time.Minute * 5 | ||
) | ||
|
||
type Config struct { | ||
k8sconfig.APIConfig `mapstructure:",squash"` | ||
|
||
Interval time.Duration `mapstructure:"interval"` | ||
|
||
Reporter string `mapstructure:"reporter"` | ||
|
||
Database *DatabaseDiscoveryConfig `mapstructure:"database"` | ||
|
||
// For mocking purposes only. | ||
makeClient func() (k8s.Interface, error) | ||
} | ||
|
||
type DatabaseDiscoveryConfig struct { | ||
ImageRules []*ImageRule `mapstructure:"image_rules"` | ||
DomainRules []*DomainRule `mapstructure:"domain_rules"` | ||
} | ||
|
||
type ImageRule struct { | ||
DatabaseType string `mapstructure:"database_type"` | ||
// regular expressions patterns to match against container images | ||
Patterns []string `mapstructure:"patterns"` | ||
PatternsCompiled []*regexp.Regexp `mapstructure:"-"` // compiled from Patterns during validation | ||
|
||
// default port for database communitation if not specified elsewhere | ||
DefaultPort int32 `mapstructure:"default_port"` | ||
} | ||
|
||
type DomainRule struct { | ||
DatabaseType string `mapstructure:"database_type"` | ||
// communication endpoint must match at least one of these patterns | ||
Patterns []string `mapstructure:"patterns"` | ||
PatternsCompiled []*regexp.Regexp `mapstructure:"-"` // compiled from Patterns during validation | ||
|
||
// in case more DomainRules match, this one will be preferred to be found in service name or endpoint self | ||
DomainHints []string `mapstructure:"domain_hints"` | ||
} | ||
|
||
func (c *Config) Validate() error { | ||
if err := c.APIConfig.Validate(); err != nil { | ||
return err | ||
} | ||
|
||
if c.Interval == 0 { | ||
c.Interval = defaultInterval | ||
} | ||
|
||
// validate that rules doesn't have databaseType empty | ||
if c.Database != nil { | ||
if err := ValidateDatabaseDiscovery(c.Database); err != nil { | ||
return err | ||
} | ||
} | ||
|
||
return nil | ||
} | ||
|
||
func ValidateDatabaseDiscovery(databaseDiscovery *DatabaseDiscoveryConfig) error { | ||
for i := range databaseDiscovery.ImageRules { | ||
r := databaseDiscovery.ImageRules[i] | ||
if r.DatabaseType == "" { | ||
return errors.New("database_type must be specified for all image_rules") | ||
} | ||
r.DatabaseType = strings.ToLower(r.DatabaseType) | ||
|
||
if len(r.Patterns) == 0 { | ||
return errors.New("at least one match pattern must be specified for all image_rules") | ||
} | ||
|
||
r.PatternsCompiled = make([]*regexp.Regexp, len(r.Patterns)) | ||
for j, pattern := range r.Patterns { | ||
compiled, err := regexp.Compile(pattern) | ||
if err != nil { | ||
return err | ||
} | ||
r.PatternsCompiled[j] = compiled | ||
} | ||
} | ||
|
||
for i := range databaseDiscovery.DomainRules { | ||
r := databaseDiscovery.DomainRules[i] | ||
if r.DatabaseType == "" { | ||
return errors.New("database_type must be specified for all domain_rules") | ||
} | ||
r.DatabaseType = strings.ToLower(r.DatabaseType) | ||
if len(r.Patterns) == 0 { | ||
return errors.New("at least one match pattern must be specified for all domain_rules") | ||
} | ||
|
||
r.PatternsCompiled = make([]*regexp.Regexp, len(r.Patterns)) | ||
for j, pattern := range r.Patterns { | ||
compiled, err := regexp.Compile(pattern) | ||
if err != nil { | ||
return err | ||
} | ||
r.PatternsCompiled[j] = compiled | ||
} | ||
|
||
for j, hint := range r.DomainHints { | ||
r.DomainHints[j] = strings.ToLower(hint) | ||
} | ||
} | ||
return nil | ||
} | ||
|
||
func (c *Config) getClient() (k8s.Interface, error) { | ||
if c.makeClient != nil { | ||
return c.makeClient() | ||
} | ||
return k8sconfig.MakeClient(c.APIConfig) | ||
} | ||
|
||
// listPods lists all pods across all namespaces using the typed client. | ||
func (c *Config) listPods(ctx context.Context, client k8s.Interface) ([]corev1.Pod, error) { | ||
pl, err := client.CoreV1().Pods(metav1.NamespaceAll).List(ctx, metav1.ListOptions{}) | ||
if err != nil { | ||
return nil, err | ||
} | ||
return pl.Items, nil | ||
} | ||
|
||
// listServices lists all services across all namespaces using the typed client. | ||
func (c *Config) listServices(ctx context.Context, client k8s.Interface) ([]corev1.Service, error) { | ||
sl, err := client.CoreV1().Services(metav1.NamespaceAll).List(ctx, metav1.ListOptions{}) | ||
if err != nil { | ||
return nil, err | ||
} | ||
return sl.Items, nil | ||
} |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why are
listPods
andlistServices
in this file and bound to*Config
instead of being in in receiver.go?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just as Client is a part of config, so I placed it here.