Skip to content

Commit 3e1c90f

Browse files
FourFifthsCodeFourFifthsCode
committed
Merge branch 'main' of github.com:christensenjairus/multicluster-runtime into kubeconfig-based-provider
2 parents b0e6cd9 + b21016e commit 3e1c90f

File tree

5 files changed

+838
-0
lines changed

5 files changed

+838
-0
lines changed

‎examples/kubeconfig/README.md

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
# Kubeconfig Provider Example
2+
3+
This example demonstrates how to use the kubeconfig provider to manage multiple Kubernetes clusters using kubeconfig secrets.
4+
5+
## Overview
6+
7+
The kubeconfig provider allows you to:
8+
1. Discover and connect to multiple Kubernetes clusters using kubeconfig secrets
9+
2. Run controllers that can operate across all discovered clusters
10+
3. Manage cluster access through RBAC rules and service accounts
11+
12+
## Directory Structure
13+
14+
```
15+
examples/kubeconfig/
16+
├── controllers/ # Example controller that simply lists pods
17+
│ ├── pod_lister.go
18+
├── scripts/ # Utility scripts
19+
│ └── create-kubeconfig-secret.sh
20+
└── main.go # Example operator implementation
21+
```
22+
23+
## Usage
24+
25+
### 1. Setting Up Cluster Access
26+
27+
Before creating a kubeconfig secret, ensure that:
28+
1. The remote cluster has a service account with the necessary RBAC permissions for your operator
29+
2. The service account exists in the namespace where you want to create the kubeconfig secret
30+
31+
Use the `create-kubeconfig-secret.sh` script to create a kubeconfig secret for each cluster you want to manage:
32+
33+
```bash
34+
./scripts/create-kubeconfig-secret.sh \
35+
--name cluster1 \
36+
-n default \
37+
-c prod-cluster \
38+
-a my-service-account
39+
```
40+
41+
The script will:
42+
- Use the specified service account from the remote cluster
43+
- Generate a kubeconfig using the service account's token
44+
- Store the kubeconfig in a secret in your local cluster
45+
46+
Command line options:
47+
- `-c, --context`: Kubeconfig context to use (required)
48+
- `--name`: Name for the secret (defaults to context name)
49+
- `-n, --namespace`: Namespace to create the secret in (default: "default")
50+
- `-a, --service-account`: Service account name to use from the remote cluster (default: "default")
51+
52+
### 2. Customizing RBAC Rules
53+
54+
The service account in the remote cluster must have the necessary RBAC permissions for your operator to function. Edit the RBAC templates in the `rbac/` directory to define the permissions your operator needs:
55+
56+
```yaml
57+
# rbac/clusterrole.yaml
58+
apiVersion: rbac.authorization.k8s.io/v1
59+
kind: ClusterRole
60+
metadata:
61+
name: ${SECRET_NAME}-role
62+
rules:
63+
# Add permissions for your operator <--------------------------------
64+
- apiGroups: [""]
65+
resources: ["pods"]
66+
verbs: ["list", "get", "watch"] # watch is needed for controllers that observe resources
67+
```
68+
69+
Important RBAC considerations:
70+
- Use `watch` verb if your controller needs to observe resource changes
71+
- Use `list` and `get` for reading resources
72+
- Use `create`, `update`, `patch`, `delete` for modifying resources
73+
- Consider using `Role` instead of `ClusterRole` if you only need namespace-scoped permissions
74+
75+
### 3. Implementing Your Operator
76+
77+
Add your controllers to `main.go`:
78+
79+
```go
80+
func main() {
81+
// Import your controllers here <--------------------------------
82+
"sigs.k8s.io/multicluster-runtime/examples/kubeconfig/controllers"
83+
84+
//...
85+
86+
// Run your controllers here <--------------------------------
87+
podWatcher := controllers.NewPodWatcher(mgr)
88+
if err := mgr.Add(podWatcher); err != nil {
89+
entryLog.Error(err, "Unable to add pod watcher")
90+
os.Exit(1)
91+
}
92+
}
93+
```
94+
95+
Your controllers can then use the manager to access any cluster and view the resources that the RBAC permissions allow.
96+
97+
## How It Works
98+
99+
1. The kubeconfig provider watches for secrets with a specific label in a namespace
100+
2. When a new secret is found, it:
101+
- Extracts the kubeconfig data
102+
- Creates a new controller-runtime cluster
103+
- Makes the cluster available to your controllers
104+
3. Your controllers can access any cluster through the manager
105+
4. RBAC rules ensure your operator has the necessary permissions in each cluster
106+
107+
## Labels and Configuration
108+
109+
The provider uses the following labels and keys by default:
110+
- Label: `sigs.k8s.io/multicluster-runtime-kubeconfig: "true"`
111+
- Secret data key: `kubeconfig`
112+
113+
You can customize these in the provider options when creating it.

‎examples/kubeconfig/controllers/pod_lister.go

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
/*
2+
Copyright 2025 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package controllers
18+
19+
import (
20+
"context"
21+
"time"
22+
23+
"github.com/go-logr/logr"
24+
25+
corev1 "k8s.io/api/core/v1"
26+
27+
"sigs.k8s.io/controller-runtime/pkg/client"
28+
"sigs.k8s.io/controller-runtime/pkg/cluster"
29+
ctrllog "sigs.k8s.io/controller-runtime/pkg/log"
30+
31+
mcmanager "sigs.k8s.io/multicluster-runtime/pkg/manager"
32+
)
33+
34+
// PodWatcher is a simple controller that watches pods across multiple clusters
35+
type PodWatcher struct {
36+
Manager mcmanager.Manager
37+
Log logr.Logger
38+
}
39+
40+
// NewPodWatcher creates a new PodWatcher
41+
func NewPodWatcher(mgr mcmanager.Manager) *PodWatcher {
42+
return &PodWatcher{
43+
Manager: mgr,
44+
Log: ctrllog.Log.WithName("pod-watcher"),
45+
}
46+
}
47+
48+
// Start implements Runnable
49+
func (p *PodWatcher) Start(ctx context.Context) error {
50+
// Nothing to do here - we'll handle everything in Engage
51+
return nil
52+
}
53+
54+
// Engage implements multicluster.Aware and gets called when a new cluster is engaged
55+
func (p *PodWatcher) Engage(ctx context.Context, clusterName string, cl cluster.Cluster) error {
56+
log := p.Log.WithValues("cluster", clusterName)
57+
log.Info("Engaging cluster")
58+
59+
// Start a goroutine to periodically list pods
60+
go func() {
61+
ticker := time.NewTicker(30 * time.Second)
62+
defer ticker.Stop()
63+
64+
// Initial list
65+
if err := p.listPods(ctx, cl, clusterName, log); err != nil {
66+
log.Error(err, "Failed to list pods")
67+
}
68+
69+
for {
70+
select {
71+
case <-ctx.Done():
72+
log.Info("Context done, stopping pod watcher")
73+
return
74+
case <-ticker.C:
75+
if err := p.listPods(ctx, cl, clusterName, log); err != nil {
76+
log.Error(err, "Failed to list pods")
77+
}
78+
}
79+
}
80+
}()
81+
82+
return nil
83+
}
84+
85+
// listPods lists pods in the default namespace
86+
func (p *PodWatcher) listPods(ctx context.Context, cl cluster.Cluster, clusterName string, log logr.Logger) error {
87+
var pods corev1.PodList
88+
if err := cl.GetClient().List(ctx, &pods, &client.ListOptions{
89+
Namespace: "default",
90+
}); err != nil {
91+
return err
92+
}
93+
94+
log.Info("Pods in default namespace", "count", len(pods.Items))
95+
for _, pod := range pods.Items {
96+
log.Info("Pod",
97+
"name", pod.Name,
98+
"status", pod.Status.Phase)
99+
}
100+
101+
return nil
102+
}

‎examples/kubeconfig/main.go

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
/*
2+
Copyright 2025 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package main
18+
19+
import (
20+
"flag"
21+
"os"
22+
23+
// Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.)
24+
// to ensure that exec-entrypoint and run can make use of them.
25+
"k8s.io/client-go/kubernetes/scheme"
26+
_ "k8s.io/client-go/plugin/pkg/client/auth"
27+
28+
ctrl "sigs.k8s.io/controller-runtime"
29+
ctrllog "sigs.k8s.io/controller-runtime/pkg/log"
30+
"sigs.k8s.io/controller-runtime/pkg/log/zap"
31+
"sigs.k8s.io/controller-runtime/pkg/manager"
32+
33+
// Import your controllers here <--------------------------------
34+
"sigs.k8s.io/multicluster-runtime/examples/kubeconfig/controllers"
35+
36+
mcmanager "sigs.k8s.io/multicluster-runtime/pkg/manager"
37+
kubeconfigprovider "sigs.k8s.io/multicluster-runtime/providers/kubeconfig"
38+
)
39+
40+
func main() {
41+
var namespace string
42+
var kubeconfigSecretLabel string
43+
var kubeconfigSecretKey string
44+
45+
flag.StringVar(&namespace, "namespace", "default", "Namespace where kubeconfig secrets are stored")
46+
flag.StringVar(&kubeconfigSecretLabel, "kubeconfig-label", "sigs.k8s.io/multicluster-runtime-kubeconfig",
47+
"Label used to identify secrets containing kubeconfig data")
48+
flag.StringVar(&kubeconfigSecretKey, "kubeconfig-key", "kubeconfig", "Key in the secret data that contains the kubeconfig")
49+
opts := zap.Options{
50+
Development: true,
51+
}
52+
opts.BindFlags(flag.CommandLine)
53+
flag.Parse()
54+
55+
ctrllog.SetLogger(zap.New(zap.UseFlagOptions(&opts)))
56+
entryLog := ctrllog.Log.WithName("entrypoint")
57+
ctx := ctrl.SetupSignalHandler()
58+
59+
entryLog.Info("Starting application", "namespace", namespace, "kubeconfigSecretLabel", kubeconfigSecretLabel)
60+
61+
// Create the kubeconfig provider with options
62+
providerOpts := kubeconfigprovider.Options{
63+
Namespace: namespace,
64+
KubeconfigSecretLabel: kubeconfigSecretLabel,
65+
KubeconfigSecretKey: kubeconfigSecretKey,
66+
}
67+
68+
// Create the provider first, then the manager with the provider
69+
entryLog.Info("Creating provider")
70+
provider := kubeconfigprovider.New(providerOpts)
71+
72+
// Create the multicluster manager with the provider
73+
entryLog.Info("Creating manager")
74+
75+
// Modify manager options to avoid waiting for cache sync
76+
managerOpts := manager.Options{
77+
// Don't block main thread on leader election
78+
LeaderElection: false,
79+
// Add the scheme
80+
Scheme: scheme.Scheme,
81+
}
82+
83+
mgr, err := mcmanager.New(ctrl.GetConfigOrDie(), provider, managerOpts)
84+
if err != nil {
85+
entryLog.Error(err, "Unable to create manager")
86+
os.Exit(1)
87+
}
88+
89+
// Add our controllers
90+
entryLog.Info("Adding controllers")
91+
92+
// Run your controllers here <--------------------------------
93+
podWatcher := controllers.NewPodWatcher(mgr)
94+
if err := mgr.Add(podWatcher); err != nil {
95+
entryLog.Error(err, "Unable to add pod watcher")
96+
os.Exit(1)
97+
}
98+
99+
// Start provider in a goroutine
100+
entryLog.Info("Starting provider")
101+
go func() {
102+
err := provider.Run(ctx, mgr)
103+
if err != nil && ctx.Err() == nil {
104+
entryLog.Error(err, "Provider exited with error")
105+
}
106+
}()
107+
108+
// Start the manager
109+
entryLog.Info("Starting manager")
110+
if err := mgr.Start(ctx); err != nil {
111+
entryLog.Error(err, "Error running manager")
112+
os.Exit(1)
113+
}
114+
}

0 commit comments

Comments
 (0)