Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
f942914
Initial attempt, reviewed by corentone
christensenjairus Mar 25, 2025
0a9c619
Super simplify - reviewed by stts and embik
christensenjairus Mar 25, 2025
b21016e
Simplify provider and main.go further, making them look like other pr…
christensenjairus Mar 27, 2025
3e1c90f
Merge branch 'main' of github.com:christensenjairus/multicluster-runt…
FourFifthsCode Jun 5, 2025
e72b5a8
feat: use configmap example in kubeconfig provider, cleanup unused op…
FourFifthsCode Jun 5, 2025
1bc0776
test: add tests for kubeconfig provider
FourFifthsCode Jun 6, 2025
d1481a0
feat: add basic hash to kubeconfig to prevent re-engaging existing cl…
FourFifthsCode Jun 6, 2025
389fe13
fix: move unlock after read
FourFifthsCode Jun 9, 2025
f40b8f6
fix: guard against some race conditions and add a basic race test
FourFifthsCode Jun 9, 2025
558cd9f
docs: remove pod watcher controller and use builder as example
FourFifthsCode Jun 9, 2025
9759a8b
Merge branch 'kubernetes-sigs:main' into kubeconfig-based-provider
FourFifthsCode Jun 9, 2025
c976645
chore: add go.mod for kubeconfig provider and example
FourFifthsCode Jun 9, 2025
9729c81
Update kubeconfig provider to log options and remove unnecessary secr…
FourFifthsCode Jun 16, 2025
0595ad0
Updated kubeconfig provider to use controller instead of informer for…
FourFifthsCode Jun 16, 2025
b545226
Refactor kubeconfig provider to separate logic in controller into sma…
FourFifthsCode Jun 17, 2025
49611e9
Update kubeconfig provider to return multicluster.ErrClusterNotFound …
FourFifthsCode Jun 20, 2025
eb889d6
Update kubeconfig provider script and documentation for rbac and serv…
FourFifthsCode Jun 20, 2025
6d4cb6f
Remove unnecessary error path and function for parsing kubeconfig fro…
FourFifthsCode Jun 30, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
178 changes: 178 additions & 0 deletions examples/kubeconfig/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
# Kubeconfig Provider Example

This example demonstrates how to use the kubeconfig provider to manage multiple Kubernetes clusters using kubeconfig secrets.

## Overview

The kubeconfig provider allows you to:
1. Discover and connect to multiple Kubernetes clusters using kubeconfig secrets
2. Run controllers that can operate across all discovered clusters
3. Manage cluster access through RBAC rules and service accounts

## Directory Structure

```
examples/kubeconfig/
├── scripts/ # Utility scripts
│ ├── create-kubeconfig-secret.sh
│ └── rules.yaml # Default RBAC rules template
└── main.go # Example operator implementation
```

## Usage

### 1. Setting Up Cluster Access

Before creating a kubeconfig secret, ensure that:
1. The remote cluster has a service account with the necessary RBAC permissions for your operator
2. The service account exists in the namespace where you want to create the kubeconfig secret

Use the `create-kubeconfig-secret.sh` script to create a kubeconfig secret for each cluster you want to manage:

```bash
./scripts/create-kubeconfig-secret.sh \
--name cluster1 \
-n default \
-c prod-cluster \
-a my-service-account
```

The script will:
- Use the specified service account from the remote cluster
- Generate a kubeconfig using the service account's token
- Store the kubeconfig in a secret in your local cluster
- Automatically create RBAC resources (Role/ClusterRole and bindings) with permissions defined in `rules.yaml`

#### Command Line Options

- `-c, --context`: Kubeconfig context to use (required)
- `--name`: Name for the secret (defaults to context name)
- `-n, --namespace`: Namespace to create the secret in (default: "default")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we really want to use default namespace for credentials?
maybe make it required?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think default namespace is ok, but I did change the default service account to use a different name, don't think we should use default for that.

- `-a, --service-account`: Service account name to use from the remote cluster (default: "multicluster-kubeconfig-provider")
- `-t, --role-type`: Create Role or ClusterRole (`role`|`clusterrole`) (default: "clusterrole")
- `-r, --rules-file`: Path to custom rules file (default: `rules.yaml` in script directory)
- `--skip-create-rbac`: Skip creating RBAC resources (Role/ClusterRole and bindings)
- `-h, --help`: Show help message

#### Examples

```bash
# Basic usage with default settings
./scripts/create-kubeconfig-secret.sh -c prod-cluster

# Create namespace-scoped Role instead of ClusterRole
./scripts/create-kubeconfig-secret.sh -c prod-cluster -t role

# Use custom RBAC rules file
./scripts/create-kubeconfig-secret.sh -c prod-cluster -r ./custom-rules.yaml

# Skip RBAC creation (manual RBAC setup)
./scripts/create-kubeconfig-secret.sh -c prod-cluster --skip-create-rbac

# Full example with all options
./scripts/create-kubeconfig-secret.sh \
--name my-cluster \
-n my-namespace \
-c prod-cluster \
-a my-service-account \
-t clusterrole \
-r ./my-rules.yaml
```

### 2. RBAC Configuration

The script automatically creates RBAC resources with the necessary permissions for your operator. By default, it uses the rules defined in `scripts/rules.yaml`:

```yaml
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["list", "get", "watch"]
```

#### Customizing RBAC Rules

You can customize the RBAC permissions by:

1. **Editing the default rules file** (`scripts/rules.yaml`):
```yaml
rules:
- apiGroups: [""]
resources: ["configmaps", "secrets", "pods"]
verbs: ["list", "get", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["list", "get", "watch"]
```

2. **Using a custom rules file** with the `-r` option:
```bash
./scripts/create-kubeconfig-secret.sh -c prod-cluster -r ./my-custom-rules.yaml
```

3. **Choosing between Role and ClusterRole**:
- Use `-t role` for namespace-scoped permissions
- Use `-t clusterrole` (default) for cluster-wide permissions

#### RBAC Resource Creation

The script creates the following RBAC resources automatically:

- **Service Account**: If it doesn't exist, creates the specified service account
- **Role/ClusterRole**: With the permissions defined in the rules file
- **RoleBinding/ClusterRoleBinding**: Binds the service account to the role

#### Skipping RBAC Creation

If you prefer to manage RBAC manually, use the `--skip-create-rbac` flag:

```bash
./scripts/create-kubeconfig-secret.sh -c prod-cluster --skip-create-rbac
```

This will only create the kubeconfig secret without setting up any RBAC resources.

### 3. Implementing Your Operator

Add your controllers to `main.go`:

```go
func main() {
err = mcbuilder.ControllerManagedBy(mgr).
Named("multicluster-configmaps").
For(&corev1.ConfigMap{}). // object to watch
Complete(mcreconcile.Func(
func(ctx context.Context, req mcreconcile.Request) (ctrl.Result, error) {
// reconcile logic

return ctrl.Result{}, nil
},
))
}
```

Your controllers can then use the manager to access any cluster and view the resources that the RBAC permissions allow.

## How It Works

1. The kubeconfig provider watches for secrets with a specific label in a namespace

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't it always be a secret within the same namespace as the controller?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Namespace if configurable on the provider

2. When a new secret is found, it:
- Extracts the kubeconfig data
- Creates a new controller-runtime cluster
- Makes the cluster available to your controllers
3. Your controllers can access any cluster through the manager
4. RBAC Rules on the remote clusters ensure the SA operator on the cluster of the controller has the necessary permissions in the remote clusters

## Labels and Configuration

The provider uses the following labels and keys by default:
- Label: `sigs.k8s.io/multicluster-runtime-kubeconfig: "true"`
- Secret data key: `kubeconfig`

You can customize these in the provider options when creating it.

## Prerequisites

- `kubectl` configured with access to both the local and remote clusters
- `yq` command-line tool installed (required for RBAC rule processing)
- Service account with appropriate permissions in the remote cluster (if not using automatic RBAC creation script)
71 changes: 71 additions & 0 deletions examples/kubeconfig/go.mod
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
module sigs.k8s.io/multicluster-runtime/examples/kubeconfig

go 1.24.0

replace (
sigs.k8s.io/multicluster-runtime => ../..
sigs.k8s.io/multicluster-runtime/providers/kubeconfig => ../../providers/kubeconfig
)

require (
golang.org/x/sync v0.15.0
k8s.io/api v0.33.1
k8s.io/apimachinery v0.33.1
sigs.k8s.io/controller-runtime v0.21.0
sigs.k8s.io/multicluster-runtime v0.0.0-00010101000000-000000000000
)

require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
github.com/evanphx/json-patch/v5 v5.9.11 // indirect
github.com/fsnotify/fsnotify v1.7.0 // indirect
github.com/fxamacker/cbor/v2 v2.7.0 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/zapr v1.3.0 // indirect
github.com/go-openapi/jsonpointer v0.21.0 // indirect
github.com/go-openapi/jsonreference v0.20.2 // indirect
github.com/go-openapi/swag v0.23.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/google/btree v1.1.3 // indirect
github.com/google/gnostic-models v0.6.9 // indirect
github.com/google/go-cmp v0.7.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/prometheus/client_golang v1.22.0 // indirect
github.com/prometheus/client_model v0.6.1 // indirect
github.com/prometheus/common v0.62.0 // indirect
github.com/prometheus/procfs v0.15.1 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/x448/float16 v0.8.4 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.27.0 // indirect
golang.org/x/net v0.38.0 // indirect
golang.org/x/oauth2 v0.27.0 // indirect
golang.org/x/sys v0.32.0 // indirect
golang.org/x/term v0.30.0 // indirect
golang.org/x/text v0.23.0 // indirect
golang.org/x/time v0.9.0 // indirect
gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect
google.golang.org/protobuf v1.36.5 // indirect
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
k8s.io/apiextensions-apiserver v0.33.0 // indirect
k8s.io/client-go v0.33.1 // indirect
k8s.io/klog/v2 v2.130.1 // indirect
k8s.io/kube-openapi v0.0.0-20250318190949-c8a335a9a2ff // indirect
k8s.io/utils v0.0.0-20241104100929-3ea5e8cea738 // indirect
sigs.k8s.io/json v0.0.0-20241010143419-9aa6b5e7a4b3 // indirect
sigs.k8s.io/randfill v1.0.0 // indirect
sigs.k8s.io/structured-merge-diff/v4 v4.6.0 // indirect
sigs.k8s.io/yaml v1.4.0 // indirect
)
Loading