-
Notifications
You must be signed in to change notification settings - Fork 38
✨ Add Kubeconfig-Based provider #45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
k8s-ci-robot
merged 18 commits into
kubernetes-sigs:main
from
FourFifthsCode:kubeconfig-based-provider
Jul 5, 2025
Merged
Changes from all commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
f942914
Initial attempt, reviewed by corentone
christensenjairus 0a9c619
Super simplify - reviewed by stts and embik
christensenjairus b21016e
Simplify provider and main.go further, making them look like other pr…
christensenjairus 3e1c90f
Merge branch 'main' of github.com:christensenjairus/multicluster-runt…
FourFifthsCode e72b5a8
feat: use configmap example in kubeconfig provider, cleanup unused op…
FourFifthsCode 1bc0776
test: add tests for kubeconfig provider
FourFifthsCode d1481a0
feat: add basic hash to kubeconfig to prevent re-engaging existing cl…
FourFifthsCode 389fe13
fix: move unlock after read
FourFifthsCode f40b8f6
fix: guard against some race conditions and add a basic race test
FourFifthsCode 558cd9f
docs: remove pod watcher controller and use builder as example
FourFifthsCode 9759a8b
Merge branch 'kubernetes-sigs:main' into kubeconfig-based-provider
FourFifthsCode c976645
chore: add go.mod for kubeconfig provider and example
FourFifthsCode 9729c81
Update kubeconfig provider to log options and remove unnecessary secr…
FourFifthsCode 0595ad0
Updated kubeconfig provider to use controller instead of informer for…
FourFifthsCode b545226
Refactor kubeconfig provider to separate logic in controller into sma…
FourFifthsCode 49611e9
Update kubeconfig provider to return multicluster.ErrClusterNotFound …
FourFifthsCode eb889d6
Update kubeconfig provider script and documentation for rbac and serv…
FourFifthsCode 6d4cb6f
Remove unnecessary error path and function for parsing kubeconfig fro…
FourFifthsCode File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,178 @@ | ||
| # Kubeconfig Provider Example | ||
|
|
||
| This example demonstrates how to use the kubeconfig provider to manage multiple Kubernetes clusters using kubeconfig secrets. | ||
|
|
||
| ## Overview | ||
|
|
||
| The kubeconfig provider allows you to: | ||
| 1. Discover and connect to multiple Kubernetes clusters using kubeconfig secrets | ||
| 2. Run controllers that can operate across all discovered clusters | ||
| 3. Manage cluster access through RBAC rules and service accounts | ||
|
|
||
| ## Directory Structure | ||
|
|
||
| ``` | ||
| examples/kubeconfig/ | ||
| ├── scripts/ # Utility scripts | ||
| │ ├── create-kubeconfig-secret.sh | ||
| │ └── rules.yaml # Default RBAC rules template | ||
| └── main.go # Example operator implementation | ||
| ``` | ||
|
|
||
| ## Usage | ||
|
|
||
| ### 1. Setting Up Cluster Access | ||
|
|
||
| Before creating a kubeconfig secret, ensure that: | ||
| 1. The remote cluster has a service account with the necessary RBAC permissions for your operator | ||
| 2. The service account exists in the namespace where you want to create the kubeconfig secret | ||
|
|
||
| Use the `create-kubeconfig-secret.sh` script to create a kubeconfig secret for each cluster you want to manage: | ||
|
|
||
| ```bash | ||
| ./scripts/create-kubeconfig-secret.sh \ | ||
| --name cluster1 \ | ||
| -n default \ | ||
| -c prod-cluster \ | ||
| -a my-service-account | ||
| ``` | ||
|
|
||
| The script will: | ||
| - Use the specified service account from the remote cluster | ||
| - Generate a kubeconfig using the service account's token | ||
| - Store the kubeconfig in a secret in your local cluster | ||
| - Automatically create RBAC resources (Role/ClusterRole and bindings) with permissions defined in `rules.yaml` | ||
|
|
||
| #### Command Line Options | ||
|
|
||
| - `-c, --context`: Kubeconfig context to use (required) | ||
| - `--name`: Name for the secret (defaults to context name) | ||
| - `-n, --namespace`: Namespace to create the secret in (default: "default") | ||
| - `-a, --service-account`: Service account name to use from the remote cluster (default: "multicluster-kubeconfig-provider") | ||
| - `-t, --role-type`: Create Role or ClusterRole (`role`|`clusterrole`) (default: "clusterrole") | ||
| - `-r, --rules-file`: Path to custom rules file (default: `rules.yaml` in script directory) | ||
| - `--skip-create-rbac`: Skip creating RBAC resources (Role/ClusterRole and bindings) | ||
| - `-h, --help`: Show help message | ||
|
|
||
| #### Examples | ||
|
|
||
| ```bash | ||
| # Basic usage with default settings | ||
| ./scripts/create-kubeconfig-secret.sh -c prod-cluster | ||
|
|
||
| # Create namespace-scoped Role instead of ClusterRole | ||
| ./scripts/create-kubeconfig-secret.sh -c prod-cluster -t role | ||
|
|
||
| # Use custom RBAC rules file | ||
| ./scripts/create-kubeconfig-secret.sh -c prod-cluster -r ./custom-rules.yaml | ||
|
|
||
| # Skip RBAC creation (manual RBAC setup) | ||
| ./scripts/create-kubeconfig-secret.sh -c prod-cluster --skip-create-rbac | ||
|
|
||
| # Full example with all options | ||
| ./scripts/create-kubeconfig-secret.sh \ | ||
| --name my-cluster \ | ||
| -n my-namespace \ | ||
| -c prod-cluster \ | ||
| -a my-service-account \ | ||
| -t clusterrole \ | ||
| -r ./my-rules.yaml | ||
| ``` | ||
|
|
||
| ### 2. RBAC Configuration | ||
|
|
||
| The script automatically creates RBAC resources with the necessary permissions for your operator. By default, it uses the rules defined in `scripts/rules.yaml`: | ||
|
|
||
| ```yaml | ||
| rules: | ||
| - apiGroups: [""] | ||
| resources: ["configmaps"] | ||
| verbs: ["list", "get", "watch"] | ||
| ``` | ||
|
|
||
| #### Customizing RBAC Rules | ||
|
|
||
| You can customize the RBAC permissions by: | ||
|
|
||
| 1. **Editing the default rules file** (`scripts/rules.yaml`): | ||
| ```yaml | ||
| rules: | ||
| - apiGroups: [""] | ||
| resources: ["configmaps", "secrets", "pods"] | ||
| verbs: ["list", "get", "watch", "create", "update", "patch", "delete"] | ||
| - apiGroups: ["apps"] | ||
| resources: ["deployments"] | ||
| verbs: ["list", "get", "watch"] | ||
| ``` | ||
|
|
||
| 2. **Using a custom rules file** with the `-r` option: | ||
| ```bash | ||
| ./scripts/create-kubeconfig-secret.sh -c prod-cluster -r ./my-custom-rules.yaml | ||
| ``` | ||
|
|
||
| 3. **Choosing between Role and ClusterRole**: | ||
| - Use `-t role` for namespace-scoped permissions | ||
| - Use `-t clusterrole` (default) for cluster-wide permissions | ||
|
|
||
| #### RBAC Resource Creation | ||
|
|
||
| The script creates the following RBAC resources automatically: | ||
|
|
||
| - **Service Account**: If it doesn't exist, creates the specified service account | ||
| - **Role/ClusterRole**: With the permissions defined in the rules file | ||
| - **RoleBinding/ClusterRoleBinding**: Binds the service account to the role | ||
|
|
||
| #### Skipping RBAC Creation | ||
|
|
||
| If you prefer to manage RBAC manually, use the `--skip-create-rbac` flag: | ||
|
|
||
| ```bash | ||
| ./scripts/create-kubeconfig-secret.sh -c prod-cluster --skip-create-rbac | ||
| ``` | ||
|
|
||
| This will only create the kubeconfig secret without setting up any RBAC resources. | ||
|
|
||
| ### 3. Implementing Your Operator | ||
|
|
||
| Add your controllers to `main.go`: | ||
|
|
||
| ```go | ||
| func main() { | ||
| err = mcbuilder.ControllerManagedBy(mgr). | ||
| Named("multicluster-configmaps"). | ||
| For(&corev1.ConfigMap{}). // object to watch | ||
| Complete(mcreconcile.Func( | ||
| func(ctx context.Context, req mcreconcile.Request) (ctrl.Result, error) { | ||
| // reconcile logic | ||
|
|
||
| return ctrl.Result{}, nil | ||
| }, | ||
| )) | ||
| } | ||
| ``` | ||
|
|
||
| Your controllers can then use the manager to access any cluster and view the resources that the RBAC permissions allow. | ||
|
|
||
| ## How It Works | ||
|
|
||
| 1. The kubeconfig provider watches for secrets with a specific label in a namespace | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. shouldn't it always be a secret within the same namespace as the controller?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Namespace if configurable on the provider |
||
| 2. When a new secret is found, it: | ||
| - Extracts the kubeconfig data | ||
| - Creates a new controller-runtime cluster | ||
| - Makes the cluster available to your controllers | ||
| 3. Your controllers can access any cluster through the manager | ||
| 4. RBAC Rules on the remote clusters ensure the SA operator on the cluster of the controller has the necessary permissions in the remote clusters | ||
|
|
||
| ## Labels and Configuration | ||
|
|
||
| The provider uses the following labels and keys by default: | ||
| - Label: `sigs.k8s.io/multicluster-runtime-kubeconfig: "true"` | ||
| - Secret data key: `kubeconfig` | ||
|
|
||
| You can customize these in the provider options when creating it. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - `kubectl` configured with access to both the local and remote clusters | ||
| - `yq` command-line tool installed (required for RBAC rule processing) | ||
| - Service account with appropriate permissions in the remote cluster (if not using automatic RBAC creation script) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| module sigs.k8s.io/multicluster-runtime/examples/kubeconfig | ||
|
|
||
| go 1.24.0 | ||
|
|
||
| replace ( | ||
| sigs.k8s.io/multicluster-runtime => ../.. | ||
| sigs.k8s.io/multicluster-runtime/providers/kubeconfig => ../../providers/kubeconfig | ||
| ) | ||
|
|
||
| require ( | ||
| golang.org/x/sync v0.15.0 | ||
| k8s.io/api v0.33.1 | ||
| k8s.io/apimachinery v0.33.1 | ||
| sigs.k8s.io/controller-runtime v0.21.0 | ||
| sigs.k8s.io/multicluster-runtime v0.0.0-00010101000000-000000000000 | ||
| ) | ||
|
|
||
| require ( | ||
| github.com/beorn7/perks v1.0.1 // indirect | ||
| github.com/cespare/xxhash/v2 v2.3.0 // indirect | ||
| github.com/davecgh/go-spew v1.1.1 // indirect | ||
| github.com/emicklei/go-restful/v3 v3.11.0 // indirect | ||
| github.com/evanphx/json-patch/v5 v5.9.11 // indirect | ||
| github.com/fsnotify/fsnotify v1.7.0 // indirect | ||
| github.com/fxamacker/cbor/v2 v2.7.0 // indirect | ||
| github.com/go-logr/logr v1.4.3 // indirect | ||
| github.com/go-logr/zapr v1.3.0 // indirect | ||
| github.com/go-openapi/jsonpointer v0.21.0 // indirect | ||
| github.com/go-openapi/jsonreference v0.20.2 // indirect | ||
| github.com/go-openapi/swag v0.23.0 // indirect | ||
| github.com/gogo/protobuf v1.3.2 // indirect | ||
| github.com/google/btree v1.1.3 // indirect | ||
| github.com/google/gnostic-models v0.6.9 // indirect | ||
| github.com/google/go-cmp v0.7.0 // indirect | ||
| github.com/google/uuid v1.6.0 // indirect | ||
| github.com/josharian/intern v1.0.0 // indirect | ||
| github.com/json-iterator/go v1.1.12 // indirect | ||
| github.com/mailru/easyjson v0.7.7 // indirect | ||
| github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect | ||
| github.com/modern-go/reflect2 v1.0.2 // indirect | ||
| github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect | ||
| github.com/pkg/errors v0.9.1 // indirect | ||
| github.com/prometheus/client_golang v1.22.0 // indirect | ||
| github.com/prometheus/client_model v0.6.1 // indirect | ||
| github.com/prometheus/common v0.62.0 // indirect | ||
| github.com/prometheus/procfs v0.15.1 // indirect | ||
| github.com/spf13/pflag v1.0.5 // indirect | ||
| github.com/x448/float16 v0.8.4 // indirect | ||
| go.uber.org/multierr v1.11.0 // indirect | ||
| go.uber.org/zap v1.27.0 // indirect | ||
| golang.org/x/net v0.38.0 // indirect | ||
| golang.org/x/oauth2 v0.27.0 // indirect | ||
| golang.org/x/sys v0.32.0 // indirect | ||
| golang.org/x/term v0.30.0 // indirect | ||
| golang.org/x/text v0.23.0 // indirect | ||
| golang.org/x/time v0.9.0 // indirect | ||
| gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect | ||
| google.golang.org/protobuf v1.36.5 // indirect | ||
| gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect | ||
| gopkg.in/inf.v0 v0.9.1 // indirect | ||
| gopkg.in/yaml.v3 v3.0.1 // indirect | ||
| k8s.io/apiextensions-apiserver v0.33.0 // indirect | ||
| k8s.io/client-go v0.33.1 // indirect | ||
| k8s.io/klog/v2 v2.130.1 // indirect | ||
| k8s.io/kube-openapi v0.0.0-20250318190949-c8a335a9a2ff // indirect | ||
| k8s.io/utils v0.0.0-20241104100929-3ea5e8cea738 // indirect | ||
| sigs.k8s.io/json v0.0.0-20241010143419-9aa6b5e7a4b3 // indirect | ||
| sigs.k8s.io/randfill v1.0.0 // indirect | ||
| sigs.k8s.io/structured-merge-diff/v4 v4.6.0 // indirect | ||
| sigs.k8s.io/yaml v1.4.0 // indirect | ||
| ) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we really want to use default namespace for credentials?
maybe make it required?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think default namespace is ok, but I did change the default service account to use a different name, don't think we should use default for that.