Skip to content

Commit 4d79e4e

Browse files
sergelogvinovsergelogvinov
committed
docs: install instruction
Move the copy/paste instructions to a separate document Signed-off-by: Serge Logvinov <serge.logvinov@sinextra.dev>
1 parent 5876cd4 commit 4d79e4e

File tree

3 files changed

+164
-73
lines changed

3 files changed

+164
-73
lines changed

‎README.md

Lines changed: 19 additions & 71 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@ Originally, it was designed to work with [Talos CCM](https://github.com/siderola
1212
The CCM does a few things: it initialises new nodes, applies common labels to them, and removes them when they're deleted. It also supports multiple clusters, meaning you can have one kubernetes cluster across multiple Proxmox clusters.
1313

1414
The basic definitions:
15-
* kubernetes `region` is a Proxmox cluster `clusters[].region`
16-
* kubernetes `zone` is a hypervisor host machine name
15+
* kubernetes label `topology.kubernetes.io/region` is a Proxmox cluster `clusters[].region`
16+
* kubernetes label `topology.kubernetes.io/zone` is a hypervisor host machine name
1717

1818
This makes it possible for me to use pods affinity/anti-affinity.
1919

@@ -24,8 +24,10 @@ This makes it possible for me to use pods affinity/anti-affinity.
2424
clusters:
2525
- url: https://cluster-api-1.exmple.com:8006/api2/json
2626
insecure: false
27+
# Proxox auth token
2728
token_id: "user!token-id"
2829
token_secret: "secret"
30+
# Uniq region name
2931
region: cluster-1
3032
- url: https://cluster-api-2.exmple.com:8006/api2/json
3133
insecure: false
@@ -42,12 +44,18 @@ kind: Node
4244
metadata:
4345
labels:
4446
...
47+
# Type generated base on CPU and RAM
4548
node.kubernetes.io/instance-type: 2VCPU-2GB
49+
# Proxmox cluster name as in the config
4650
topology.kubernetes.io/region: cluster-1
51+
# Proxmox hypervisor host machine name
4752
topology.kubernetes.io/zone: pve-node-1
4853
name: worker-1
4954
spec:
5055
...
56+
# providerID - magic string:
57+
# cluster-1 - cluster name as in the config
58+
# 123 - Proxmox VM ID
5159
providerID: proxmox://cluster-1/123
5260
status:
5361
addresses:
@@ -57,79 +65,19 @@ status:
5765
type: Hostname
5866
```
5967
60-
# Install
68+
## Install
6169
62-
## Create a token
70+
See [Install](docs/install.md) for installation instructions.
6371
64-
Official [documentation](https://pve.proxmox.com/wiki/User_Management)
72+
## Controllers
6573
66-
```shell
67-
# Create role CCM
68-
pveum role add CCM -privs "VM.Audit"
69-
# Create user and grant permissions
70-
pveum user add kubernetes@pve
71-
pveum aclmod / -user kubernetes@pve -role CCM
72-
pveum user token add kubernetes@pve ccm -privsep 0
73-
```
74-
75-
## Deploy CCM
76-
77-
Create the proxmox credentials
78-
79-
```yaml
80-
# config.yaml
81-
clusters:
82-
- url: https://cluster-api-1.exmple.com:8006/api2/json
83-
insecure: false
84-
token_id: "kubernetes@pve!ccm"
85-
token_secret: "secret"
86-
region: cluster-1
87-
```
88-
89-
Upload it to the kubernetes:
90-
91-
```shell
92-
kubectl -n kube-system create secret generic proxmox-cloud-controller-manager --from-file=config.yaml
93-
```
94-
95-
### Method 1: kubectl
96-
97-
Deploy Proxmox CCM with `cloud-node,cloud-node-lifecycle` controllers
98-
99-
```shell
100-
kubectl apply -f https://raw.githubusercontent.com/sergelogvinov/proxmox-cloud-controller-manager/main/docs/deploy/cloud-controller-manager.yml
101-
```
102-
103-
Deploy Proxmox CCM with `cloud-node-lifecycle` controller (for Talos)
104-
105-
```shell
106-
kubectl apply -f https://raw.githubusercontent.com/sergelogvinov/proxmox-cloud-controller-manager/main/docs/deploy/cloud-controller-manager-talos.yml
107-
```
108-
109-
### Method 2: helm chart
110-
111-
Create the config file:
112-
113-
```yaml
114-
# proxmox-ccm.yaml
115-
config:
116-
clusters:
117-
- url: https://cluster-api-1.exmple.com:8006/api2/json
118-
insecure: false
119-
token_id: "kubernetes@pve!ccm"
120-
token_secret: "secret"
121-
region: cluster-1
122-
```
123-
124-
Deploy Proxmox CCM
125-
126-
```shell
127-
helm upgrade -i --namespace=kube-system -f proxmox-ccm.yaml \
128-
proxmox-cloud-controller-manager \
129-
oci://ghcr.io/sergelogvinov/charts/proxmox-cloud-controller-manager
130-
```
74+
Support controllers:
13175
132-
More options you can find [here](charts/proxmox-cloud-controller-manager)
76+
* cloud-node
77+
* Updates node resource.
78+
* Assigns labels and taints based on Proxmox VM configuration.
79+
* cloud-node-lifecycle
80+
* Cleans up node resource when Proxmox VM is deleted.
13381
13482
## Contributing
13583

‎docs/install.md

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
# Install
2+
3+
Proxmox Cloud Controller Manager (CCM) supports controllers:
4+
* cloud-node
5+
* cloud-node-lifecycle
6+
7+
`cloud-node` - detects new node launched in the cluster and registers them in the cluster.
8+
Assigns labels and taints based on Proxmox VM configuration.
9+
10+
`cloud-node-lifecycle` - detects node deletion on Proxmox side and removes them from the cluster.
11+
12+
## Create a Proxmox token
13+
14+
Official [documentation](https://pve.proxmox.com/wiki/User_Management)
15+
16+
```shell
17+
# Create role CCM
18+
pveum role add CCM -privs "VM.Audit"
19+
# Create user and grant permissions
20+
pveum user add kubernetes@pve
21+
pveum aclmod / -user kubernetes@pve -role CCM
22+
pveum user token add kubernetes@pve ccm -privsep 0
23+
```
24+
25+
## Deploy CCM
26+
27+
Create the proxmox credentials config file:
28+
29+
```yaml
30+
clusters:
31+
# List of Proxmox clusters, region mast be unique
32+
- url: https://cluster-api-1.exmple.com:8006/api2/json
33+
insecure: false
34+
token_id: "kubernetes@pve!ccm"
35+
# Token from the previous step
36+
token_secret: "secret"
37+
# Region name, can be any string, it will use as for kubernetes topology.kubernetes.io/region label
38+
region: cluster-1
39+
```
40+
41+
### Method 1: kubectl
42+
43+
Upload it to the kubernetes:
44+
45+
```shell
46+
kubectl -n kube-system create secret generic proxmox-cloud-controller-manager --from-file=config.yaml
47+
```
48+
49+
Deploy Proxmox CCM with `cloud-node,cloud-node-lifecycle` controllers
50+
51+
```shell
52+
kubectl apply -f https://raw.githubusercontent.com/sergelogvinov/proxmox-cloud-controller-manager/main/docs/deploy/cloud-controller-manager.yml
53+
```
54+
55+
Deploy Proxmox CCM with `cloud-node-lifecycle` controller (for Talos)
56+
57+
```shell
58+
kubectl apply -f https://raw.githubusercontent.com/sergelogvinov/proxmox-cloud-controller-manager/main/docs/deploy/cloud-controller-manager-talos.yml
59+
```
60+
61+
### Method 2: helm chart
62+
63+
Create the config file
64+
65+
```yaml
66+
# proxmox-ccm.yaml
67+
config:
68+
clusters:
69+
- url: https://cluster-api-1.exmple.com:8006/api2/json
70+
insecure: false
71+
token_id: "kubernetes@pve!ccm"
72+
token_secret: "secret"
73+
region: cluster-1
74+
```
75+
76+
Deploy Proxmox CCM (deployment mode)
77+
78+
```shell
79+
helm upgrade -i --namespace=kube-system -f proxmox-ccm.yaml \
80+
proxmox-cloud-controller-manager \
81+
oci://ghcr.io/sergelogvinov/charts/proxmox-cloud-controller-manager
82+
```
83+
84+
Deploy Proxmox CCM (daemonset mode)
85+
86+
It makes sense to deploy on all control-plane nodes. Do not forget to set the nodeSelector.
87+
88+
```shell
89+
helm upgrade -i --namespace=kube-system -f proxmox-ccm.yaml \
90+
--set useDaemonSet=true \
91+
proxmox-cloud-controller-manager \
92+
oci://ghcr.io/sergelogvinov/charts/proxmox-cloud-controller-manager
93+
```
94+
95+
More options you can find [here](charts/proxmox-cloud-controller-manager)
96+
97+
## Deploy CCM (Rancher)
98+
99+
Official [documentation](https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters)
100+
101+
Rancher RKE2 configuration:
102+
103+
```yaml
104+
machineGlobalConfig:
105+
# Kubelet predefined value --cloud-provider=external
106+
cloud-provider-name: external
107+
# Disable Rancher CCM
108+
disable-cloud-controller: true
109+
```
110+
111+
Create the helm values file:
112+
113+
```yaml
114+
# proxmox-ccm.yaml
115+
config:
116+
clusters:
117+
- url: https://cluster-api-1.exmple.com:8006/api2/json
118+
insecure: false
119+
token_id: "kubernetes@pve!ccm"
120+
token_secret: "secret"
121+
region: cluster-1
122+
123+
# Use host resolv.conf to resolve proxmox connection url
124+
useDaemonSet: true
125+
126+
# Set nodeSelector in daemonset mode is required
127+
nodeSelector:
128+
node-role.kubernetes.io/control-plane: ""
129+
```
130+
131+
Deploy Proxmox CCM (daemondset mode)
132+
133+
```shell
134+
helm upgrade -i --namespace=kube-system -f proxmox-ccm.yaml \
135+
proxmox-cloud-controller-manager \
136+
oci://ghcr.io/sergelogvinov/charts/proxmox-cloud-controller-manager
137+
```
138+
139+
## Deploy CCM with load balancer (optional)
140+
141+
This optional setup to improve the Proxmox API availability.
142+
143+
See [load balancer](loadbalancer.md) for installation instructions.

‎docs/loadbalancer.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,10 +41,10 @@ kubectl apply -f proxmox-service.yaml
4141

4242
Second, we need to deploy proxmox CCM with sidecar load balancer.
4343
Haproxy will resolve the `proxmox.kube-system.svc.cluster.local` service and uses IPs from the endpoints to distribute traffic.
44-
Proxmox CCM will use the `proxmox.domain.com` domain to connect to the proxmox cluster wich is resolved to the load balancer IP (127.0.0.1).
44+
Proxmox CCM will use the `proxmox.domain.com` domain to connect to the proxmox cluster which is resolved to the load balancer IP (127.0.0.1).
4545

4646
```yaml
47-
# Helm Chart values
47+
# CCM helm chart values
4848

4949
config:
5050
clusters:

0 commit comments

Comments
 (0)