Skip to content
This repository was archived by the owner on Sep 6, 2022. It is now read-only.

Commit 7eeebb1

Browse files
Simon Emmsmrsimonemms
authored andcommitted
add installer support for external db, storage, container
1 parent 1b6ce26 commit 7eeebb1

File tree

3 files changed

+129
-98
lines changed

3 files changed

+129
-98
lines changed

README.md

Lines changed: 37 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -47,22 +47,21 @@ The whole process takes around twenty minutes. In the end, the following resourc
4747
NAME READY STATUS RESTARTS AGE
4848
proxy-5998488f4c-t8vkh 0/1 Init 0/1 0 5m
4949
```
50+
51+
The most likely reason is because the [DNS01 challenge](https://cert-manager.io/docs/configuration/acme/dns01/) has yet to resolve. If using `SETUP_MANAGED_DNS`, you will need to update your DNS records to point to the GCP Cloud DNS nameserver.
5052

51-
One of the reason could be related to the [DNS01 challenge](https://cert-manager.io/docs/configuration/acme/dns01/) validation for the wildcard certificates can take several minutes (*DNS propagation*).
52-
So, once the Certificate is `Ready`, maybe it will be needed to restart the deployments.
53-
The DNS01 challenge needs create a TXT record on the domain name. If the domain name is not in the `PROJECT_NAME`, this will need to be done manually. The `SETUP_MANAGED_DNS` should be set to `false`.
53+
Once the DNS record has been updated, you will need to delete all Cert Manager pods to retrigger the certificate request
5454

5555
```shell
56-
❯ kubectl get certificate
57-
NAME READY SECRET AGE
58-
proxy-config-certificates True proxy-config-certificates 5m
56+
❯ kubectl delete pods -n cert-manager --all
5957
```
60-
61-
running the commands:
58+
59+
After a few minutes, you should see the `https-certificate` become ready.
6260

6361
```shell
64-
kubectl rollout restart deployment/server
65-
kubectl rollout restart deployment/ws-proxy
62+
❯ kubectl get certificate
63+
NAME READY SECRET AGE
64+
https-certificates True https-certificates 5m
6665
```
6766

6867
## Verify the installation
@@ -71,21 +70,33 @@ First, check that Gitpod components are running.
7170

7271
```shell
7372
kubectl get pods
74-
NAME READY STATUS RESTARTS AGE
75-
blobserve-6bdb9c7f89-lvhxd 2/2 Running 0 6m17s
76-
content-service-59bd58bc4d-xgv48 1/1 Running 0 6m17s
77-
dashboard-6ffdf8984-b6f7j 1/1 Running 0 6m17s
78-
image-builder-5df5694848-wsdvk 3/3 Running 0 6m16s
79-
jaeger-8679bf6676-zz57m 1/1 Running 0 4h28m
80-
messagebus-0 1/1 Running 0 4h11m
81-
proxy-56c4cdd799-bbfbx 1/1 Running 0 5m33s
82-
registry-6b75f99844-bhhqd 1/1 Running 0 4h11m
83-
registry-facade-f7twj 2/2 Running 0 6m12s
84-
server-64f9cf6b9b-bllgg 2/2 Running 0 6m16s
85-
ws-daemon-bh6h6 2/2 Running 0 2m47s
86-
ws-manager-5d57746845-t74n5 2/2 Running 0 6m16s
87-
ws-manager-bridge-79f7fcb5-7w4p5 1/1 Running 0 6m16s
88-
ws-proxy-7fc9665-rchr9 1/1 Running 0 5m57s
73+
NAME READY STATUS RESTARTS AGE
74+
agent-smith-bz97r 2/2 Running 0 95m
75+
agent-smith-dll6b 2/2 Running 0 95m
76+
agent-smith-kvrs5 2/2 Running 0 95m
77+
blobserve-74599b4b98-5t9dq 2/2 Running 0 95m
78+
cloudsqlproxy-cloud-sql-proxy-7556c57c4d-zqptx 1/1 Running 0 95m
79+
cloudsqlproxy-session-ldl9h 0/1 Completed 0 95m
80+
content-service-758878c6c-b4sqh 1/1 Running 0 95m
81+
dashboard-758f94ccf5-qk8cm 1/1 Running 0 95m
82+
image-builder-mk3-55f948b89f-m6qbk 2/2 Running 0 95m
83+
jaeger-operator-6cc9f79cc8-dfkjh 1/1 Running 0 95m
84+
messagebus-0 1/1 Running 0 95m
85+
migrations-5jmkx 0/1 Completed 0 95m
86+
openvsx-proxy-0 1/1 Running 0 95m
87+
proxy-55665d8765-n2b2w 2/2 Running 0 95m
88+
registry-facade-4c7tz 2/2 Running 0 95m
89+
registry-facade-5dxfx 2/2 Running 0 95m
90+
registry-facade-gxmhv 2/2 Running 0 95m
91+
server-ccb459f85-8vgfn 2/2 Running 0 95m
92+
ws-daemon-6qrjf 2/2 Running 0 95m
93+
ws-daemon-hslz7 2/2 Running 0 95m
94+
ws-daemon-hzw8h 2/2 Running 0 95m
95+
ws-manager-8f6bc54f4-r67kk 1/1 Running 0 95m
96+
ws-manager-bridge-56d7978664-6l6ht 2/2 Running 0 95m
97+
ws-proxy-fbc47486d-kqdvj 1/1 Running 0 95m
98+
ws-scheduler-5c5d9f998-6zxmz 2/2 Running 0 95m
99+
89100
```
90101

91102
### Test Gitpod workspaces
@@ -98,7 +109,7 @@ Load balancer IP address: XXX.XXX.XXX.XXX
98109
```
99110

100111
Please open the URL `https://<domain>/workspaces`.
101-
It should display the gitpod login page similar to the next image.
112+
It should display the Gitpod login page similar to the next image.
102113

103114
*DNS propagation* can take several minutes.
104115

charts/assets/issuer.yaml

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,6 @@ apiVersion: cert-manager.io/v1
33
kind: Issuer
44
metadata:
55
name: gitpod-issuer
6-
namespace: default
76
spec:
87
acme:
98
email: $LETSENCRYPT_EMAIL
@@ -15,16 +14,15 @@ spec:
1514
cloudDNS:
1615
project: $PROJECT_NAME
1716
serviceAccountSecretRef:
18-
name: clouddns-dns01-solver-svc-acct
17+
name: $CLOUD_DNS_SECRET
1918
key: key.json
2019
---
2120
apiVersion: cert-manager.io/v1
2221
kind: Certificate
2322
metadata:
24-
name: https-certificates
25-
namespace: default
23+
name: $CERT_NAME
2624
spec:
27-
secretName: https-certificates
25+
secretName: $CERT_NAME
2826
issuerRef:
2927
name: gitpod-issuer
3028
kind: Issuer

setup.sh

Lines changed: 89 additions & 67 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,15 @@ DNS_SA_EMAIL="${DNS_SA}"@"${PROJECT_NAME}".iam.gserviceaccount.com
2929
# Name of the node-pools for Gitpod services and workspaces
3030
SERVICES_POOL="workload-services"
3131
WORKSPACES_POOL="workload-workspaces"
32+
# Secrets
33+
SECRET_DATABASE="gcp-sql-token"
34+
SECRET_REGISTRY="gcp-registry-token"
35+
SECRET_STORAGE="gcp-storage-token"
3236

33-
GITPOD_VERSION=${GITPOD_VERSION:="aledbf-mk3.68"}
37+
REGISTRY_URL="gcr.io/${PROJECT_NAME}/gitpod"
38+
MYSQL_GITPOD_USERNAME="gitpod"
39+
MYSQL_GITPOD_ENCRYPTION_KEY='[{"name":"general","version":1,"primary":true,"material":"4uGh1q8y2DYryJwrVMHs0kWXJlqvHWWt/KJuNi04edI="}]'
40+
CERT_NAME="https-certificates"
3441

3542
function check_prerequisites() {
3643
if [ -z "${PROJECT_NAME}" ]; then
@@ -86,12 +93,39 @@ function create_node_pool() {
8693
"${PREEMPTIBLE}"
8794
}
8895

96+
function create_secrets() {
97+
# Assume that these values can change so create each run time
98+
99+
echo "Create database secret..."
100+
kubectl create secret generic "${SECRET_DATABASE}" \
101+
--from-literal=credentials.json="$(cat ./mysql-credentials.json)" \
102+
--from-literal=encryptionKeys="${MYSQL_GITPOD_ENCRYPTION_KEY}" \
103+
--from-literal=password="${MYSQL_GITPOD_PASSWORD}" \
104+
--from-literal=username="${MYSQL_GITPOD_USERNAME}" \
105+
--dry-run=client -o yaml | \
106+
kubectl replace --force -f -
107+
108+
echo "Create registry secret..."
109+
kubectl create secret docker-registry "${SECRET_REGISTRY}" \
110+
--docker-server="${REGISTRY_URL}" \
111+
--docker-username=_json_key \
112+
--docker-password="$(cat gs-credentials.json)" \
113+
--dry-run=client -o yaml | \
114+
kubectl replace --force -f -
115+
116+
echo "Create storage secret..."
117+
kubectl create secret generic "${SECRET_STORAGE}" \
118+
--from-file=service-account.json=./gs-credentials.json \
119+
--dry-run=client -o yaml | \
120+
kubectl replace --force -f -
121+
}
122+
89123
function setup_mysql_database() {
90124
if [ "$(gcloud sql instances list --filter="name:${MYSQL_INSTANCE_NAME}" --format="value(name)" | grep "${MYSQL_INSTANCE_NAME}" || echo "empty")" == "${MYSQL_INSTANCE_NAME}" ]; then
91125
echo "Cloud SQL (MySQL) Instance already exists."
92126
else
93127
# https://cloud.google.com/sql/docs/mysql/create-instance
94-
echo "Creating Mysql instance..."
128+
echo "Creating MySQL instance..."
95129
gcloud sql instances create "${MYSQL_INSTANCE_NAME}" \
96130
--database-version=MYSQL_5_7 \
97131
--storage-size=20 \
@@ -104,14 +138,14 @@ function setup_mysql_database() {
104138
gcloud sql instances patch "${MYSQL_INSTANCE_NAME}" --database-flags \
105139
explicit_defaults_for_timestamp=off --quiet
106140

107-
echo "Creating gitpod Mysql database..."
141+
echo "Creating Gitpod MySQL database..."
108142
gcloud sql databases create gitpod --instance="${MYSQL_INSTANCE_NAME}"
109143
fi
110144

111-
echo "Creating gitpod Mysql user and setting a password..."
145+
echo "Creating Gitpod MySQL user and setting a password..."
112146
MYSQL_GITPOD_PASSWORD=$(openssl rand -base64 20)
113147
export MYSQL_GITPOD_PASSWORD
114-
gcloud sql users create gitpod \
148+
gcloud sql users create "${MYSQL_GITPOD_USERNAME}" \
115149
--instance="${MYSQL_INSTANCE_NAME}" --password="${MYSQL_GITPOD_PASSWORD}"
116150
}
117151

@@ -130,27 +164,6 @@ function create_service_account() {
130164
fi
131165
}
132166

133-
function create_namespace() {
134-
local NAMESPACE=$1
135-
if ! kubectl get namespace "${NAMESPACE}" >/dev/null 2>&1;then
136-
kubectl create namespace "${NAMESPACE}"
137-
fi
138-
}
139-
140-
function install_jaeger_operator(){
141-
echo "Installing Jaeger operator..."
142-
create_namespace jaeger-operator
143-
kubectl apply -f https://raw.githubusercontent.com/jaegertracing/helm-charts/main/charts/jaeger-operator/crds/crd.yaml
144-
helm upgrade --install --namespace jaeger-operator \
145-
jaegeroperator jaegertracing/jaeger-operator \
146-
--set crd.install=false \
147-
-f "${DIR}/charts/assets/jaeger-values.yaml"
148-
149-
kubectl wait --for=condition=available --timeout=300s \
150-
deployment/jaegeroperator-jaeger-operator -n jaeger-operator
151-
kubectl apply -f "${DIR}/charts/assets/jaeger-gitpod.yaml"
152-
}
153-
154167
function setup_managed_dns() {
155168
if [ -n "${SETUP_MANAGED_DNS}" ] && [ "${SETUP_MANAGED_DNS}" == "true" ]; then
156169
if [ "$(gcloud iam service-accounts list --filter="displayName:${DNS_SA}" --format="value(displayName)" | grep "${DNS_SA}" || echo "empty")" == "${DNS_SA}" ]; then
@@ -173,20 +186,28 @@ function setup_managed_dns() {
173186
fi
174187

175188
echo "Installing external-dns..."
176-
create_namespace external-dns
177-
helm upgrade --install external-dns \
189+
helm upgrade \
190+
--atomic \
191+
--cleanup-on-fail \
192+
--create-namespace \
193+
--install \
178194
--namespace external-dns \
179-
bitnami/external-dns \
195+
--reset-values \
180196
--set provider=google \
181197
--set google.project="${PROJECT_NAME}" \
182198
--set logFormat=json \
183-
--set google.serviceAccountSecretKey=dns-credentials.json
199+
--set google.serviceAccountSecretKey=dns-credentials.json \
200+
--wait \
201+
external-dns \
202+
bitnami/external-dns
184203

185-
if ! kubectl get secret --namespace cert-manager clouddns-dns01-solver-svc-acct; then
186-
echo "Creating secret for Cloud DNS Issuer..."
187-
kubectl create secret generic clouddns-dns01-solver-svc-acct \
188-
--namespace cert-manager --from-file=key.json="${DIR}/dns-credentials.json"
189-
fi
204+
echo "Creating secret for Cloud DNS Issuer..."
205+
export CLOUD_DNS_SECRET=clouddns-dns01-solver
206+
207+
kubectl create secret generic "${CLOUD_DNS_SECRET}" \
208+
--from-file=key.json="${DIR}/dns-credentials.json" \
209+
--dry-run=client -o yaml | \
210+
kubectl replace --force -f -
190211

191212
echo "Installing cert-manager certificate issuer..."
192213
envsubst < "${DIR}/charts/assets/issuer.yaml" | kubectl apply -f -
@@ -195,16 +216,18 @@ function setup_managed_dns() {
195216

196217
function install_cert_manager() {
197218
echo "Installing cert-manager..."
198-
helm upgrade cert-manager jetstack/cert-manager \
199-
--namespace='cert-manager' \
200-
--install \
219+
helm upgrade \
220+
--atomic \
221+
--cleanup-on-fail \
201222
--create-namespace \
223+
--install \
224+
--namespace cert-manager \
225+
--reset-values \
202226
--set installCRDs=true \
203227
--set 'extraArgs={--dns01-recursive-nameservers-only=true,--dns01-recursive-nameservers=8.8.8.8:53\,1.1.1.1:53}' \
204-
--atomic
205-
206-
# ensure cert-manager and CRDs are installed and running
207-
kubectl wait --for=condition=available --timeout=300s deployment/cert-manager -n cert-manager
228+
--wait \
229+
cert-manager \
230+
jetstack/cert-manager
208231
}
209232

210233
function install_gitpod() {
@@ -214,8 +237,22 @@ function install_gitpod() {
214237

215238
gitpod-installer init > "${CONFIG_FILE}"
216239

240+
echo "Updating config..."
241+
yq e -i ".certificate.name = \"${CERT_NAME}\"" "${CONFIG_FILE}"
242+
yq e -i ".containerRegistry.inCluster = false" "${CONFIG_FILE}"
243+
yq e -i ".containerRegistry.external.url = \"${REGISTRY_URL}\"" "${CONFIG_FILE}"
244+
yq e -i ".containerRegistry.external.certificate.kind = \"secret\"" "${CONFIG_FILE}"
245+
yq e -i ".containerRegistry.external.certificate.name = \"${SECRET_REGISTRY}\"" "${CONFIG_FILE}"
246+
yq e -i ".database.inCluster = false" "${CONFIG_FILE}"
247+
yq e -i ".database.cloudSQL.instance = \"${PROJECT_NAME}:${REGION}:${MYSQL_INSTANCE_NAME}\"" "${CONFIG_FILE}"
248+
yq e -i ".database.cloudSQL.serviceAccount.kind = \"secret\"" "${CONFIG_FILE}"
249+
yq e -i ".database.cloudSQL.serviceAccount.name = \"${SECRET_DATABASE}\"" "${CONFIG_FILE}"
217250
yq e -i ".domain = \"${DOMAIN}\"" "${CONFIG_FILE}"
218251
yq e -i ".metadata.region = \"${REGION}\"" "${CONFIG_FILE}"
252+
yq e -i ".objectStorage.inCluster = false" "${CONFIG_FILE}"
253+
yq e -i ".objectStorage.cloudStorage.project = \"${PROJECT_NAME}\"" "${CONFIG_FILE}"
254+
yq e -i ".objectStorage.cloudStorage.serviceAccount.kind = \"secret\"" "${CONFIG_FILE}"
255+
yq e -i ".objectStorage.cloudStorage.serviceAccount.name = \"${SECRET_STORAGE}\"" "${CONFIG_FILE}"
219256
yq e -i '.workspace.runtime.containerdRuntimeDir = "/var/lib/containerd/io.containerd.runtime.v2.task/k8s.io"' "${CONFIG_FILE}"
220257

221258
gitpod-installer \
@@ -234,25 +271,6 @@ function service_account_exists() {
234271
fi
235272
}
236273

237-
function wait_for_load_balancer() {
238-
sleep 10
239-
240-
COUNT=0
241-
LB_IP_ADDRESS=""
242-
while [ "${LB_IP_ADDRESS}" == "" ] && [ "${COUNT}" -lt 5 ];do
243-
printf "."
244-
LB_IP_ADDRESS=$(kubectl get service proxy -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')
245-
((COUNT+=1))
246-
sleep 5
247-
done
248-
249-
if [ -n "${LB_IP_ADDRESS}" ];then
250-
printf '\nLoad balancer IP address: %s\n' "${LB_IP_ADDRESS}"
251-
else
252-
printf '\n The load balancer is still being provisioned. Wait a couple of minutes.'
253-
fi
254-
}
255-
256274
function install() {
257275
echo "Gitpod installer version: $(gitpod-installer version | jq -r '.version')"
258276

@@ -369,20 +387,24 @@ function install() {
369387
--clusterrole=cluster-admin --user="$(gcloud config get-value core/account)"
370388
fi
371389

390+
CONTAINER_REGISTRY_BUCKET="container-registry-${CLUSTER_NAME}-${PROJECT_ID}"
391+
export CONTAINER_REGISTRY_BUCKET
392+
# the bucket must exists before installing the docker-registry.
393+
if ! gsutil acl get "gs://${CONTAINER_REGISTRY_BUCKET}" >/dev/null 2>&1;then
394+
gsutil mb "gs://${CONTAINER_REGISTRY_BUCKET}"
395+
fi
396+
372397
install_cert_manager
373398
setup_managed_dns
374-
# setup_mysql_database
399+
setup_mysql_database
400+
create_secrets
375401
install_gitpod
376402

377-
wait_for_load_balancer
378-
379-
# The load balancer wait clips message - extra line solves that
380403
cat << EOF
381-
382404
==========================
383405
Gitpod is now installed on your cluster
384406
385-
Please update your DNS record with the relevant nameserver.
407+
Please update your DNS records with the relevant nameserver.
386408
EOF
387409
}
388410

0 commit comments

Comments
 (0)