-
Notifications
You must be signed in to change notification settings - Fork 10
Sample Deployment
Below is a quick writeup of a sample deployment of bb-portal in kubernetes. It assumes you have a basic working knowledge of kubernetes and kustomize. It also assumes you're connecting to an external postgres service. Using sqlite is also possible (and easier).
bbportal
├── bb_portal.jsonnet
├── deployment.yml
├── .env
├── ext.secret.yml
├── ingress.yml
├── kustomization.yml
├── pvc.yml
└── svc.yml
This file is used by kustomize to generate a configMap object which is used to configure the backend container
{
httpServers: [{
listenAddresses: [':8081'],
authenticationPolicy: { allow: {} },
}],
grpcServers: [{
listenAddresses: [':8082'],
authenticationPolicy: { allow: {} },
maximumReceivedMessageSizeBytes: 10*1024*1024
}],
}This file is used to specify the particulars of the deployment. It assumes a very simple deployment that leverages a single pod, with a single init container and both the frontend and backend containers running in the same pod. There are lots of different ways you can deploy this.
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: bb-portal
labels:
app: bb-portal
spec:
replicas: 1
revisionHistoryLimit: 3
selector:
matchLabels:
app: bb-portal
template:
metadata:
labels:
app: bb-portal
spec:
initContainers:
#init
- name: init
image: busybox
command: ["/bin/sh"]
args:
- "-c"
- mkdir -pm 0777 /data/ /data/bep-files/ /data/blob-archive/ /data/db/
volumeMounts:
- mountPath: /data
name: data
containers:
#frontend
- name: frontend
image: ghcr.io/buildbarn/bb-portal-frontend
imagePullPolicy: Always
resources:
requests:
memory: "8Gi"
cpu: "1"
limits:
memory: "16Gi"
cpu: "2"
ports:
- containerPort: 3000 #http
volumeMounts:
- mountPath: /data
name: data
- mountPath: /app/.env
subPath: .env
name: frontend-config
#backend
- name: backend
image: ghcr.io/buildbarn/bb-portal-backend
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: bb-portal-db
key: password
imagePullPolicy: Always
resources:
requests:
memory: "8Gi"
cpu: "1"
limits:
memory: "16Gi"
cpu: "2"
args:
[
"--config-file=/config/bb_portal.jsonnet",
"--datasource-driver=pgx",
"--datasource-url=postgresql://postgres:$(POSTGRES_PASSWORD)@bbportal.postgres.exampleco.io:5432/bbportal",
"--bep-folder=/data/bep-files/",
"--blob-archive-folder=/data/blob-archive/",
]
ports:
- containerPort: 8082 # grpc
- containerPort: 8081 # reverse proxy for fe
livenessProbe:
grpc:
port: 8082
initialDelaySeconds: 120
periodSeconds: 5
volumeMounts:
- mountPath: /data/
name: data
- mountPath: /config/bb_portal.jsonnet
subPath: bb_portal.jsonnet
name: backend-config
volumes:
- name: data
persistentVolumeClaim:
claimName: bb-portal
- name: backend-config
configMap:
name: bbportal-backend-config
- name: frontend-config
configMap:
name: bbportal-frontend-configThis file is used by kustomize to generate a configMap object which will be consumed by the frontend container
NEXT_PUBLIC_BES_BACKEND_URL=https://bbportal.exampleco.io
NEXT_PUBLIC_BES_GRPC_BACKEND_URL=grpcs://bbportal-grpc.exampleco.io
NEXT_PUBLIC_BROWSER_URL=https://build.exampleco.io
NEXT_PUBLIC_COMPANY_NAME="ExampleCo"
NEXT_PUBLIC_COMPANY_SLACK_CHANNEL_NAME=ExampleCoSlack
NEXT_PUBLIC_COMPANY_SLACK_CHANNEL_URL=https://exampleco.enterprise.slack.com/archives/CXXXXXXIf you leverage the external secrets operator in your cluster, this is an example of how you might create an external secret.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: bb-portal-db
annotations:
argocd.argoproj.io/sync-wave: "-2" #if you're using argocd and need the secret rolled out prior to other parts of your application
spec:
secretStoreRef:
name: my-secret-store
kind: SecretStore
target:
name: bb-portal-db
template:
engineVersion: v2
data:
password: "{{ .POSTGRES_PASSWORD }}"
data:
- secretKey: POSTGRES_PASSWORD
remoteRef:
key: "BB-Portal"
property: postgres_password
This is a sample ingress that uses the nginx ingress operator. You may need to adjust depending on your ingress solution. It also contains annotations that you may find useful with cert manager, if you have that deployed in your cluster.
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: grpc-bbportal-ingress
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
nginx.ingress.kubernetes.io/use-forwarded-headers: "true"
nginx.ingress.kubernetes.io/server-snippet: |
grpc_read_timeout 3600s;
grpc_send_timeout 3600s;
client_body_timeout 3600s;
ignore_invalid_headers off;
nginx.ingress.kubernetes.io/server-alias: bbportal-grpc.exampleco.io
kubernetes.io/tls-acme: "true"
cert-manager.io/common-name: bbportal-grpc.exampleco.io
cert-manager.io/usages: server auth, client auth
spec:
ingressClassName: nginx
rules:
- host: bbportal-grpc.exampleco.io
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: bb-portal
port:
number: 8082
tls:
- secretName: bbportal-grpc-tls # pragma: allowlist secret
hosts:
- bbportal-grpc.exampleco.io
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/server-alias: bbportal.exampleco.io
kubernetes.io/tls-acme: "true"
cert-manager.io/common-name: bbportal.exampleco.io
cert-manager.io/usages: server auth, client auth
name: bbportal-http-ingress
spec:
ingressClassName: nginx
rules:
- host: bbportal.exampleco.io
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: bb-portal
port:
number: 8081
tls:
- secretName: bbportal-http-tls
hosts:
- bbportal.exampleco.io
This kustomization file can be used to generate all the yaml that will be deployed to your cluster with kubectl kustomize . > all.yml or you can apply directly to your cluster assuming you have permissions with to deploy with kubectl apply -k .
---
namespace: bbportal
resources:
- pvc.yml
- deployment.yml
- ext.secret.yml
- svc.yml
- ingress.yml
configMapGenerator:
- name: bbportal-frontend-config
files:
- .env
- name: bbportal-backend-config
files:
- bb_portal.jsonnet
images:
# frontend
- name: ghcr.io/buildbarn/bb-portal-frontend
newTag: 94f72e8e190ca566a2fac54bff423c447b172c53 # pragma: allowlist secret
# backend
- name: ghcr.io/buildbarn/bb-portal-backend
newTag: 20250220T144241Z-94f72e8 # pragma: allowlist secret
Again, your milage may vary depending on your storage needs and mechanisms
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: bb-portal
namespace: bbportal
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
volumeName: some-volume-name
resources:
requests:
storage: 50Gi
storageClassName: local-storage-immediateManifest for a simple bbportal service definition
---
apiVersion: v1
kind: Service
metadata:
name: bb-portal
namespace: bbportal
labels:
app: bb-portal
spec:
type: LoadBalancer
ports:
- port: 8081
targetPort: 8081
protocol: TCP
name: frontend
- port: 8082
targetPort: 8082
protocol: TCP
name: backend
selector:
app: bb-portal