Skip to content

add probes #4762

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 23, 2025
Merged

add probes #4762

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d0fa596
add probes
May 23, 2025
59965c7
lint fixes
May 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions deployment/helm/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,20 @@
* from source root run the following. This does a very basic test against the web server
* ct install --all --helm-extra-set-args="--set=nginx.enabled=false" --debug --config ct.yaml

## Output template to file and inspect
* cd charts/onyx
* helm template test-output . > test-output.yaml

## Test the entire cluster manually
* cd charts/onyx
* helm install onyx . -n onyx --set postgresql.primary.persistence.enabled=false
* the postgres flag is to keep the storage ephemeral for testing, you probably don't want to set that in prod
* the postgres flag is to keep the storage ephemeral for testing. You probably don't want to set that in prod.
* no flag for ephemeral vespa storage yet, might be good for testing
* kubectl -n onyx port-forward service/onyx-nginx 8080:80
* this will forward the local port 8080 to the installed chart for you to run tests, etc.
* When you are finished
* helm uninstall onyx -n onyx
* Vespa leaves behind a PVC - delete it if you are completely done
* Vespa leaves behind a PVC. Delete it if you are completely done.
* k -n onyx get pvc
* k -n onyx delete pvc vespa-storage-da-vespa-0
Comment on lines 29 to 30
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style: Consider using full kubectl command instead of shorthand 'k' in documentation for clarity

Suggested change
* k -n onyx get pvc
* k -n onyx delete pvc vespa-storage-da-vespa-0
* kubectl -n onyx get pvc
* kubectl -n onyx delete pvc vespa-storage-da-vespa-0

* If you didn't disable Postgres persistence earlier, you may want to delete that PVC too.
22 changes: 22 additions & 0 deletions deployment/helm/charts/onyx/templates/celery-worker-heavy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,25 @@ spec:
name: {{ .Values.config.envConfigMapName }}
env:
{{- include "onyx-stack.envSecrets" . | nindent 12}}
startupProbe:
{{ .Values.celery_shared.startupProbe | toYaml | nindent 12}}
readinessProbe:
{{ .Values.celery_shared.readinessProbe | toYaml | nindent 12}}
exec:
command:
- /bin/bash
- -c
- >
python onyx/background/celery/celery_k8s_probe.py
--probe readiness
--filename /tmp/onyx_k8s_heavy_readiness.txt
livenessProbe:
{{ .Values.celery_shared.livenessProbe | toYaml | nindent 12}}
exec:
command:
- /bin/bash
- -c
- >
python onyx/background/celery/celery_k8s_probe.py
--probe liveness
--filename /tmp/onyx_k8s_heavy_liveness.txt
22 changes: 22 additions & 0 deletions deployment/helm/charts/onyx/templates/celery-worker-indexing.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,3 +59,25 @@ spec:
- name: ENABLE_MULTIPASS_INDEXING
value: "{{ .Values.celery_worker_indexing.enableMiniChunk }}"
{{- include "onyx-stack.envSecrets" . | nindent 12}}
startupProbe:
{{ .Values.celery_shared.startupProbe | toYaml | nindent 12}}
readinessProbe:
{{ .Values.celery_shared.readinessProbe | toYaml | nindent 12}}
exec:
command:
- /bin/bash
- -c
- >
python onyx/background/celery/celery_k8s_probe.py
--probe readiness
--filename /tmp/onyx_k8s_indexing_readiness.txt
livenessProbe:
{{ .Values.celery_shared.livenessProbe | toYaml | nindent 12}}
exec:
command:
- /bin/bash
- -c
- >
python onyx/background/celery/celery_k8s_probe.py
--probe liveness
--filename /tmp/onyx_k8s_indexing_liveness.txt
22 changes: 22 additions & 0 deletions deployment/helm/charts/onyx/templates/celery-worker-light.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,25 @@ spec:
name: {{ .Values.config.envConfigMapName }}
env:
{{- include "onyx-stack.envSecrets" . | nindent 12}}
startupProbe:
{{ .Values.celery_shared.startupProbe | toYaml | nindent 12}}
readinessProbe:
{{ .Values.celery_shared.readinessProbe | toYaml | nindent 12}}
exec:
command:
- /bin/bash
- -c
- >
python onyx/background/celery/celery_k8s_probe.py
--probe readiness
--filename /tmp/onyx_k8s_light_readiness.txt
livenessProbe:
{{ .Values.celery_shared.livenessProbe | toYaml | nindent 12}}
exec:
command:
- /bin/bash
- -c
- >
python onyx/background/celery/celery_k8s_probe.py
--probe liveness
--filename /tmp/onyx_k8s_light_liveness.txt
22 changes: 22 additions & 0 deletions deployment/helm/charts/onyx/templates/celery-worker-monitoring.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,25 @@ spec:
name: {{ .Values.config.envConfigMapName }}
env:
{{- include "onyx-stack.envSecrets" . | nindent 12}}
startupProbe:
{{ .Values.celery_shared.startupProbe | toYaml | nindent 12}}
readinessProbe:
{{ .Values.celery_shared.readinessProbe | toYaml | nindent 12}}
exec:
command:
- /bin/bash
- -c
- >
python onyx/background/celery/celery_k8s_probe.py
--probe readiness
--filename /tmp/onyx_k8s_monitoring_readiness.txt
livenessProbe:
{{ .Values.celery_shared.livenessProbe | toYaml | nindent 12}}
exec:
command:
- /bin/bash
- -c
- >
python onyx/background/celery/celery_k8s_probe.py
--probe liveness
--filename /tmp/onyx_k8s_monitoring_liveness.txt
22 changes: 22 additions & 0 deletions deployment/helm/charts/onyx/templates/celery-worker-primary.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,25 @@ spec:
name: {{ .Values.config.envConfigMapName }}
env:
{{- include "onyx-stack.envSecrets" . | nindent 12}}
startupProbe:
{{ .Values.celery_shared.startupProbe | toYaml | nindent 12}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logic: The startupProbe configuration is incomplete - it's missing the exec command block that's present in the other probes

readinessProbe:
{{ .Values.celery_shared.readinessProbe | toYaml | nindent 12}}
exec:
Comment on lines +63 to +64
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logic: The readinessProbe has duplicate configuration - values from celery_shared.readinessProbe may conflict with the explicit exec command block

command:
- /bin/bash
- -c
- >
python onyx/background/celery/celery_k8s_probe.py
--probe readiness
--filename /tmp/onyx_k8s_primary_readiness.txt
livenessProbe:
{{ .Values.celery_shared.livenessProbe | toYaml | nindent 12}}
exec:
command:
- /bin/bash
- -c
- >
python onyx/background/celery/celery_k8s_probe.py
--probe liveness
--filename /tmp/onyx_k8s_primary_liveness.txt
22 changes: 22 additions & 0 deletions deployment/helm/charts/onyx/templates/celery-worker-user-files-indexing.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,25 @@ spec:
name: {{ .Values.config.envConfigMapName }}
env:
{{- include "onyx-stack.envSecrets" . | nindent 12}}
startupProbe:
{{ .Values.celery_shared.startupProbe | toYaml | nindent 12}}
readinessProbe:
{{ .Values.celery_shared.readinessProbe | toYaml | nindent 12}}
exec:
command:
- /bin/bash
- -c
- >
python onyx/background/celery/celery_k8s_probe.py
--probe readiness
--filename /tmp/onyx_k8s_userfilesindexing_readiness.txt
livenessProbe:
{{ .Values.celery_shared.livenessProbe | toYaml | nindent 12}}
exec:
command:
- /bin/bash
- -c
- >
python onyx/background/celery/celery_k8s_probe.py
--probe liveness
--filename /tmp/onyx_k8s_userfilesindexing_liveness.txt
20 changes: 20 additions & 0 deletions deployment/helm/charts/onyx/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -359,6 +359,26 @@ celery_shared:
repository: onyxdotapp/onyx-backend
pullPolicy: IfNotPresent
tag: "" # Overrides the image tag whose default is the chart appVersion.
startupProbe:
# startupProbe fails after 2m
exec:
command: ["test", "-f", "/app/onyx/main.py"]
failureThreshold: 24
periodSeconds: 5
timeoutSeconds: 3
readinessProbe:
# readinessProbe fails after 15s + 2m of inactivity
# it's ok to see the readinessProbe fail transiently while the container starts
initialDelaySeconds: 15
periodSeconds: 5
failureThreshold: 24
timeoutSeconds: 3
livenessProbe:
# livenessProbe fails after 5m of inactivity
initialDelaySeconds: 60
periodSeconds: 60
failureThreshold: 5
timeoutSeconds: 3

celery_beat:
replicaCount: 1
Expand Down