Merge pull request #3616 from gardleopard/documentation_of_metrics_pulling

docs(prometheus): Document how to set up scraping in kubernetes

Author: GeorgianaElena

docs/source/administrator/troubleshooting.md

@@ -24,3 +24,32 @@ managing cost with JupyterHub, see {ref}`cost`.
 Each Helm Chart is packaged with a specific version of JupyterHub (and
 other software as well). See see the [Helm Chart repository](https://github.com/jupyterhub/helm-chart#release-notes)
 for information about the versions of relevant software packages.
+
+## Metrics scraping with prometheus or vmagent
+
+Network policy needs to be modified in order for prometheus or vmagent to be able to reach the metrics endpoint. The recommended way is by setting [interNamespaceAccessLabels=accept](https://z2jh.jupyter.org/en/stable/resources/reference.html#hub-networkpolicy-internamespaceaccesslabels). This makes the hub pod's associated NetworkPolicy accept ingress from pods in other namespaces that have specific access labels.
+
+```yaml
+hub:
+  networkPolicy:
+    interNamespaceAccessLabels: accept
+```
+
+And then set [`prometheus.server.podLabels`](https://github.com/prometheus-community/helm-charts/blob/0c7bf42ac2265d13845ffe0c499d16e6b8cdedea/charts/prometheus/values.yaml#L554) to be `hub.jupyter.org/network-access-hub: "true"` to enable prometheus to reach the hub.
+
+Alternatively you can also set an explicit ingress rule to allow the prometheus or vmagent pod to reach the hub pod to scrape metrics.
+
+```yaml
+hub:
+  networkPolicy:
+    ingress:
+      - from:
+          - namespaceSelector:
+              matchLabels:
+                # namespace where your prometheus or vmagent is running
+                name: <namespace>
+          - podSelector:
+              matchLabels:
+                # a valid selector for the pod that needs to reach jupyterhub
+                app.kubernetes.io/instance: vmagent
+```