From de16e17cd95092a950bce062df33c01717073a95 Mon Sep 17 00:00:00 2001 From: Jintao Zhang Date: Sat, 6 Sep 2025 00:17:03 +0800 Subject: [PATCH 1/2] docs: add reasoning model metrics Signed-off-by: Jintao Zhang --- website/docs/api/router.md | 43 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/website/docs/api/router.md b/website/docs/api/router.md index d422419d..abb86615 100644 --- a/website/docs/api/router.md +++ b/website/docs/api/router.md @@ -175,6 +175,49 @@ semantic_router_pii_detections_total{action="block"} 23 semantic_router_jailbreak_attempts_total{action="block"} 5 ``` +### Reasoning Mode Metrics + +The router exposes dedicated Prometheus counters to monitor reasoning mode decisions and template usage across model families. These metrics are emitted by the router and can be scraped by your Prometheus server. + +- llm_reasoning_decisions_total{category, model, enabled, effort} + - Description: Count of reasoning decisions made per category and selected model, with whether reasoning was enabled and the applied effort level. + - Labels: + - category: category name determined during routing + - model: final selected model for the request + - enabled: "true" or "false" depending on the decision + - effort: effort level used when enabled (e.g., low|medium|high) + +- llm_reasoning_template_usage_total{family, param} + - Description: Count of times a model-family-specific template parameter was applied to requests. + - Labels: + - family: normalized model family (e.g., qwen3, deepseek, gpt-oss, gpt) + - param: name of the template knob applied (e.g., enable_thinking, thinking, reasoning_effort) + +- llm_reasoning_effort_usage_total{family, effort} + - Description: Count of times a reasoning effort level was set for a given model family. + - Labels: + - family: normalized model family + - effort: effort level (e.g., low|medium|high) + +Example PromQL: + +```prometheus +# Reasoning decisions by category and model (last 5m) +sum by (category, model, enabled, effort) ( + rate(llm_reasoning_decisions_total[5m]) +) + +# Template usage by model family and parameter (last 5m) +sum by (family, param) ( + rate(llm_reasoning_template_usage_total[5m]) +) + +# Effort distribution by model family (last 5m) +sum by (family, effort) ( + rate(llm_reasoning_effort_usage_total[5m]) +) +``` + ## gRPC ExtProc API For direct integration with the ExtProc protocol: From 4089a3b77704a06869961264a9558cb453d47bac Mon Sep 17 00:00:00 2001 From: Jintao Zhang Date: Sat, 6 Sep 2025 00:41:26 +0800 Subject: [PATCH 2/2] fix docs build error Signed-off-by: Jintao Zhang --- website/docs/api/router.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/api/router.md b/website/docs/api/router.md index abb86615..26a1e557 100644 --- a/website/docs/api/router.md +++ b/website/docs/api/router.md @@ -179,7 +179,7 @@ semantic_router_jailbreak_attempts_total{action="block"} 5 The router exposes dedicated Prometheus counters to monitor reasoning mode decisions and template usage across model families. These metrics are emitted by the router and can be scraped by your Prometheus server. -- llm_reasoning_decisions_total{category, model, enabled, effort} +- `llm_reasoning_decisions_total{category, model, enabled, effort}` - Description: Count of reasoning decisions made per category and selected model, with whether reasoning was enabled and the applied effort level. - Labels: - category: category name determined during routing @@ -187,13 +187,13 @@ The router exposes dedicated Prometheus counters to monitor reasoning mode decis - enabled: "true" or "false" depending on the decision - effort: effort level used when enabled (e.g., low|medium|high) -- llm_reasoning_template_usage_total{family, param} +- `llm_reasoning_template_usage_total{family, param}` - Description: Count of times a model-family-specific template parameter was applied to requests. - Labels: - family: normalized model family (e.g., qwen3, deepseek, gpt-oss, gpt) - param: name of the template knob applied (e.g., enable_thinking, thinking, reasoning_effort) -- llm_reasoning_effort_usage_total{family, effort} +- `llm_reasoning_effort_usage_total{family, effort}` - Description: Count of times a reasoning effort level was set for a given model family. - Labels: - family: normalized model family