Skip to content

Commit d339990

Browse files
emerzonaronszanto
authored andcommitted
Handle exception for token cost calculation (onyx-dot-app#4474)
The code for token cost calculation fails when using a LiteLLM proxy due to mismatch with the provider naming. For now, just handle this exception and assume cost 0 when that happens instead of breaking the flow - A more precise, LiteLLM proxy based cost calculation (relying in the `/model/info`) LiteLLM Proxy method will be needed
1 parent c81ebd5 commit d339990

File tree

1 file changed

+12
-6
lines changed

1 file changed

+12
-6
lines changed

backend/onyx/llm/utils.py

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -498,15 +498,21 @@ def get_llm_contextual_cost(
498498
num_input_tokens += num_tokens + num_docs * DOCUMENT_SUMMARY_TOKEN_ESTIMATE
499499
num_output_tokens += num_docs * MAX_CONTEXT_TOKENS
500500

501-
usd_per_prompt, usd_per_completion = litellm.cost_per_token(
502-
model=llm.config.model_name,
503-
prompt_tokens=num_input_tokens,
504-
completion_tokens=num_output_tokens,
505-
)
501+
try:
502+
usd_per_prompt, usd_per_completion = litellm.cost_per_token(
503+
model=llm.config.model_name,
504+
prompt_tokens=num_input_tokens,
505+
completion_tokens=num_output_tokens,
506+
)
507+
except Exception:
508+
logger.exception(
509+
f"An unexpected error occurred while calculating cost for model {llm.config.model_name} (potentially due to malformed name). Assuming cost is 0."
510+
)
511+
return 0
512+
506513
# Costs are in USD dollars per million tokens
507514
return usd_per_prompt + usd_per_completion
508515

509-
510516
def get_llm_max_tokens(
511517
model_map: dict,
512518
model_name: str,

0 commit comments

Comments
 (0)