Token Counter Widget

[emerzon]

Description

This change adds a Token counter widget near the Chat Inputbox

Data Flow

1. Token Capture: When LiteLLM processes a request, it returns usage data that includes prompt_tokens, completion_tokens, and total_tokens
2. Storage: The token_usage_tracker stores this data in thread-local storage for cross-thread access
3. Message Creation: When creating chat messages, the token usage is retrieved and stored in the database
4. Frontend Display: The frontend receives token usage with each message and updates the counter
5. Session Management: Token counters reset when switching between chat sessions

Key Features

• Real-time Tracking: Updates token usage as messages stream
• Visual Feedback: Color-coded progress bar (green → yellow → red based on usage)
• Detailed Breakdown: Tooltip shows billed vs context tokens, including reasoning tokens for o1 models
• Model-aware: Displays correct token limits based on the selected LLM model
• Incremental Billing: Tracks actual billable tokens (not counting repeated context)

How Has This Been Tested?

Start a new chat session, after the LLM response, the widget should appear.

[vercel]

@emerzon is attempting to deploy a commit to the Danswer Team on Vercel.

A member of the Team first needs to authorize it.

[greptile-apps]

PR Summary

Implements a token counter widget with complete backend-to-frontend infrastructure for tracking LLM token usage in chat sessions.

• Added .orig and .rej files in backend/onyx/file_processing/ appear to be merge artifacts that should be removed
• The token_usage_tracker.py implementation using thread-local storage could have thread safety issues, particularly around the 60-second cleanup mechanism
• Document processing library imports (docx, openpyxl, pptx) were removed from extract_file_text.py while their functions are still referenced
• Duplicated token limit calculation logic exists between ChatPage.tsx and ChatInputBar.tsx that should be centralized
• New database JSONB column token_usage properly tracks structured usage data (prompt_tokens, completion_tokens, total_tokens) for billing accuracy

_{17 files reviewed, 19 comments
Edit PR Review Bot Settings | Greptile}

[greptile-apps]

style: 200000 should be defined as a named constant at the module level for better maintainability

[greptile-apps]

style: The token_usage field should have a more specific type rather than dict[str, Any]. Consider creating a TokenUsage Pydantic model to enforce the expected structure (prompt_tokens, completion_tokens, total_tokens).

[emerzon]

#4995 (comment)

[greptile-apps]

style: Add field description in the comment to explain what token_usage metrics (prompt_tokens, completion_tokens, total_tokens) can be expected in this field

[emerzon]

Probably better to not define this. Several types of tokens could show up in the response in the future (audio tokens, image tokens, etc)

[greptile-apps]

logic: Migration ID should be longer (typical format is 32 chars). Current format differs from other migrations like 58c50ef19f08

[emerzon]

This will need review at merge time I believe

[github-actions]

This PR is stale because it has been open 75 days with no activity. Remove stale label or comment or this will be closed in 15 days.