feat: sync pools added to core for memory optimisations #572
+145,778
−4,606
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…sts (#443) ## Improve streaming response handling for Anthropic, Bedrock, and other providers This PR enhances the streaming response handling across multiple providers, focusing on consistent end-of-stream behavior and proper usage information collection. ## Changes - Added `AnthropicUsage` type to properly handle Anthropic-specific usage information - Refactored streaming handlers to collect usage and finish reason throughout the stream and send them in the final response - Created utility functions `createBifrostChatCompletionChunkResponse` and `handleStreamEndWithSuccess` to standardize stream completion behavior - Improved AWS Event Stream parsing in Bedrock provider to handle large buffers and binary data more reliably - Fixed import ordering in Anthropic provider - Standardized stream chunk indexing to increment only when actual content is sent - Enhanced test scenarios for speech synthesis and transcription with round-trip validation ## Type of change - [x] Bug fix - [x] Feature - [x] Refactor ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (Next.js) - [ ] Docs ## How to test Run the provider tests to verify streaming behavior: ```sh # Core/Transports go test ./tests/core-providers/... ``` ## Breaking changes - [ ] Yes - [x] No ## Related issues Improves streaming response handling and fixes inconsistencies in stream completion. ## Checklist - [x] I added/updated tests where appropriate - [x] I verified builds succeed (Go) - [x] I verified the CI pipeline passes locally
## Summary Improve handling of redacted values for Bedrock provider configurations by properly merging keys when updating provider settings. ## Changes - Added support for handling Bedrock config redacted values in the `mergeKeys` function - Enhanced the `IsRedacted` function to detect any string containing "***" as a redacted value, rather than only matching a specific pattern of 4 chars + 24 asterisks + 4 chars ## Type of change - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [x] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (Next.js) - [ ] Docs ## How to test 1. Configure a Bedrock provider with valid credentials 2. Update the provider configuration with redacted values 3. Verify that the original credentials are preserved and not replaced with redacted values ```sh # Core/Transports go version go test ./... ``` ## Breaking changes - [ ] Yes - [x] No ## Security considerations This change improves the handling of AWS Bedrock credentials, ensuring they are properly preserved when updating provider configurations with redacted values. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [x] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [x] I verified the CI pipeline passes locally if applicable
## Summary Added environment variable support for provider configurations and improved provider CRUD operations in the SQLite config store. ## Changes - Added `processEnvValue` function to handle environment variable references in configuration values - Implemented granular provider CRUD operations in SQLite config store: - `AddProvider`: Creates a new provider configuration - `UpdateProvider`: Updates an existing provider without recreating all providers - `DeleteProvider`: Removes a provider and its associated keys - Updated the HTTP transport to use these new specific methods instead of the generic `UpdateProvidersConfig` - Added environment variable processing for provider keys and their configurations (Azure, Vertex, Bedrock) ## Type of change - [x] Feature - [x] Refactor ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [x] Providers/Integrations ## How to test ```sh # Test environment variable support export MY_API_KEY="test-key-value" # Configure a provider with "env.MY_API_KEY" as the key value # Verify the key is properly resolved when retrieved # Test provider CRUD operations go test ./framework/configstore/... go test ./transports/bifrost-http/... ``` ## Breaking changes - [x] No ## Security considerations This change improves security by allowing API keys and other sensitive configuration to be stored as environment variables rather than directly in the database. ## Checklist - [x] I added/updated tests where appropriate - [x] I verified builds succeed (Go and UI)
…ection strings (#437) ## Summary Implement environment variable processing for MCP client connection strings, allowing users to securely store sensitive connection information as environment variables. ## Changes - Added context to MCPManager initialization in Bifrost - Enhanced the SQLiteConfigStore to process environment variables in MCP client connection strings - Modified the ConfigStore interface to include environment variable mapping when updating MCP configurations - Added a new utility function `substituteMCPEnvVars` to handle environment variable substitution for MCP configs - Updated the HTTP transport layer to properly handle environment variables in MCP configurations - Fixed an issue where MCP config wasn't being properly assigned after loading ## Type of change - [x] Feature - [x] Bug fix ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) ## How to test 1. Configure an MCP client with a connection string containing an environment variable: ```sh # Set environment variable export MCP_CONNECTION_STRING="your-connection-string-here" # Configure MCP client with env.MCP_CONNECTION_STRING as the connection string ``` 2. Verify that the environment variable is properly resolved when the MCP client is used 3. Check that when saving and loading configurations, the environment variable references are preserved ## Breaking changes - [x] No ## Security considerations This change improves security by allowing sensitive connection strings to be stored as environment variables rather than in configuration files or databases. ## Checklist - [x] I added/updated tests where appropriate - [x] I verified builds succeed (Go and UI)
## Summary This PR adds a new Pricing Module to the framework package, enhances documentation for the framework components, and makes several improvements to existing features. ## Changes - Added a new Pricing Module for dynamic model pricing and cost calculation - Updated the Weaviate vector store documentation with improved interface usage examples - Added a "What is framework?" overview document explaining the purpose of the framework package - Updated the Docker installation section in the release script (removed "Recommended" label) - Enhanced telemetry documentation with cost monitoring examples - Fixed log store filtering to support min/max cost ranges - Added config store validation in the governance handler ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [x] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [ ] UI (Next.js) - [x] Docs ## How to test Test the pricing module functionality: ```sh # Core/Transports go test ./framework/pricing/... # Test log filtering with cost ranges curl 'http://localhost:8080/api/logs?min_cost=0.001&max_cost=10' ``` ## Breaking changes - [ ] Yes - [x] No ## Related issues Implements the pricing module for cost tracking and calculation. ## Security considerations The pricing module handles sensitive cost data but doesn't introduce new security concerns. ## Checklist - [x] I added/updated tests where appropriate - [x] I updated documentation where needed - [x] I verified builds succeed (Go and UI)
…448) ## Summary Implement governance configuration retrieval from the database and improve configuration loading logic to prioritize database-stored configurations over file-based ones. ## Changes - Simplified `GovernanceConfig` struct by removing `Enabled` and `IsVkMandatory` fields - Added `GetGovernanceConfig()` method to retrieve governance data from the database - Added `AfterFind` hook for `TableVirtualKey` to sanitize sensitive key data when retrieving virtual keys - Enhanced configuration loading to follow a consistent pattern: check database first, fall back to config file if needed, update database with file values when appropriate - Improved logging during configuration loading process - Implemented transaction-based governance config updates when loading from file - Bumped framework version to 1.0.13 and transports version to 1.2.10 ## Type of change - [x] Feature - [x] Refactor ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) ## How to test ```sh # Core/Transports go version go test ./... # Test configuration loading with both empty and populated database # Verify governance configuration is properly retrieved from database # Verify fallback to file-based configuration works when database is empty ``` ## Breaking changes - [x] No ## Security considerations The implementation adds security improvements by sanitizing sensitive key data when retrieving virtual keys through the `AfterFind` hook, ensuring API keys and credentials aren't unnecessarily exposed.
## Summary Improved error handling for pricing manager initialization and standardized warning messages across plugins when pricing manager is unavailable. solves issue #451 ## Changes - Updated warning message in governance plugin to clarify that cost calculations will be skipped - Updated warning message in logging plugin to match the governance plugin's wording - Added missing logger parameter to telemetry plugin's Init function - Added warning message in telemetry plugin when pricing manager is nil - Changed fatal error to regular error when failing to initialize pricing manager - Removed unnecessary log message when initializing log store ## Type of change - [x] Bug fix - [ ] Feature - [x] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [ ] Providers/Integrations - [x] Plugins - [ ] UI (Next.js) - [ ] Docs ## How to test ```sh # Core/Transports go test ./plugins/governance/... go test ./plugins/logging/... go test ./plugins/telemetry/... go test ./transports/bifrost-http/... # Start the server without pricing manager configuration # Verify warning messages appear in logs but service continues to run ``` ## Breaking changes - [ ] Yes - [x] No ## Security considerations No security implications. This PR only improves error handling and messaging. ## Checklist - [x] I added/updated tests where appropriate - [x] I verified builds succeed (Go and UI)
## Add configurable maximum request body size This PR adds a configurable maximum request body size setting to improve security and prevent potential denial of service attacks through extremely large requests. ## Changes - Added `MaxRequestBodySizeMB` field to the `ClientConfig` struct with a default value of 100MB - Implemented backward compatibility to handle existing configurations without this field - Applied the setting to the FastHTTP server configuration - Added UI controls to allow administrators to adjust this setting - Updated the configuration schema to include the new field - Renamed the telemetry plugin from "bifrost-http-telemetry" to "telemetry" for consistency - Extracted HTTP transport configuration defaults into constants - Bumped transport version from 1.2.10 to 1.2.11 ## Type of change - [ ] Bug fix - [x] Feature - [x] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [ ] Providers/Integrations - [x] Plugins - [x] UI (Next.js) - [ ] Docs ## How to test ```sh # Core/Transports go version go test ./... # Verify the server starts with the new configuration defaults go run transports/bifrost-http/main.go ``` ## Breaking changes - [ ] Yes - [x] No ## Security considerations The addition of a maximum request body size (100MB) improves security by preventing potential denial of service attacks through extremely large requests. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [x] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [x] I verified the CI pipeline passes locally if applicable
chore: bump plugin versions for next release
Pratham-Mishra04
force-pushed
the
10-07-feat_sync_pools_added_to_all_internal_structs
branch
from
290c61b to
e284b38
Compare
Pratham-Mishra04
force-pushed
the
08-30-feat_core_extended
branch
from
d80798a to
e13e8a7
Compare
## Add LiteLLM Fallback Support for Text Completions with Groq
This PR adds support for LiteLLM's fallback mechanism that allows text completion requests to work with providers that only support chat completions (like Groq). When the `x-litellm-fallback` header is set to "true", Bifrost will convert text completion requests to chat completion requests for providers that don't natively support text completions.
## Changes
- Added support for the `x-litellm-fallback` header in context conversion
- Implemented text-to-chat completion conversion in the Groq provider
- Added `ToBifrostChatRequest()` method to convert text completion requests to chat requests
- Added `ToTextCompletionResponse()` method to convert chat responses back to text completion format
- Updated error message from "text not provided" to "prompt not provided" for consistency
- Refactored header processing in context conversion to use `strings.CutPrefix()`
## Type of change
- [x] Feature
- [x] Refactor
## Affected areas
- [x] Core (Go)
- [x] Transports (HTTP)
- [x] Providers/Integrations
## How to test
Test text completion requests with the Groq provider using the `x-litellm-fallback` header:
```sh
curl -X POST http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-H "x-litellm-fallback: true" \
-d '{
"model": "groq/llama2-70b-4096",
"prompt": "Write a poem about AI",
"max_tokens": 100
}'
```
## Breaking changes
- [x] No
## Related issues
Improves compatibility with LiteLLM clients that expect text completion support for all providers.
## Security considerations
No additional security implications as this only affects request/response format conversion.
## Checklist
- [x] I added/updated tests where appropriate
- [x] I verified builds succeed (Go and UI)
Pratham-Mishra04
changed the base branch from
08-30-feat_core_extended
to
graphite-base/572
Pratham-Mishra04
force-pushed
the
graphite-base/572
branch
from
e13e8a7 to
5a18bbc
Compare
Pratham-Mishra04
force-pushed
the
10-07-feat_sync_pools_added_to_all_internal_structs
branch
from
e284b38 to
0ed8c8e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Refactored the Cohere provider implementation to simplify message content handling and improve code maintainability.
Changes
CohereMessageContentstruct instead of helper methodsNewStringContent,NewBlocksContent,IsString,IsBlocks,GetString,GetBlocks)Type of change
Affected areas
How to test
Verify that Cohere API requests and responses are properly handled with both string content and block content.
Breaking changes
Checklist