Implement Function Calling, Streamlined Model Listing, and OpenAI-Compatible Chunking #57
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hello @traderpedroso, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
Summary of Changes
This pull request introduces function calling, streamlined model listing, and OpenAI-compatible chunking to enhance the project's functionality and compatibility with the OpenAI API. The model list is now fetched directly from the API, ensuring it's up-to-date. Streaming output is modified to align with the OpenAI API format. Function calling capabilities are implemented, enabling the model to dynamically request and utilize external functions.
Highlights
- Function Calling: Implements function calling, allowing the model to request and use external functions during generation. The changes are primarily in
pkg/adapter/chat.goandpkg/adapter/tools.go, with modifications to handlegenai.FunctionCalland convert OpenAI tools to Gemini tools. - Streamlined Model Listing: Fetches the model list directly from the API instead of hardcoding it locally. This ensures the list is always current. The changes are in
api/handler.goandpkg/adapter/models.go, including fetching models from the Gemini API and initializing them. - OpenAI-Compatible Chunking: Modifies the streaming output to use chunking aligned with the OpenAI API format, ensuring seamless integration. The changes are in
pkg/adapter/chat.go, where the streaming response is formatted character by character.
Changelog
Click here to see the changelog
- api/handler.go
- Fetches the model list from the API instead of using a hardcoded list.
- Initializes Gemini models using the authorization header.
- Handles cases where model mapping is enabled or disabled.
- Initializes Gemini models in the ChatProxyHandler.
- pkg/adapter/chat.go
- Modifies the streaming output to use chunking aligned with the OpenAI API format, sending each character individually.
- Handles
genai.FunctionCallwithin the streaming output. - Adds logic to send the finish reason if present.
- Updates
genaiResponseToStreamCompletionResponseto handle function calls and set the appropriate finish reason. - Adds usage metadata to the OpenAI response.
- pkg/adapter/models.go
- Adds functions to fetch available models from the Gemini API (
FetchGeminiModels). - Adds functions to initialize the Gemini models (
InitGeminiModels) and retrieve them (GetAvailableGeminiModels). - Adds a function to check if a model is a valid Gemini model (
IsValidGeminiModel). - Adds logic to fallback to default models if the requested model is invalid.
- Adds locking to GeminiModels to prevent race conditions.
- Adds functions to fetch available models from the Gemini API (
- pkg/adapter/struct.go
- Adds
ToolsandToolChoicefields to theChatCompletionRequeststruct. - Adds
RoleandToolCallsfields to theCompletionChoicestruct.
- Adds
- pkg/adapter/tools.go
- New file added to convert OpenAI tools to Gemini tools, including handling function parameters and schemas.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Trivia time!
What company developed the Go programming language?
Click here for the answer
Google developed the Go programming language.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request introduces function calling, streamlined model listing, and OpenAI-compatible chunking, which are significant enhancements. The implementation appears well-structured, but there are a few areas that could benefit from further refinement to ensure robustness and clarity.
Summary of Findings
- Error Handling in Model Initialization: The
InitGeminiModelsfunction falls back to default models upon failing to fetch Gemini models, but this fallback might not be sufficient in all cases. Consider providing a mechanism to alert the user or retry the initialization process. - Inconsistent Error Handling: The
ChatProxyHandlerandEmbeddingProxyHandlerfunctions have duplicated code for extracting the API key from the Authorization header. This could be refactored into a common utility function to reduce redundancy and improve maintainability. - Chunking Logic: The chunking logic in
handleStreamIterstreams each character individually, which might introduce overhead. Investigate whether chunking by word or sentence would provide a better balance between compatibility and efficiency.
Merge Readiness
The pull request introduces valuable features and improvements. However, addressing the identified issues related to error handling and chunking logic would enhance the robustness and maintainability of the code. I recommend addressing these points before merging. I am unable to approve this pull request, and others should review and approve this code before merging.
| _, err := fmt.Sscanf(authorizationHeader, "Bearer %s", &apiKey) | ||
| if err == nil { | ||
| adapter.InitGeminiModels(apiKey) |
There was a problem hiding this comment.
Consider handling the error returned by adapter.InitGeminiModels. If initialization fails, the application might not function correctly. At a minimum, log the error.
| _, err := fmt.Sscanf(authorizationHeader, "Bearer %s", &apiKey) | |
| if err == nil { | |
| adapter.InitGeminiModels(apiKey) | |
| if err == nil { | |
| adapter.InitGeminiModels(apiKey) | |
| } else { | |
| log.Printf("Error initializing Gemini models: %v", err) | |
| } |
pkg/adapter/chat.go
Outdated
| // Stream each character individually | ||
| text := string(pp) | ||
| for _, char := range text { | ||
| openaiResp := &CompletionResponse{ | ||
| ID: fmt.Sprintf("chatcmpl-%s", respID), | ||
| Object: "chat.completion.chunk", | ||
| Created: created, | ||
| Model: GetMappedModel(model), | ||
| Choices: []CompletionChoice{ | ||
| { | ||
| Index: 0, | ||
| Delta: struct { | ||
| Content string `json:"content,omitempty"` | ||
| Role string `json:"role,omitempty"` | ||
| ToolCalls []openai.ToolCall `json:"tool_calls,omitempty"` | ||
| }{ | ||
| Content: string(char), | ||
| }, | ||
| }, | ||
| }, | ||
| } | ||
| resp, _ := json.Marshal(openaiResp) | ||
| dataChan <- string(resp) | ||
| } |
There was a problem hiding this comment.
Streaming each character individually might introduce significant overhead. Consider buffering the text into words or sentences before streaming to improve efficiency while maintaining compatibility.
pkg/adapter/models.go
Outdated
| if err != nil { | ||
| log.Printf("Failed to fetch Gemini models: %v\n", err) | ||
| // Fallback to default models | ||
| geminiModelsLock.Lock() | ||
| GeminiModels = []string{Gemini1Dot5Pro, Gemini1Dot5Flash, Gemini1Dot5ProV, Gemini2FlashExp, TextEmbedding004} | ||
| geminiModelsLock.Unlock() | ||
| return |
There was a problem hiding this comment.
While falling back to default models is a good safety net, consider adding a mechanism to alert the user that the model list could not be fetched from the API. This could be done via logging or an application-level notification.
| // Check if the model is valid | ||
| if IsValidGeminiModel(req.Model) { | ||
| return req.Model | ||
| } | ||
|
|
||
| // Fallback to default model if not valid | ||
| log.Printf("Invalid model: %s, falling back to %s\n", req.Model, Gemini1Dot5Flash) | ||
| return Gemini1Dot5Flash |
There was a problem hiding this comment.
Consider returning an error to the caller function, so that it can handle the error appropriately, rather than just logging it and falling back to a default model.
|
@traderpedroso I'm glad to see your PR; it's an important improvement. I'll review it later. Please fix the lint errors first. |
|
@traderpedroso I've just added support for function call responses based on your PR, making the function call feature fully functional. |
This pull request introduces several key improvements to the project, bringing it closer to feature parity with the OpenAI API and enhancing its overall functionality.
Key Changes:
Function Calling: Implemented function calling capabilities, enabling the model to dynamically request and utilize external functions during the generation process. This allows for more complex and interactive applications.
Direct Model Listing via API: The model list is now directly fetched from API rather than being locally hardcoded. This ensures the list of available models is always up-to-date and accurate, reflecting the latest available options without requiring manual updates.
OpenAI-Compatible Chunking: Modified the streaming output to use chunking that is precisely aligned with the OpenAI API format. This ensures seamless integration and compatibility with existing OpenAI-based clients and tools.
Benefits:
Enhanced Functionality: The addition of function calling significantly expands the potential applications of the model.
Improved Model Management: Fetching model list from the API simplifies model management and ensures accuracy.
Seamless Integration: The OpenAI-compatible chunking greatly improves compatibility with existing tools and libraries, reducing integration effort.
Testing:
Thoroughly tested the function calling implementation with various function definitions and scenarios.
Verified that the model list is correctly retrieved and displayed.
Confirmed that the streaming output matches the OpenAI API format.
Next Steps:
Further optimization of function calling performance.
Adding documentation and examples for the new features.