add default_sampling_params

[pathfinder-pf]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @pathfinder-pf, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the engine's capabilities by integrating a mechanism to manage default sampling parameters. It introduces a dedicated attribute to store these parameters and a method to retrieve them efficiently, ensuring that the engine can consistently apply predefined sampling configurations. This change aims to streamline the use of default sampling settings, making the engine more flexible and user-friendly.

Highlights

• Default Sampling Parameters: Introduced a new instance variable default_sampling_params in the engine's constructor to store default sampling configurations.
• Sampling Parameter Retrieval: Added a get_default_sampling_params method to the engine, which fetches and caches default sampling parameters from the model configuration, returning them as a SamplingParams object.
• Type Hinting and Imports: Updated type imports to include Any and added an import for the SamplingParams class, enhancing type safety and clarity.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

The pull request introduces a new method get_default_sampling_params to the Engine class to provide default sampling parameters. However, the implementation has several critical issues that will cause runtime errors, such as referencing non-existent attributes and methods. The code needs to be corrected to properly fetch the configuration from the backend, likely via an RPC call.

[gemini-code-assist]

This new method has several issues that will cause it to fail at runtime:

1. self.llm_engine is not an attribute of the Engine class. The Engine class acts as a client and does not hold a reference to the LLMEngine instance, which runs in a separate process.
2. The ModelConfig class does not have a method named get_diff_sampling_param. This call will result in an AttributeError.
3. The SamplingParams class does not have a class method named from_optional. This will also raise an AttributeError.

It seems the intention is to fetch default sampling parameters from the model's configuration. This likely requires fetching this information from the scheduler process, for example by introducing a new RPC call.

Additionally, the logic if self.default_sampling_params: might be incorrect if an empty dictionary is a valid return value for sampling parameters, as it would evaluate to False. You might want to check for is not None instead.

To instantiate SamplingParams from a dictionary, you can use its constructor directly, e.g., SamplingParams(**params_dict).